<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: renanpyd</title>
    <description>The latest articles on DEV Community by renanpyd (@renanpyd).</description>
    <link>https://dev.to/renanpyd</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F687401%2F52339473-a6e4-4292-b2f6-f7286506a222.jpeg</url>
      <title>DEV Community: renanpyd</title>
      <link>https://dev.to/renanpyd</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/renanpyd"/>
    <language>en</language>
    <item>
      <title>Como aprendi Apache Spark — Parte 18: Performance Tuning avançado</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Mon, 09 Nov 2026 19:49:26 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-18-performance-tuning-avancado-20gd</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-18-performance-tuning-avancado-20gd</guid>
      <description>&lt;h1&gt;
  
  
  Como aprendi Apache Spark — Parte 18: Performance Tuning avançado com Catalyst, AQE e Spark UI
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Nota histórica: esta série revisita minha trajetória em Engenharia de Dados iniciada em 2012 e conecta seus fundamentos às tecnologias modernas. Recursos como Adaptive Query Execution e as otimizações atuais do Spark SQL surgiram e amadureceram posteriormente.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nos capítulos anteriores, exploramos os fundamentos da execução distribuída do Apache Spark.&lt;/p&gt;

&lt;p&gt;Entendemos Driver, Executors, Jobs, Stages, Tasks, partições, shuffle, memória, joins e armazenamento colunar.&lt;/p&gt;

&lt;p&gt;Também estudamos Structured Streaming, CDC, Lakehouse, Data Quality, Airflow e DataOps.&lt;/p&gt;

&lt;p&gt;Agora chegamos a um tema que conecta praticamente tudo isso:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Performance Tuning.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Imagine que um pipeline Spark processa 500 milhões de registros.&lt;/p&gt;

&lt;p&gt;A execução demora 90 minutos.&lt;/p&gt;

&lt;p&gt;A primeira reação de muitas equipes é aumentar a infraestrutura.&lt;/p&gt;

&lt;p&gt;Mais Executors.&lt;/p&gt;

&lt;p&gt;Mais memória.&lt;/p&gt;

&lt;p&gt;Mais CPU.&lt;/p&gt;

&lt;p&gt;Mas será que o problema realmente está na capacidade computacional?&lt;/p&gt;

&lt;p&gt;Talvez o Spark esteja lendo colunas desnecessárias.&lt;/p&gt;

&lt;p&gt;Talvez exista um shuffle gigantesco.&lt;/p&gt;

&lt;p&gt;Talvez um join esteja utilizando uma estratégia inadequada.&lt;/p&gt;

&lt;p&gt;Talvez uma única partição esteja concentrando metade dos dados.&lt;/p&gt;

&lt;p&gt;Ou talvez os Executors estejam passando grande parte do tempo aguardando operações de I/O.&lt;/p&gt;

&lt;p&gt;Antes de aumentar recursos, precisamos compreender o comportamento da aplicação.&lt;/p&gt;

&lt;p&gt;A principal ideia deste capítulo é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Otimização de Spark começa pelo diagnóstico, não pela configuração.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  1. Performance não é apenas tempo de execução
&lt;/h2&gt;

&lt;p&gt;Quando avaliamos performance, normalmente começamos pela duração do Job.&lt;/p&gt;

&lt;p&gt;Mas esse indicador sozinho não explica o problema.&lt;/p&gt;

&lt;p&gt;Considere duas aplicações:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application A
Execution Time: 10 minutes
Infrastructure Cost: High

Application B
Execution Time: 12 minutes
Infrastructure Cost: Low
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Qual delas é melhor?&lt;/p&gt;

&lt;p&gt;Depende dos requisitos.&lt;/p&gt;

&lt;p&gt;Se o SLA é de 15 minutos, a aplicação B pode oferecer melhor eficiência econômica.&lt;/p&gt;

&lt;p&gt;Agora imagine que a aplicação A atende um serviço crítico com SLA de 11 minutos.&lt;/p&gt;

&lt;p&gt;Nesse caso, a conclusão pode ser diferente.&lt;/p&gt;

&lt;p&gt;Por isso, devemos avaliar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tempo de execução.&lt;/li&gt;
&lt;li&gt;Throughput.&lt;/li&gt;
&lt;li&gt;Latência.&lt;/li&gt;
&lt;li&gt;CPU.&lt;/li&gt;
&lt;li&gt;Memória.&lt;/li&gt;
&lt;li&gt;Shuffle.&lt;/li&gt;
&lt;li&gt;Spill.&lt;/li&gt;
&lt;li&gt;I/O.&lt;/li&gt;
&lt;li&gt;Custo.&lt;/li&gt;
&lt;li&gt;Confiabilidade.&lt;/li&gt;
&lt;li&gt;Cumprimento de SLA.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Performance Tuning é uma atividade de otimização multidimensional.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. O caminho de uma consulta Spark SQL
&lt;/h2&gt;

&lt;p&gt;Quando escrevemos uma transformação com DataFrames, não estamos especificando diretamente todas as operações físicas.&lt;/p&gt;

&lt;p&gt;O Spark constrói e otimiza planos de execução.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;           DATAFRAME / SQL
                  |
                  v
           UNRESOLVED PLAN
                  |
                  v
             ANALYSIS
                  |
                  v
        ANALYZED LOGICAL PLAN
                  |
                  v
         LOGICAL OPTIMIZATION
                  |
                  v
       OPTIMIZED LOGICAL PLAN
                  |
                  v
         PHYSICAL PLANNING
                  |
                  v
           PHYSICAL PLAN
                  |
                  v
              EXECUTION
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse processo envolve componentes importantes do Spark SQL, especialmente o Catalyst Optimizer.&lt;/p&gt;

&lt;p&gt;Compreender esse fluxo ajuda a entender por que duas consultas aparentemente semelhantes podem apresentar desempenhos muito diferentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. O que é Catalyst Optimizer?
&lt;/h2&gt;

&lt;p&gt;O &lt;strong&gt;Catalyst Optimizer&lt;/strong&gt; é o framework de otimização de consultas utilizado pelo Spark SQL.&lt;/p&gt;

&lt;p&gt;Ele trabalha com representações estruturadas de expressões e planos.&lt;/p&gt;

&lt;p&gt;Entre suas responsabilidades estão:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Análise e resolução de atributos.&lt;/li&gt;
&lt;li&gt;Aplicação de regras de otimização lógica.&lt;/li&gt;
&lt;li&gt;Simplificação de expressões.&lt;/li&gt;
&lt;li&gt;Reorganização de determinadas operações.&lt;/li&gt;
&lt;li&gt;Apoio ao planejamento físico.&lt;/li&gt;
&lt;li&gt;Seleção de estratégias de execução.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por exemplo, considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Spark pode utilizar técnicas como:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Predicate Pushdown&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Tentar aproximar filtros da leitura dos dados, quando o formato e a fonte oferecem suporte.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Column Pruning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Evitar a leitura ou propagação de colunas desnecessárias.&lt;/p&gt;

&lt;p&gt;Essas otimizações podem reduzir significativamente o trabalho executado.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Predicate Pushdown
&lt;/h2&gt;

&lt;p&gt;Imagine uma tabela Parquet com 100 milhões de registros.&lt;/p&gt;

&lt;p&gt;Queremos apenas pedidos pagos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;pedidos_pagos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se a fonte e o plano permitirem, o Spark poderá encaminhar parte da condição para o mecanismo de leitura.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WITHOUT PUSHDOWN

Read All Data
      |
      v
Spark Filter
      |
      v
Result
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WITH PUSHDOWN

Storage Reader
      |
      v
Apply Supported Predicate
      |
      v
Read Relevant Data
      |
      v
Result
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas existe uma ressalva importante.&lt;/p&gt;

&lt;p&gt;Predicate Pushdown não significa que o Spark necessariamente deixará de ler todos os arquivos ou todos os bytes.&lt;/p&gt;

&lt;p&gt;O benefício depende do formato, das estatísticas disponíveis, da organização dos dados e das capacidades do leitor.&lt;/p&gt;

&lt;p&gt;Em Parquet, por exemplo, estatísticas de row groups podem permitir eliminar determinadas regiões de leitura.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Column Pruning
&lt;/h2&gt;

&lt;p&gt;Imagine uma tabela com 80 colunas.&lt;/p&gt;

&lt;p&gt;Mas precisamos apenas de três.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em formatos colunares, o Spark pode evitar ler colunas que não são necessárias.&lt;/p&gt;

&lt;p&gt;Isso reduz:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;I/O.&lt;/li&gt;
&lt;li&gt;Desserialização.&lt;/li&gt;
&lt;li&gt;Memória.&lt;/li&gt;
&lt;li&gt;Transferência de dados.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;É uma das razões pelas quais formatos como Parquet e ORC são tão importantes em workloads analíticos.&lt;/p&gt;

&lt;p&gt;Mas devemos lembrar que nem todas as fontes oferecem as mesmas capacidades de projeção.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Inspecionando o plano com &lt;code&gt;explain()&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Uma das ferramentas mais importantes para Performance Tuning é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;formatted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;extended&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Também podemos solicitar informações de custo quando disponíveis:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cost&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O modo &lt;code&gt;cost&lt;/code&gt; pode apresentar estatísticas e estimativas, dependendo da disponibilidade de metadados.&lt;/p&gt;

&lt;p&gt;O plano físico mostra como o Spark pretende executar a consulta.&lt;/p&gt;

&lt;p&gt;Ele pode revelar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Scans.&lt;/li&gt;
&lt;li&gt;Filters.&lt;/li&gt;
&lt;li&gt;Projections.&lt;/li&gt;
&lt;li&gt;Exchanges.&lt;/li&gt;
&lt;li&gt;Sorts.&lt;/li&gt;
&lt;li&gt;Joins.&lt;/li&gt;
&lt;li&gt;Aggregations.&lt;/li&gt;
&lt;li&gt;Estratégias de particionamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O plano é uma das primeiras evidências que devemos analisar.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Entendendo &lt;code&gt;Exchange&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Quando observamos um plano físico, um operador merece atenção especial:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Exchange
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ele frequentemente indica uma redistribuição de dados entre partições.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HashAggregate
      |
      v
Exchange hashpartitioning(...)
      |
      v
HashAggregate
      |
      v
Scan
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa redistribuição pode envolver shuffle.&lt;/p&gt;

&lt;p&gt;Como estudamos na Parte 4, shuffle pode exigir:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Serialização.&lt;/li&gt;
&lt;li&gt;Escrita em disco.&lt;/li&gt;
&lt;li&gt;Transferência pela rede.&lt;/li&gt;
&lt;li&gt;Leitura remota.&lt;/li&gt;
&lt;li&gt;Ordenação.&lt;/li&gt;
&lt;li&gt;Gerenciamento de memória.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por isso, operadores &lt;code&gt;Exchange&lt;/code&gt; são pontos importantes de investigação.&lt;/p&gt;

&lt;p&gt;Mas nem todo Exchange é um problema.&lt;/p&gt;

&lt;p&gt;Muitas operações distribuídas precisam redistribuir dados para produzir resultados corretos.&lt;/p&gt;

&lt;p&gt;O objetivo é eliminar redistribuições desnecessárias e tornar as necessárias mais eficientes.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. O papel do Tungsten
&lt;/h2&gt;

&lt;p&gt;Outro conjunto importante de otimizações do Spark é associado ao projeto &lt;strong&gt;Tungsten&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Tungsten introduziu e consolidou técnicas de execução orientadas à eficiência de CPU e memória.&lt;/p&gt;

&lt;p&gt;Entre elas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Representações internas compactas.&lt;/li&gt;
&lt;li&gt;Gerenciamento eficiente de memória.&lt;/li&gt;
&lt;li&gt;Geração de código.&lt;/li&gt;
&lt;li&gt;Redução de overhead de objetos.&lt;/li&gt;
&lt;li&gt;Execução otimizada de operadores.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma dessas técnicas é conhecida como &lt;strong&gt;Whole-Stage Code Generation&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;O Spark pode combinar determinadas operações físicas em código gerado, reduzindo overhead de execução.&lt;/p&gt;

&lt;p&gt;Em planos físicos, podemos encontrar marcadores associados a operadores que participam de code generation.&lt;/p&gt;

&lt;p&gt;Isso não significa que todos os operadores sejam automaticamente combinados.&lt;/p&gt;

&lt;p&gt;A aplicação depende das operações e das capacidades do mecanismo.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Whole-Stage Code Generation
&lt;/h2&gt;

&lt;p&gt;Imagine uma sequência:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Scan
  |
  v
Filter
  |
  v
Project
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em determinadas condições, o Spark pode gerar código para executar operações compatíveis de maneira integrada.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WITHOUT FUSION

Scan
  |
  v
Filter
  |
  v
Project
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WITH CODE GENERATION

+---------------------+
| Generated Execution |
| Scan + Filter       |
| + Project           |
+---------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O objetivo é reduzir overhead entre operadores.&lt;/p&gt;

&lt;p&gt;Entretanto, a presença de code generation não garante que uma consulta seja rápida.&lt;/p&gt;

&lt;p&gt;Se o principal gargalo for I/O remoto ou shuffle, outras otimizações poderão ter impacto maior.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Adaptive Query Execution
&lt;/h2&gt;

&lt;p&gt;Agora chegamos a um dos recursos mais importantes do Spark SQL moderno.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Adaptive Query Execution (AQE)&lt;/strong&gt; permite que o Spark ajuste determinadas decisões de execução utilizando estatísticas coletadas durante o processamento.&lt;/p&gt;

&lt;p&gt;Sem adaptação, decisões físicas são tomadas com base nas informações disponíveis antes da execução.&lt;/p&gt;

&lt;p&gt;Mas estimativas podem estar incorretas.&lt;/p&gt;

&lt;p&gt;Com AQE, o Spark pode revisar partes do plano após obter informações reais de estágios executados.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;INITIAL PHYSICAL PLAN
          |
          v
EXECUTE QUERY STAGE
          |
          v
COLLECT RUNTIME STATISTICS
          |
          v
ADAPT PHYSICAL PLAN
          |
          v
CONTINUE EXECUTION
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;AQE não significa que o Spark reescreve arbitrariamente toda a aplicação.&lt;/p&gt;

&lt;p&gt;Ele adapta determinadas decisões dentro das capacidades do mecanismo.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. Habilitando AQE
&lt;/h2&gt;

&lt;p&gt;Em versões modernas do Spark SQL, AQE normalmente já vem habilitado por padrão.&lt;/p&gt;

&lt;p&gt;Podemos verificar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;conf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.sql.adaptive.enabled&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou configurar explicitamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;conf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.sql.adaptive.enabled&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A configuração deve ser analisada conforme a versão do Spark.&lt;/p&gt;

&lt;p&gt;Também devemos considerar que determinadas otimizações adaptativas possuem parâmetros próprios.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. Coalescing de partições pós-shuffle
&lt;/h2&gt;

&lt;p&gt;Imagine que configuramos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;conf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.sql.shuffle.partitions&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;200&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas, após um shuffle, descobrimos que a quantidade de dados é pequena.&lt;/p&gt;

&lt;p&gt;Executar 200 tarefas muito pequenas pode introduzir overhead desnecessário.&lt;/p&gt;

&lt;p&gt;AQE pode combinar partições de shuffle.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;BEFORE AQE

200 Small Partitions
        |
        v
200 Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AFTER AQE

Coalesced Partitions
        |
        v
Fewer Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O número final depende das estatísticas e das configurações adaptativas.&lt;/p&gt;

&lt;p&gt;Essa otimização pode reduzir overhead de scheduling e melhorar a eficiência.&lt;/p&gt;

&lt;p&gt;Mas não devemos interpretar isso como uma razão para ignorar completamente o particionamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. Conversão adaptativa de joins
&lt;/h2&gt;

&lt;p&gt;Imagine que o Spark inicialmente planeja um Sort-Merge Join.&lt;/p&gt;

&lt;p&gt;Durante a execução, descobre que um dos lados é suficientemente pequeno.&lt;/p&gt;

&lt;p&gt;Dependendo das condições, AQE pode converter a estratégia para Broadcast Hash Join.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;INITIAL PLAN

SortMergeJoin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ADAPTIVE PLAN

BroadcastHashJoin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa mudança pode evitar trabalho desnecessário de shuffle e sort.&lt;/p&gt;

&lt;p&gt;Mas a conversão depende de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Estatísticas.&lt;/li&gt;
&lt;li&gt;Tipo de join.&lt;/li&gt;
&lt;li&gt;Tamanho dos dados.&lt;/li&gt;
&lt;li&gt;Configurações.&lt;/li&gt;
&lt;li&gt;Restrições do plano.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Não é garantida em qualquer consulta.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. Broadcast Hash Join
&lt;/h2&gt;

&lt;p&gt;Broadcast Join é especialmente útil quando um dos datasets é pequeno.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;FACT TABLE
500 million rows

DIMENSION TABLE
50 thousand rows
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se a dimensão couber confortavelmente nos limites de broadcast, podemos distribuí-la aos Executors.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SMALL DIMENSION
       |
       v
    BROADCAST
       |
       +------ Executor 1
       |
       +------ Executor 2
       |
       +------ Executor 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode evitar a redistribuição completa de ambos os lados.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;broadcast&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fatos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dimensao&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;produto_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;how&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;inner&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas forçar broadcast sem avaliar o tamanho real pode causar pressão de memória ou falhas.&lt;/p&gt;

&lt;p&gt;O objetivo não é utilizar broadcast sempre.&lt;/p&gt;

&lt;p&gt;É utilizá-lo quando fizer sentido.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Sort-Merge Join
&lt;/h2&gt;

&lt;p&gt;Sort-Merge Join é uma estratégia comum para joins distribuídos de grande volume.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;LEFT DATASET
     |
     v
SHUFFLE BY KEY
     |
     v
SORT
     |
     +--------+
              |
              v
             JOIN
              ^
     +--------+
     |
    SORT
     ^
     |
SHUFFLE BY KEY
     ^
     |
RIGHT DATASET
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa estratégia pode ser eficiente para grandes datasets.&lt;/p&gt;

&lt;p&gt;Mas também pode envolver shuffle e sort custosos.&lt;/p&gt;

&lt;p&gt;Precisamos observar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Volume transferido.&lt;/li&gt;
&lt;li&gt;Distribuição das chaves.&lt;/li&gt;
&lt;li&gt;Número de partições.&lt;/li&gt;
&lt;li&gt;Spill.&lt;/li&gt;
&lt;li&gt;Duração das tarefas.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  16. Data Skew
&lt;/h2&gt;

&lt;p&gt;Data Skew acontece quando os dados estão distribuídos de maneira desigual entre partições.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Partition 1: 1 million rows
Partition 2: 1 million rows
Partition 3: 1 million rows
Partition 4: 80 million rows
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mesmo que três Tasks terminem rapidamente, a quarta pode demorar muito.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task 1  ███
Task 2  ███
Task 3  ███
Task 4  ███████████████████████
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse é um dos problemas mais importantes em processamento distribuído.&lt;/p&gt;

&lt;p&gt;Adicionar Executors pode não resolver o gargalo se uma única tarefa continuar concentrando o trabalho.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Skew Join Optimization
&lt;/h2&gt;

&lt;p&gt;AQE pode identificar determinadas partições de shuffle desproporcionalmente grandes em joins compatíveis.&lt;/p&gt;

&lt;p&gt;Quando as condições são atendidas, o Spark pode dividir o processamento dessas partições.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SKEWED PARTITION
        |
        v
    SPLIT WORK
        |
   +----+----+
   |    |    |
   v    v    v
 Task Task Task
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode melhorar o paralelismo.&lt;/p&gt;

&lt;p&gt;Mas a otimização depende de parâmetros como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.sql.adaptive.skewJoin.enabled
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Também existem limiares relacionados ao tamanho e à desproporção das partições.&lt;/p&gt;

&lt;p&gt;É importante verificar o plano adaptativo e as métricas reais para confirmar se a otimização foi aplicada.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. Identificando Skew na prática
&lt;/h2&gt;

&lt;p&gt;Uma maneira simples de investigar a distribuição de uma chave é contar sua frequência.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;distribuicao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;orderBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;count&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;distribuicao&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode revelar chaves muito frequentes.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cliente_id | count
-----------|---------
101        | 50000000
102        | 120000
103        | 95000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas essa consulta também possui custo.&lt;/p&gt;

&lt;p&gt;Em datasets muito grandes, devemos avaliar amostragem, estatísticas e métricas já disponíveis antes de executar diagnósticos adicionais.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. Salting
&lt;/h2&gt;

&lt;p&gt;Uma técnica para lidar com determinadas situações de skew é conhecida como &lt;strong&gt;salting&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A ideia é distribuir registros associados a uma chave muito frequente entre diferentes subchaves artificiais.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Original Key:

cliente_id = 101
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Transformamos em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;101_0
101_1
101_2
101_3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode distribuir o processamento.&lt;/p&gt;

&lt;p&gt;Mas salting precisa ser implementado corretamente nos dois lados de um join.&lt;/p&gt;

&lt;p&gt;Também pode aumentar o volume de dados e introduzir complexidade.&lt;/p&gt;

&lt;p&gt;Portanto, devemos considerá-lo somente após analisar as estratégias nativas do Spark.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Shuffle Partitions
&lt;/h2&gt;

&lt;p&gt;Uma configuração conhecida é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;conf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.sql.shuffle.partitions&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;200&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ela define um número inicial de partições para determinadas operações de shuffle em Spark SQL.&lt;/p&gt;

&lt;p&gt;Mas qual é o número ideal?&lt;/p&gt;

&lt;p&gt;Não existe uma resposta universal.&lt;/p&gt;

&lt;p&gt;Depende de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Volume de dados.&lt;/li&gt;
&lt;li&gt;Tamanho das partições.&lt;/li&gt;
&lt;li&gt;Recursos disponíveis.&lt;/li&gt;
&lt;li&gt;Operações.&lt;/li&gt;
&lt;li&gt;Skew.&lt;/li&gt;
&lt;li&gt;AQE.&lt;/li&gt;
&lt;li&gt;Características do cluster.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Um número muito pequeno pode produzir tarefas grandes.&lt;/p&gt;

&lt;p&gt;Um número excessivo pode aumentar overhead.&lt;/p&gt;

&lt;p&gt;O objetivo é encontrar um equilíbrio.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. &lt;code&gt;repartition()&lt;/code&gt; vs. &lt;code&gt;coalesce()&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Essas operações já apareceram em capítulos anteriores, mas merecem atenção no contexto de tuning.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;repartition()&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode redistribuir dados com shuffle.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_reparticionado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;repartition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;&lt;code&gt;coalesce()&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Normalmente reduz o número de partições sem exigir um shuffle completo.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_reduzido&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;coalesce&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas não devemos assumir que &lt;code&gt;coalesce()&lt;/code&gt; é sempre mais rápido.&lt;/p&gt;

&lt;p&gt;Reduzir demais o número de partições pode diminuir o paralelismo e concentrar trabalho.&lt;/p&gt;

&lt;p&gt;A escolha depende da operação seguinte e da distribuição dos dados.&lt;/p&gt;




&lt;h2&gt;
  
  
  22. Dynamic Partition Pruning
&lt;/h2&gt;

&lt;p&gt;Dynamic Partition Pruning é uma otimização que pode reduzir a leitura de partições em determinadas consultas com joins.&lt;/p&gt;

&lt;p&gt;Imagine uma tabela de fatos particionada por data.&lt;/p&gt;

&lt;p&gt;Queremos apenas datas presentes em uma tabela menor.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DIMENSION / FILTER
        |
        v
RELEVANT PARTITION VALUES
        |
        v
PRUNE FACT PARTITIONS
        |
        v
READ LESS DATA
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Spark pode utilizar informações derivadas da consulta para evitar a leitura de partições desnecessárias.&lt;/p&gt;

&lt;p&gt;Mas a aplicação depende do plano, da estrutura das tabelas e das configurações.&lt;/p&gt;

&lt;p&gt;Não devemos assumir que qualquer filtro em um join produzirá pruning automaticamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. Partition Pruning vs. Predicate Pushdown
&lt;/h2&gt;

&lt;p&gt;Esses conceitos são relacionados, mas diferentes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Partition Pruning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Evita ler partições de armazenamento que não atendem ao filtro.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Predicate Pushdown&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Encaminha predicados para a fonte de dados quando suportado.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PARTITION PRUNING

Read only:
date=2026-11-05
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PREDICATE PUSHDOWN

Within selected files,
apply supported filter
during reading.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ambas as técnicas podem reduzir I/O.&lt;/p&gt;

&lt;p&gt;Mas atuam em níveis diferentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Caching
&lt;/h2&gt;

&lt;p&gt;O Spark permite persistir DataFrames para reutilização.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas o cache é materializado quando uma Action apropriada executa o plano.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois disso, operações posteriores podem reutilizar dados armazenados em cache.&lt;/p&gt;

&lt;p&gt;Entretanto, cache não deve ser utilizado indiscriminadamente.&lt;/p&gt;

&lt;p&gt;Ele consome recursos.&lt;/p&gt;

&lt;p&gt;Se o DataFrame for utilizado apenas uma vez, o custo de materialização pode superar o benefício.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. Quando utilizar cache?
&lt;/h2&gt;

&lt;p&gt;Caching pode fazer sentido quando:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Um DataFrame é reutilizado várias vezes.&lt;/li&gt;
&lt;li&gt;O cálculo original é caro.&lt;/li&gt;
&lt;li&gt;Existe memória suficiente.&lt;/li&gt;
&lt;li&gt;O custo de recomputação é significativo.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Pode ser inadequado quando:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;O dataset é utilizado uma única vez.&lt;/li&gt;
&lt;li&gt;O volume é muito grande.&lt;/li&gt;
&lt;li&gt;Existe pressão de memória.&lt;/li&gt;
&lt;li&gt;O cache provoca eviction ou spill excessivo.&lt;/li&gt;
&lt;li&gt;A origem já oferece leitura eficiente.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma prática importante é liberar recursos quando o cache não for mais necessário.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unpersist&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  26. Spill
&lt;/h2&gt;

&lt;p&gt;Spill acontece quando determinadas operações precisam utilizar armazenamento externo porque os dados não cabem nos recursos de memória disponíveis para aquela execução.&lt;/p&gt;

&lt;p&gt;Isso pode ocorrer em:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Sort.&lt;/li&gt;
&lt;li&gt;Aggregation.&lt;/li&gt;
&lt;li&gt;Join.&lt;/li&gt;
&lt;li&gt;Shuffle.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;IN-MEMORY PROCESSING
         |
         v
MEMORY PRESSURE
         |
         v
DISK SPILL
         |
         v
ADDITIONAL I/O
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Spill não significa necessariamente falha.&lt;/p&gt;

&lt;p&gt;É um mecanismo que permite continuar determinadas operações.&lt;/p&gt;

&lt;p&gt;Mas volumes elevados de spill podem indicar gargalos.&lt;/p&gt;

&lt;p&gt;Precisamos observar métricas de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Memory Bytes Spilled.&lt;/li&gt;
&lt;li&gt;Disk Bytes Spilled.&lt;/li&gt;
&lt;li&gt;Shuffle Read.&lt;/li&gt;
&lt;li&gt;Shuffle Write.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  27. Spark UI
&lt;/h2&gt;

&lt;p&gt;A Spark UI é uma das ferramentas mais importantes para diagnóstico.&lt;/p&gt;

&lt;p&gt;Ela oferece informações sobre:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Jobs.&lt;/li&gt;
&lt;li&gt;Stages.&lt;/li&gt;
&lt;li&gt;Tasks.&lt;/li&gt;
&lt;li&gt;SQL Queries.&lt;/li&gt;
&lt;li&gt;Executors.&lt;/li&gt;
&lt;li&gt;Storage.&lt;/li&gt;
&lt;li&gt;Environment.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Durante uma aplicação local, a interface normalmente pode ser acessada em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;http://localhost:4040
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A porta pode mudar quando já está ocupada.&lt;/p&gt;

&lt;p&gt;Em ambientes distribuídos, o acesso depende da configuração e da infraestrutura.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. O que observar na Spark UI?
&lt;/h2&gt;

&lt;p&gt;Quando uma aplicação está lenta, começo por perguntas como:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Qual Stage está consumindo mais tempo?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Existe uma Task muito mais lenta do que as demais?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quanto shuffle está sendo realizado?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Existe spill?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Os Executors estão ocupados ou ociosos?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Há falhas ou retries?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O plano SQL corresponde à estratégia esperada?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Essas perguntas ajudam a direcionar a investigação.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. Interpretando Tasks lentas
&lt;/h2&gt;

&lt;p&gt;Imagine um Stage com 100 Tasks.&lt;/p&gt;

&lt;p&gt;Noventa e nove terminam em 10 segundos.&lt;/p&gt;

&lt;p&gt;Uma demora cinco minutos.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;99 Tasks: 10 seconds
 1 Task : 300 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse padrão sugere investigar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Data Skew.&lt;/li&gt;
&lt;li&gt;Tamanho desigual das partições.&lt;/li&gt;
&lt;li&gt;Problemas localizados de I/O.&lt;/li&gt;
&lt;li&gt;Garbage Collection.&lt;/li&gt;
&lt;li&gt;Executor específico.&lt;/li&gt;
&lt;li&gt;Localidade dos dados.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Não devemos concluir automaticamente que é skew.&lt;/p&gt;

&lt;p&gt;Precisamos cruzar as métricas.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. Garbage Collection
&lt;/h2&gt;

&lt;p&gt;O Spark executa sobre a JVM em grande parte de sua arquitetura.&lt;/p&gt;

&lt;p&gt;Pressão de memória e criação excessiva de objetos podem aumentar o tempo gasto em Garbage Collection.&lt;/p&gt;

&lt;p&gt;Na Spark UI, métricas de GC podem ajudar a identificar esse comportamento.&lt;/p&gt;

&lt;p&gt;Mas GC elevado pode ser consequência de diferentes problemas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Objetos em excesso.&lt;/li&gt;
&lt;li&gt;Partições grandes.&lt;/li&gt;
&lt;li&gt;Estruturas intermediárias.&lt;/li&gt;
&lt;li&gt;Configuração inadequada.&lt;/li&gt;
&lt;li&gt;Pressão de memória.&lt;/li&gt;
&lt;li&gt;UDFs ou operações específicas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A solução depende da causa.&lt;/p&gt;

&lt;p&gt;Aumentar memória indiscriminadamente pode não resolver o problema.&lt;/p&gt;




&lt;h2&gt;
  
  
  31. Python UDFs vs. funções nativas
&lt;/h2&gt;

&lt;p&gt;Em PySpark, podemos criar User Defined Functions.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.types&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;StringType&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;udf&lt;/span&gt;

&lt;span class="nd"&gt;@udf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;classificar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ALTO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;valor&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;BAIXO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas, quando possível, funções nativas do Spark SQL costumam oferecer melhor integração com o mecanismo de otimização.&lt;/p&gt;

&lt;p&gt;Uma alternativa:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;classificacao&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ALTO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;otherwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;BAIXO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Funções nativas podem permitir execução mais eficiente e melhor otimização.&lt;/p&gt;

&lt;p&gt;Isso não significa que todas as UDFs sejam inadequadas.&lt;/p&gt;

&lt;p&gt;Mas devemos evitar UDFs quando uma expressão nativa resolve o problema de maneira simples.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Pandas UDFs e execução vetorizada
&lt;/h2&gt;

&lt;p&gt;Pandas UDFs podem utilizar Apache Arrow para transferências colunares entre JVM e Python em operações compatíveis.&lt;/p&gt;

&lt;p&gt;Isso pode reduzir overhead em comparação com determinadas UDFs Python convencionais.&lt;/p&gt;

&lt;p&gt;Entretanto, seu desempenho depende de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tipo de operação.&lt;/li&gt;
&lt;li&gt;Tamanho dos batches.&lt;/li&gt;
&lt;li&gt;Conversão de dados.&lt;/li&gt;
&lt;li&gt;Memória.&lt;/li&gt;
&lt;li&gt;Versões.&lt;/li&gt;
&lt;li&gt;Compatibilidade.&lt;/li&gt;
&lt;li&gt;Custo da função.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Não devemos assumir que uma Pandas UDF será sempre mais rápida do que uma expressão nativa Spark SQL.&lt;/p&gt;

&lt;p&gt;Em muitos cenários, funções nativas continuam sendo a primeira opção.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. Small Files Problem
&lt;/h2&gt;

&lt;p&gt;Imagine uma tabela com milhares de arquivos muito pequenos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;file_0001.parquet
file_0002.parquet
file_0003.parquet
...
file_9999.parquet
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mesmo que o volume total seja moderado, o número de arquivos pode aumentar overhead de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Listagem.&lt;/li&gt;
&lt;li&gt;Planejamento.&lt;/li&gt;
&lt;li&gt;Abertura.&lt;/li&gt;
&lt;li&gt;Leitura.&lt;/li&gt;
&lt;li&gt;Metadados.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Esse problema aparece frequentemente em pipelines incrementais e streaming.&lt;/p&gt;

&lt;p&gt;Formatos Lakehouse podem oferecer mecanismos de manutenção e compactação, dependendo da implementação.&lt;/p&gt;

&lt;p&gt;Mas a estratégia deve considerar frequência, custo e concorrência.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Tamanho de arquivos e particionamento
&lt;/h2&gt;

&lt;p&gt;Não existe um tamanho universal ideal de arquivo.&lt;/p&gt;

&lt;p&gt;A escolha depende de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Formato.&lt;/li&gt;
&lt;li&gt;Engine.&lt;/li&gt;
&lt;li&gt;Storage.&lt;/li&gt;
&lt;li&gt;Padrão de leitura.&lt;/li&gt;
&lt;li&gt;Frequência de atualização.&lt;/li&gt;
&lt;li&gt;Concorrência.&lt;/li&gt;
&lt;li&gt;Volume.&lt;/li&gt;
&lt;li&gt;Custo.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Arquivos muito pequenos aumentam overhead.&lt;/p&gt;

&lt;p&gt;Arquivos excessivamente grandes podem limitar paralelismo ou tornar determinadas operações menos eficientes.&lt;/p&gt;

&lt;p&gt;Precisamos medir o comportamento real.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. Benchmarking
&lt;/h2&gt;

&lt;p&gt;Otimização sem medição é especulação.&lt;/p&gt;

&lt;p&gt;Uma abordagem correta compara versões sob condições controladas.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Baseline:
Execution Time = 40 minutes

Optimized:
Execution Time = 25 minutes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos calcular:&lt;/p&gt;

&lt;p&gt;[&lt;/p&gt;

&lt;h1&gt;
  
  
  \text{Speedup}
&lt;/h1&gt;

&lt;p&gt;\frac{T_{\text{baseline}}}&lt;br&gt;
{T_{\text{optimized}}}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Nesse exemplo:&lt;/p&gt;

&lt;p&gt;[&lt;/p&gt;
&lt;h1&gt;
  
  
  \text{Speedup}
&lt;/h1&gt;
&lt;h1&gt;
  
  
  \frac{40}{25}
&lt;/h1&gt;

&lt;p&gt;1.6&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Ou seja, a execução otimizada foi 1,6 vez mais rápida.&lt;/p&gt;

&lt;p&gt;A redução percentual do tempo foi:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
\frac{40-25}{40}\times100=37.5\%&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Mas precisamos garantir que os resultados sejam equivalentes.&lt;/p&gt;

&lt;p&gt;Uma otimização que produz dados incorretos não representa melhoria.&lt;/p&gt;


&lt;h2&gt;
  
  
  36. Como realizar um benchmark confiável?
&lt;/h2&gt;

&lt;p&gt;Uma metodologia básica:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Definir o workload.&lt;/li&gt;
&lt;li&gt;Registrar versões e configurações.&lt;/li&gt;
&lt;li&gt;Fixar ou documentar o volume de dados.&lt;/li&gt;
&lt;li&gt;Medir o baseline.&lt;/li&gt;
&lt;li&gt;Identificar o gargalo.&lt;/li&gt;
&lt;li&gt;Alterar uma variável relevante.&lt;/li&gt;
&lt;li&gt;Reexecutar.&lt;/li&gt;
&lt;li&gt;Comparar métricas.&lt;/li&gt;
&lt;li&gt;Validar resultados.&lt;/li&gt;
&lt;li&gt;Registrar conclusões.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Também devemos considerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Cache.&lt;/li&gt;
&lt;li&gt;Aquecimento da JVM.&lt;/li&gt;
&lt;li&gt;Concorrência.&lt;/li&gt;
&lt;li&gt;Variabilidade de I/O.&lt;/li&gt;
&lt;li&gt;Estado do cluster.&lt;/li&gt;
&lt;li&gt;Estatísticas.&lt;/li&gt;
&lt;li&gt;Número de execuções.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma única execução pode não ser representativa.&lt;/p&gt;


&lt;h2&gt;
  
  
  37. Laboratório prático: analisando planos de execução
&lt;/h2&gt;

&lt;p&gt;Vamos criar dois DataFrames e observar o plano de um join.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;functions&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;master&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;local[*]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SparkPerformanceLab&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.sql.adaptive.enabled&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;fatos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1_000_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;produto_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;dimensao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumnRenamed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;produto_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;categoria&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;concat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CAT_&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;produto_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora realizamos o join:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fatos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dimensao&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;produto_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;how&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;inner&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Inspecionamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;formatted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Dependendo das estatísticas e configurações, o Spark poderá escolher uma estratégia de broadcast.&lt;/p&gt;




&lt;h2&gt;
  
  
  38. Comparando com Broadcast Hint
&lt;/h2&gt;

&lt;p&gt;Podemos criar uma versão com broadcast explícito.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;broadcast&lt;/span&gt;

&lt;span class="n"&gt;resultado_broadcast&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fatos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dimensao&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;produto_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;how&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;inner&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado_broadcast&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;formatted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora podemos comparar os planos.&lt;/p&gt;

&lt;p&gt;O objetivo não é provar que o hint é sempre melhor.&lt;/p&gt;

&lt;p&gt;É observar se o Spark já havia escolhido broadcast automaticamente e se o hint realmente modifica o plano.&lt;/p&gt;




&lt;h2&gt;
  
  
  39. Medindo tempo de execução
&lt;/h2&gt;

&lt;p&gt;Podemos utilizar &lt;code&gt;time.perf_counter()&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="n"&gt;inicio&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;duracao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;inicio&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Registros:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Duração:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;duracao&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois repetimos para outra estratégia.&lt;/p&gt;

&lt;p&gt;Mas existe um detalhe importante.&lt;/p&gt;

&lt;p&gt;O otimizador pode eliminar operações que não são necessárias para calcular um simples &lt;code&gt;count()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Portanto, esse benchmark pode não medir todo o custo das transformações que queremos comparar.&lt;/p&gt;

&lt;p&gt;Para comparar joins, devemos utilizar uma Action que efetivamente dependa das colunas produzidas pelo join e verificar o plano executado.&lt;/p&gt;

&lt;p&gt;Essa é uma lição importante:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O benchmark precisa executar o trabalho que pretendemos medir.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  40. Uma metodologia de Performance Tuning
&lt;/h2&gt;

&lt;p&gt;Quando recebo uma aplicação Spark lenta, sigo uma sequência lógica.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. Understand the Workload
           |
           v
2. Inspect Physical Plan
           |
           v
3. Analyze Spark UI
           |
           v
4. Identify Bottleneck
           |
           v
5. Form Hypothesis
           |
           v
6. Apply Controlled Change
           |
           v
7. Benchmark
           |
           v
8. Validate Correctness
           |
           v
9. Document Results
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa abordagem evita alterações aleatórias.&lt;/p&gt;

&lt;p&gt;Também facilita explicar tecnicamente por que determinada otimização foi realizada.&lt;/p&gt;




&lt;h2&gt;
  
  
  41. Matriz de diagnóstico
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Sintoma&lt;/th&gt;
&lt;th&gt;Possíveis causas&lt;/th&gt;
&lt;th&gt;Primeira investigação&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Stage muito lento&lt;/td&gt;
&lt;td&gt;Shuffle, skew, I/O&lt;/td&gt;
&lt;td&gt;Spark UI e plano físico&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Uma Task muito lenta&lt;/td&gt;
&lt;td&gt;Skew, partição grande&lt;/td&gt;
&lt;td&gt;Distribuição de Tasks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spill elevado&lt;/td&gt;
&lt;td&gt;Memória, sort, agregação&lt;/td&gt;
&lt;td&gt;Métricas de spill&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Muitos Jobs pequenos&lt;/td&gt;
&lt;td&gt;Overhead, ações repetidas&lt;/td&gt;
&lt;td&gt;Jobs e código&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Leitura excessiva&lt;/td&gt;
&lt;td&gt;Falta de pruning&lt;/td&gt;
&lt;td&gt;Scan e filtros&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Join custoso&lt;/td&gt;
&lt;td&gt;Estratégia inadequada&lt;/td&gt;
&lt;td&gt;Plano de join&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GC elevado&lt;/td&gt;
&lt;td&gt;Pressão de memória&lt;/td&gt;
&lt;td&gt;Executors e GC&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Muitos arquivos&lt;/td&gt;
&lt;td&gt;Small Files&lt;/td&gt;
&lt;td&gt;Layout de armazenamento&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Baixo paralelismo&lt;/td&gt;
&lt;td&gt;Poucas partições&lt;/td&gt;
&lt;td&gt;Tasks e recursos&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Custo elevado&lt;/td&gt;
&lt;td&gt;Recursos ociosos ou excesso de trabalho&lt;/td&gt;
&lt;td&gt;Métricas e billing&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Essa matriz não substitui diagnóstico.&lt;/p&gt;

&lt;p&gt;Ela ajuda a formular hipóteses.&lt;/p&gt;




&lt;h2&gt;
  
  
  42. Erros comuns em Performance Tuning
&lt;/h2&gt;

&lt;p&gt;Alguns erros aparecem repetidamente.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Aumentar Executors antes de analisar o plano.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode elevar custos sem resolver o gargalo.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Forçar broadcast indiscriminadamente.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode causar pressão de memória.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Aumentar shuffle partitions sem medir.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode aumentar overhead.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reduzir partições demais.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode limitar paralelismo.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Utilizar cache em todos os DataFrames.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode consumir memória sem benefício.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ignorar Data Skew.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Uma única Task pode determinar a duração do Stage.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Utilizar Python UDFs para operações nativas.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode introduzir overhead desnecessário.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ignorar Small Files.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode prejudicar planejamento e leitura.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Comparar benchmarks sem controlar variáveis.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode produzir conclusões incorretas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Otimizar tempo sem validar resultado.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode comprometer a qualidade dos dados.&lt;/p&gt;




&lt;h2&gt;
  
  
  43. Performance e custo
&lt;/h2&gt;

&lt;p&gt;Uma otimização não precisa apenas reduzir tempo.&lt;/p&gt;

&lt;p&gt;Ela pode reduzir custo.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Before:
20 Executors
40 Minutes

After:
10 Executors
35 Minutes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A segunda execução utiliza menos recursos e termina mais rapidamente.&lt;/p&gt;

&lt;p&gt;Mas precisamos considerar o modelo de cobrança e a utilização real.&lt;/p&gt;

&lt;p&gt;Em plataformas cloud, custo depende de fatores como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tipo de instância.&lt;/li&gt;
&lt;li&gt;Tempo de uso.&lt;/li&gt;
&lt;li&gt;Autoscaling.&lt;/li&gt;
&lt;li&gt;Storage.&lt;/li&gt;
&lt;li&gt;Rede.&lt;/li&gt;
&lt;li&gt;Serviços gerenciados.&lt;/li&gt;
&lt;li&gt;Overhead de inicialização.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por isso, performance tuning e FinOps estão relacionados.&lt;/p&gt;

&lt;p&gt;A melhor configuração é aquela que atende os requisitos com eficiência técnica e econômica.&lt;/p&gt;




&lt;h2&gt;
  
  
  44. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando avalio uma aplicação Spark, não começo perguntando:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Quantos Executors devemos adicionar?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Começo perguntando:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qual operação está lenta?&lt;/li&gt;
&lt;li&gt;Qual Stage concentra o tempo?&lt;/li&gt;
&lt;li&gt;Existe shuffle excessivo?&lt;/li&gt;
&lt;li&gt;Há Data Skew?&lt;/li&gt;
&lt;li&gt;O Spark está lendo dados desnecessários?&lt;/li&gt;
&lt;li&gt;Qual estratégia de join foi escolhida?&lt;/li&gt;
&lt;li&gt;AQE está ativo?&lt;/li&gt;
&lt;li&gt;Existe spill?&lt;/li&gt;
&lt;li&gt;O cache realmente ajuda?&lt;/li&gt;
&lt;li&gt;O número de arquivos é adequado?&lt;/li&gt;
&lt;li&gt;O benchmark é representativo?&lt;/li&gt;
&lt;li&gt;Qual é o custo por execução?&lt;/li&gt;
&lt;li&gt;O resultado continua correto?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Essas perguntas transformam tuning em engenharia baseada em evidências.&lt;/p&gt;




&lt;h2&gt;
  
  
  45. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;O Apache Spark possui mecanismos sofisticados de otimização.&lt;/p&gt;

&lt;p&gt;Catalyst ajuda a otimizar consultas.&lt;/p&gt;

&lt;p&gt;Tungsten melhora a eficiência da execução.&lt;/p&gt;

&lt;p&gt;AQE adapta determinadas decisões com base em estatísticas reais.&lt;/p&gt;

&lt;p&gt;Spark UI permite investigar o comportamento da aplicação.&lt;/p&gt;

&lt;p&gt;Mas nenhuma dessas capacidades elimina a necessidade de compreender os dados e o workload.&lt;/p&gt;

&lt;p&gt;O princípio mais importante é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Performance Tuning não consiste em encontrar uma configuração mágica. Consiste em identificar o gargalo real, formular uma hipótese, medir e validar a melhoria.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora que entendemos como diagnosticar e otimizar Jobs Spark, vamos avançar para a operação em ambientes distribuídos e cloud.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Apache Spark em produção: Kubernetes, Databricks, AWS EMR e arquiteturas cloud.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Vamos explorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Spark Standalone.&lt;/li&gt;
&lt;li&gt;YARN.&lt;/li&gt;
&lt;li&gt;Kubernetes.&lt;/li&gt;
&lt;li&gt;Databricks.&lt;/li&gt;
&lt;li&gt;AWS EMR.&lt;/li&gt;
&lt;li&gt;Separação entre compute e storage.&lt;/li&gt;
&lt;li&gt;Deploy Mode.&lt;/li&gt;
&lt;li&gt;Autoscaling.&lt;/li&gt;
&lt;li&gt;Dynamic Allocation.&lt;/li&gt;
&lt;li&gt;Segurança.&lt;/li&gt;
&lt;li&gt;Observabilidade.&lt;/li&gt;
&lt;li&gt;Custos.&lt;/li&gt;
&lt;li&gt;Estratégias de arquitetura.&lt;/li&gt;
&lt;li&gt;Critérios para escolher uma plataforma.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central será:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como escolher e operar a infraestrutura adequada para aplicações Spark em produção?&lt;/strong&gt;&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 17:&lt;/strong&gt; Orquestração, Airflow, CI/CD e DataOps&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 19 — Spark em produção: Kubernetes, Databricks, AWS EMR e arquiteturas cloud&lt;/p&gt;

</description>
      <category>python</category>
      <category>sql</category>
      <category>apache</category>
      <category>spark</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 17: Airflow, CI/CD e DataOps</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 20:40:22 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-17-airflow-cicd-e-dataops-1l5i</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-17-airflow-cicd-e-dataops-1l5i</guid>
      <description>&lt;h1&gt;
  
  
  Como aprendi Apache Spark — Parte 17: Orquestração, Airflow, CI/CD e DataOps
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Nota histórica: esta série revisita minha trajetória em Engenharia de Dados iniciada em 2012 e conecta seus fundamentos às práticas modernas. Apache Airflow, DataOps e as ferramentas atuais de CI/CD passaram por evoluções importantes desde então.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nos primeiros capítulos desta série, concentramos nossa atenção em como o Apache Spark executa processamento distribuído.&lt;/p&gt;

&lt;p&gt;Estudamos Driver, Executors, Jobs, Stages, Tasks, partições, shuffle, memória, formatos de armazenamento, Lakehouse e Structured Streaming.&lt;/p&gt;

&lt;p&gt;Na Parte 16, exploramos Kafka, Debezium e CDC para construir pipelines incrementais.&lt;/p&gt;

&lt;p&gt;Mas existe uma diferença enorme entre executar um Job Spark manualmente e manter uma plataforma de dados funcionando todos os dias.&lt;/p&gt;

&lt;p&gt;Imagine que temos cinco processos:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Extrair dados do PostgreSQL.&lt;/li&gt;
&lt;li&gt;Validar os arquivos recebidos.&lt;/li&gt;
&lt;li&gt;Executar transformações Spark.&lt;/li&gt;
&lt;li&gt;Atualizar tabelas analíticas.&lt;/li&gt;
&lt;li&gt;Publicar métricas para consumidores.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Executar cada processo individualmente pode funcionar durante o desenvolvimento.&lt;/p&gt;

&lt;p&gt;Mas o que acontece quando a extração falha?&lt;/p&gt;

&lt;p&gt;E se o Spark terminar, mas a qualidade dos dados estiver abaixo do esperado?&lt;/p&gt;

&lt;p&gt;Como reprocessar somente um dia?&lt;/p&gt;

&lt;p&gt;Como impedir que duas execuções concorrentes publiquem resultados inconsistentes?&lt;/p&gt;

&lt;p&gt;Como garantir que uma alteração no código não quebre os pipelines existentes?&lt;/p&gt;

&lt;p&gt;Essas perguntas nos levam à &lt;strong&gt;orquestração, CI/CD e DataOps&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Processamento não é orquestração
&lt;/h2&gt;

&lt;p&gt;O Apache Spark é um mecanismo de processamento distribuído.&lt;/p&gt;

&lt;p&gt;Ele executa transformações sobre dados.&lt;/p&gt;

&lt;p&gt;O Apache Airflow é uma plataforma de orquestração de workflows.&lt;/p&gt;

&lt;p&gt;Ele organiza dependências, agendamentos, execuções e estados de tarefas.&lt;/p&gt;

&lt;p&gt;Essa diferença é fundamental.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;             APACHE AIRFLOW
                   |
          Orchestrates Tasks
                   |
        +----------+----------+
        |          |          |
        v          v          v
     Ingestion   Spark Job   Quality Check
                   |
                   v
            Distributed Data
              Processing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Airflow não substitui o Spark.&lt;/p&gt;

&lt;p&gt;E o Spark não precisa assumir sozinho todas as responsabilidades de coordenação de workflows.&lt;/p&gt;

&lt;p&gt;Em uma arquitetura corporativa, as duas tecnologias podem trabalhar juntas.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. O problema dos scripts isolados
&lt;/h2&gt;

&lt;p&gt;Imagine um pipeline executado por cron:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;0 2 &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; python processar_vendas.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa configuração pode ser suficiente para um processo simples.&lt;/p&gt;

&lt;p&gt;Mas considere uma arquitetura com dependências:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Extract Customers
        |
        v
Extract Orders
        |
        v
Transform Orders
        |
        v
Validate Quality
        |
        v
Publish Analytics
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se uma tarefa falhar, precisamos controlar quais etapas podem continuar.&lt;/p&gt;

&lt;p&gt;Também precisamos registrar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qual execução falhou.&lt;/li&gt;
&lt;li&gt;Qual tarefa falhou.&lt;/li&gt;
&lt;li&gt;Quando ocorreu.&lt;/li&gt;
&lt;li&gt;Quantas tentativas foram realizadas.&lt;/li&gt;
&lt;li&gt;Quais dados foram processados.&lt;/li&gt;
&lt;li&gt;Qual versão do código estava em execução.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Um orquestrador ajuda a organizar essas responsabilidades.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. O que é Apache Airflow?
&lt;/h2&gt;

&lt;p&gt;Apache Airflow é uma plataforma para desenvolver, agendar e monitorar workflows.&lt;/p&gt;

&lt;p&gt;Seu modelo é baseado em DAGs — &lt;em&gt;Directed Acyclic Graphs&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Uma DAG representa tarefas e dependências sem ciclos direcionados.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  START
                    |
                    v
                 EXTRACT
                    |
                    v
                TRANSFORM
                    |
                    v
                 VALIDATE
                    |
                    v
                 PUBLISH
                    |
                    v
                   END
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Airflow controla a execução dessas tarefas conforme a definição do workflow.&lt;/p&gt;

&lt;p&gt;É importante lembrar que a API e a arquitetura interna do Airflow evoluíram entre versões.&lt;/p&gt;

&lt;p&gt;Neste artigo, os exemplos utilizam a API moderna do Airflow 3, com imports do pacote &lt;code&gt;airflow.sdk&lt;/code&gt;, e devem ser executados em um ambiente compatível.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. DAG, Task e DAG Run
&lt;/h2&gt;

&lt;p&gt;Três conceitos precisam ficar claros.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DAG&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Representa o workflow e suas dependências.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Task&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Representa uma unidade de trabalho dentro da DAG.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DAG Run&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Representa uma instância de execução do workflow.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DAG: daily_sales_pipeline
           |
           +-- DAG Run: 2026-11-04
           |       |
           |       +-- extract
           |       +-- transform
           |       +-- validate
           |
           +-- DAG Run: 2026-11-05
                   |
                   +-- extract
                   +-- transform
                   +-- validate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cada execução possui seu próprio contexto e estado.&lt;/p&gt;

&lt;p&gt;Isso permite acompanhar histórico, falhas e reprocessamentos.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Uma DAG simples
&lt;/h2&gt;

&lt;p&gt;Vamos construir uma DAG utilizando TaskFlow API.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;airflow.sdk&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dag&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;


&lt;span class="nd"&gt;@dag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dag_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark_pipeline_example&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;schedule&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;@daily&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;start_date&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2026&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;catchup&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;tags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dataops&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;spark_pipeline_example&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;

    &lt;span class="nd"&gt;@task&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;extract&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Extraindo dados&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nd"&gt;@task&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;transform&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Transformando dados&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nd"&gt;@task&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;validate&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Validando dados&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nd"&gt;@task&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;publish&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Publicando resultados&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;extraction&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;extract&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;transformation&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;transform&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;validation&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;validate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;publication&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;publish&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;extraction&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;transformation&lt;/span&gt;
    &lt;span class="n"&gt;transformation&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;validation&lt;/span&gt;
    &lt;span class="n"&gt;validation&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;publication&lt;/span&gt;


&lt;span class="nf"&gt;spark_pipeline_example&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse exemplo demonstra dependências.&lt;/p&gt;

&lt;p&gt;As funções apenas imprimem mensagens; ainda não executam Spark.&lt;/p&gt;

&lt;p&gt;Em produção, cada tarefa deve possuir uma responsabilidade clara e observável.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Como o Airflow executa tarefas?
&lt;/h2&gt;

&lt;p&gt;Uma instalação Airflow pode conter componentes responsáveis por:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Gerenciamento das DAGs.&lt;/li&gt;
&lt;li&gt;Agendamento.&lt;/li&gt;
&lt;li&gt;Execução das tarefas.&lt;/li&gt;
&lt;li&gt;Persistência de metadados.&lt;/li&gt;
&lt;li&gt;Interface de acompanhamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A arquitetura específica depende do executor e da versão.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 AIRFLOW DAG
                     |
                     v
                  SCHEDULER
                     |
                     v
                TASK EXECUTION
                     |
          +----------+----------+
          |          |          |
          v          v          v
        Task A     Task B     Task C
                     |
                     v
                SPARK CLUSTER
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Airflow decide quando e sob quais dependências uma tarefa deve executar.&lt;/p&gt;

&lt;p&gt;O Spark continua responsável pelo processamento distribuído.&lt;/p&gt;

&lt;p&gt;Essa separação evita confundir orquestração com computação.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Integrando Airflow e Spark
&lt;/h2&gt;

&lt;p&gt;Existem diferentes maneiras de executar aplicações Spark a partir do Airflow.&lt;/p&gt;

&lt;p&gt;Uma opção é utilizar &lt;code&gt;SparkSubmitOperator&lt;/code&gt;, disponibilizado pelo provider Apache Spark.&lt;/p&gt;

&lt;p&gt;Exemplo ilustrativo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;airflow.sdk&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dag&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;airflow.providers.apache.spark.operators.spark_submit&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSubmitOperator&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="nd"&gt;@dag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dag_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;daily_sales_spark&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;schedule&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;@daily&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;start_date&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2026&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;catchup&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;tags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;daily_sales_spark&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;

    &lt;span class="n"&gt;process_sales&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;SparkSubmitOperator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;task_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;process_sales&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;application&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/opt/spark/jobs/process_sales.py&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;conn_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark_default&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;application_args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--processing-date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{ ds }}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="nf"&gt;daily_sales_spark&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse exemplo pressupõe:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Airflow instalado.&lt;/li&gt;
&lt;li&gt;Provider Spark compatível.&lt;/li&gt;
&lt;li&gt;Conexão &lt;code&gt;spark_default&lt;/code&gt; configurada.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;spark-submit&lt;/code&gt; disponível no ambiente apropriado.&lt;/li&gt;
&lt;li&gt;Aplicação acessível ao processo executor.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A configuração real depende do deployment.&lt;/p&gt;

&lt;p&gt;Em Kubernetes, por exemplo, podemos utilizar estratégias diferentes de submissão e gerenciamento de aplicações.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. O papel do &lt;code&gt;spark-submit&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;O &lt;code&gt;spark-submit&lt;/code&gt; é uma interface utilizada para submeter aplicações Spark.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;spark-submit &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--master&lt;/span&gt; &lt;span class="nb"&gt;local&lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="k"&gt;*&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--name&lt;/span&gt; sales-processing &lt;span class="se"&gt;\&lt;/span&gt;
  process_sales.py &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--processing-date&lt;/span&gt; 2026-11-05
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em ambientes distribuídos, a configuração pode incluir:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Cluster manager.&lt;/li&gt;
&lt;li&gt;Deploy mode.&lt;/li&gt;
&lt;li&gt;Recursos do Driver.&lt;/li&gt;
&lt;li&gt;Recursos dos Executors.&lt;/li&gt;
&lt;li&gt;Dependências.&lt;/li&gt;
&lt;li&gt;Configurações de rede.&lt;/li&gt;
&lt;li&gt;Segurança.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O Airflow pode coordenar a submissão.&lt;/p&gt;

&lt;p&gt;Mas o dimensionamento do Spark continua sendo uma responsabilidade de engenharia.&lt;/p&gt;

&lt;p&gt;Um DAG bem configurado não corrige automaticamente um Job com shuffle excessivo ou particionamento inadequado.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Data Interval e Logical Date
&lt;/h2&gt;

&lt;p&gt;Um erro comum é utilizar a data atual do servidor para determinar quais dados processar.&lt;/p&gt;

&lt;p&gt;Imagine uma DAG diária que deveria processar os dados de 5 de novembro.&lt;/p&gt;

&lt;p&gt;Por algum motivo, ela só foi executada em 7 de novembro.&lt;/p&gt;

&lt;p&gt;Se o código utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;

&lt;span class="n"&gt;processing_date&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;today&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;poderá processar o dia errado.&lt;/p&gt;

&lt;p&gt;O Airflow fornece informações de contexto sobre a execução e seu intervalo de dados.&lt;/p&gt;

&lt;p&gt;Para workflows agendados por intervalo, devemos definir claramente qual período cada execução representa.&lt;/p&gt;

&lt;p&gt;Em vez de depender do relógio da máquina, podemos passar parâmetros associados ao intervalo da DAG.&lt;/p&gt;

&lt;p&gt;Por exemplo, em uma tarefa com templates compatíveis:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{{ data_interval_start }}
{{ data_interval_end }}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A semântica exata depende do timetable utilizado.&lt;/p&gt;

&lt;p&gt;O princípio é simples:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A data lógica de processamento deve ser definida pelo contrato do pipeline, não pelo momento em que o código foi iniciado.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Retries
&lt;/h2&gt;

&lt;p&gt;Falhas temporárias acontecem.&lt;/p&gt;

&lt;p&gt;Uma API pode ficar indisponível.&lt;/p&gt;

&lt;p&gt;Uma conexão pode ser interrompida.&lt;/p&gt;

&lt;p&gt;Um serviço pode retornar erro temporário.&lt;/p&gt;

&lt;p&gt;O Airflow permite configurar tentativas adicionais.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;timedelta&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;airflow.sdk&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;


&lt;span class="nd"&gt;@task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;retry_delay&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;minutes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;extract_data&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Executando extração&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas retries precisam ser utilizados com cuidado.&lt;/p&gt;

&lt;p&gt;Repetir uma tarefa não garante que sua execução seja segura.&lt;/p&gt;

&lt;p&gt;Se a tarefa inserir dados novamente sem controle, poderá gerar duplicidades.&lt;/p&gt;

&lt;p&gt;Por isso, retries e idempotência devem ser projetados em conjunto.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. Idempotência
&lt;/h2&gt;

&lt;p&gt;Uma operação idempotente pode ser repetida sem produzir efeitos adicionais indevidos após a primeira aplicação bem-sucedida.&lt;/p&gt;

&lt;p&gt;Imagine uma tarefa que grava dados em uma partição diária.&lt;/p&gt;

&lt;p&gt;Uma abordagem potencialmente segura é substituir integralmente a partição correspondente à data lógica, desde que a operação de publicação seja implementada corretamente.&lt;/p&gt;

&lt;p&gt;Outra estratégia é utilizar uma tabela transacional com &lt;code&gt;MERGE&lt;/code&gt; e chaves de negócio.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Execution 1
     |
     v
Publish Partition 2026-11-05

Execution 2 (Retry)
     |
     v
Publish Same Logical Result
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O objetivo é que o retry não duplique registros.&lt;/p&gt;

&lt;p&gt;Entretanto, sobrescrever diretórios Parquet diretamente não garante atomicidade de publicação.&lt;/p&gt;

&lt;p&gt;Para produção, devemos considerar os recursos transacionais do destino.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. Backfill
&lt;/h2&gt;

&lt;p&gt;Um backfill é o processamento ou reprocessamento de períodos históricos.&lt;/p&gt;

&lt;p&gt;Imagine que um pipeline foi criado hoje, mas precisamos processar os últimos 90 dias.&lt;/p&gt;

&lt;p&gt;Ou que descobrimos um bug que afetou determinadas datas.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-11-01 → Reprocess
2026-11-02 → Reprocess
2026-11-03 → Reprocess
2026-11-04 → Reprocess
2026-11-05 → Reprocess
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Airflow oferece mecanismos de backfill e gerenciamento de execuções históricas, com comportamento dependente da versão e do timetable.&lt;/p&gt;

&lt;p&gt;Mas o pipeline precisa estar preparado.&lt;/p&gt;

&lt;p&gt;Um backfill confiável exige:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Parâmetros temporais explícitos.&lt;/li&gt;
&lt;li&gt;Idempotência.&lt;/li&gt;
&lt;li&gt;Controle de concorrência.&lt;/li&gt;
&lt;li&gt;Capacidade de processamento.&lt;/li&gt;
&lt;li&gt;Validação.&lt;/li&gt;
&lt;li&gt;Estratégia de publicação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Não basta disparar várias execuções antigas.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. Concorrência e dependências
&lt;/h2&gt;

&lt;p&gt;Imagine que dois DAG Runs tentam atualizar a mesma tabela.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DAG Run A ──&amp;gt; MERGE Orders
                    |
                    X
                    |
DAG Run B ──&amp;gt; MERGE Orders
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Dependendo do destino e da estratégia de escrita, podem ocorrer conflitos.&lt;/p&gt;

&lt;p&gt;O Airflow oferece controles como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Limites de DAG Runs ativos.&lt;/li&gt;
&lt;li&gt;Pools.&lt;/li&gt;
&lt;li&gt;Limites de tarefas.&lt;/li&gt;
&lt;li&gt;Dependências entre tarefas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nd"&gt;@dag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dag_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;controlled_pipeline&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;schedule&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;@daily&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;start_date&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2026&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;catchup&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;max_active_runs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;controlled_pipeline&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="bp"&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse limite reduz a concorrência entre execuções da mesma DAG.&lt;/p&gt;

&lt;p&gt;Mas não protege automaticamente contra escritores externos ou outras DAGs.&lt;/p&gt;

&lt;p&gt;A consistência do destino continua exigindo mecanismos adequados.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. Orquestração vs. Event-Driven Processing
&lt;/h2&gt;

&lt;p&gt;Nem todo pipeline precisa de um agendamento diário.&lt;/p&gt;

&lt;p&gt;Existem diferentes padrões.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Time-based&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Executar em intervalos definidos.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dependency-based&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Executar após a conclusão de outra tarefa.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data-aware&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Executar quando determinadas condições ou atualizações de dados estiverem disponíveis.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Event-driven&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Responder a eventos externos.&lt;/p&gt;

&lt;p&gt;O Airflow oferece recursos para workflows orientados a dados, incluindo Assets nas versões modernas.&lt;/p&gt;

&lt;p&gt;Mas devemos distinguir orquestração de workflows e processamento contínuo de eventos.&lt;/p&gt;

&lt;p&gt;Uma aplicação Spark Structured Streaming pode permanecer ativa continuamente.&lt;/p&gt;

&lt;p&gt;Nesse cenário, o Airflow pode coordenar seu deployment, verificações ou ciclo operacional, sem necessariamente iniciar um novo Job a cada evento.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. O que é DataOps?
&lt;/h2&gt;

&lt;p&gt;DataOps é uma abordagem que aplica princípios de automação, colaboração, testes, observabilidade e melhoria contínua ao ciclo de vida dos dados.&lt;/p&gt;

&lt;p&gt;Seu objetivo é aumentar a confiabilidade e a velocidade de entrega de produtos de dados.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;              DATAOPS LIFECYCLE

Plan
  |
  v
Develop
  |
  v
Test
  |
  v
Deploy
  |
  v
Operate
  |
  v
Monitor
  |
  v
Improve
  |
  +----------&amp;gt; Plan
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;DataOps não é uma ferramenta específica.&lt;/p&gt;

&lt;p&gt;É uma combinação de práticas, processos e responsabilidades.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. Versionamento com Git
&lt;/h2&gt;

&lt;p&gt;Um pipeline corporativo precisa de versionamento.&lt;/p&gt;

&lt;p&gt;Uma estrutura possível:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark-data-platform/
|
+-- dags/
|   +-- daily_sales.py
|
+-- src/
|   +-- pipelines/
|   |   +-- sales.py
|   |
|   +-- transformations/
|   |   +-- orders.py
|   |
|   +-- quality/
|       +-- validations.py
|
+-- tests/
|   +-- test_orders.py
|   +-- test_quality.py
|
+-- config/
|   +-- dev.yaml
|   +-- prod.yaml
|
+-- pyproject.toml
+-- README.md
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa separação evita concentrar toda a lógica dentro dos arquivos DAG.&lt;/p&gt;

&lt;p&gt;O ideal é manter a DAG como camada de orquestração.&lt;/p&gt;

&lt;p&gt;A lógica de negócio e as transformações devem ser implementadas em módulos testáveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Evitando lógica pesada no parse da DAG
&lt;/h2&gt;

&lt;p&gt;O Airflow precisa carregar e interpretar os arquivos das DAGs.&lt;/p&gt;

&lt;p&gt;Por isso, não devemos executar operações pesadas durante a importação desses arquivos.&lt;/p&gt;

&lt;p&gt;Evite:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Exemplo do que NÃO fazer no parse da DAG
&lt;/span&gt;
&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/orders&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode provocar processamento inesperado sempre que o arquivo for interpretado.&lt;/p&gt;

&lt;p&gt;O correto é definir tarefas que executem o trabalho no momento apropriado.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DAG Definition
      |
      v
Task Scheduling
      |
      v
Task Execution
      |
      v
Spark Processing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa separação melhora previsibilidade e manutenção.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. Testes unitários em PySpark
&lt;/h2&gt;

&lt;p&gt;Vamos criar uma transformação simples.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;functions&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;calcular_valor_liquido&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor_liquido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor_bruto&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;desconto&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora podemos testá-la.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;test_calcular_valor_liquido&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;entrada&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createDataFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;100.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;10.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;200.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;20.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor_bruto&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;desconto&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;calcular_valor_liquido&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;entrada&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;valores&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor_liquido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;valores&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;90.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;180.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse teste pressupõe uma fixture &lt;code&gt;spark&lt;/code&gt; configurada no pytest.&lt;/p&gt;

&lt;p&gt;A vantagem é validar a transformação sem executar toda a arquitetura.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. Fixture Spark para pytest
&lt;/h2&gt;

&lt;p&gt;Podemos criar uma fixture reutilizável.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pytest&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;


&lt;span class="nd"&gt;@pytest.fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scope&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;session&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;master&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;local[2]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PySparkTests&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.ui.enabled&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;false&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;session&lt;/span&gt;

    &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa fixture pode ser colocada em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tests/conftest.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Assim, diferentes testes podem compartilhar uma SparkSession durante a sessão de testes.&lt;/p&gt;

&lt;p&gt;Em ambientes de CI, precisamos garantir compatibilidade entre Python, Java e Spark.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Testes de integração
&lt;/h2&gt;

&lt;p&gt;Testes unitários verificam funções isoladas.&lt;/p&gt;

&lt;p&gt;Mas precisamos também testar a integração entre componentes.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Read Input
    |
    v
Transform
    |
    v
Validate
    |
    v
Write Output
    |
    v
Read Output
    |
    v
Assert Results
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Um teste de integração pode utilizar datasets pequenos e diretórios temporários.&lt;/p&gt;

&lt;p&gt;O objetivo é validar o comportamento do pipeline completo em um ambiente controlado.&lt;/p&gt;

&lt;p&gt;Para destinos transacionais, também devemos testar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;MERGE.&lt;/li&gt;
&lt;li&gt;Reprocessamento.&lt;/li&gt;
&lt;li&gt;Idempotência.&lt;/li&gt;
&lt;li&gt;Evolução de schema.&lt;/li&gt;
&lt;li&gt;Falhas.&lt;/li&gt;
&lt;li&gt;Recuperação.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  21. Data Quality no CI/CD
&lt;/h2&gt;

&lt;p&gt;Na Parte 14, estudamos Data Quality e Data Contracts.&lt;/p&gt;

&lt;p&gt;Agora podemos integrar esses conceitos ao processo de entrega.&lt;/p&gt;

&lt;p&gt;Imagine um Pull Request que modifica uma transformação.&lt;/p&gt;

&lt;p&gt;O pipeline de CI pode executar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Pull Request
     |
     v
Code Formatting
     |
     v
Static Analysis
     |
     v
Unit Tests
     |
     v
Data Contract Tests
     |
     v
Integration Tests
     |
     v
Build Validation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso ajuda a identificar problemas antes do deploy.&lt;/p&gt;

&lt;p&gt;Mas testes com dados sintéticos não substituem verificações de qualidade em produção.&lt;/p&gt;

&lt;p&gt;Precisamos de validação antes e depois da implantação.&lt;/p&gt;




&lt;h2&gt;
  
  
  22. CI/CD para pipelines Spark
&lt;/h2&gt;

&lt;p&gt;CI/CD significa:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Continuous Integration&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Integração frequente de alterações com validações automatizadas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Continuous Delivery / Deployment&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Automação da preparação e, quando apropriado, da publicação de novas versões.&lt;/p&gt;

&lt;p&gt;Uma arquitetura conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Developer
    |
    v
Git Repository
    |
    v
Pull Request
    |
    v
CI Pipeline
    |
    +-- Lint
    +-- Tests
    +-- Build
    |
    v
Artifact
    |
    v
Deployment
    |
    v
Airflow / Spark
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O objetivo é tornar o processo repetível e auditável.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. Exemplo de GitHub Actions
&lt;/h2&gt;

&lt;p&gt;Podemos utilizar GitHub Actions para executar testes PySpark.&lt;/p&gt;

&lt;p&gt;Exemplo simplificado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Spark CI&lt;/span&gt;

&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;push&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;branches&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;main&lt;/span&gt;
  &lt;span class="na"&gt;pull_request&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;

&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;tests&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;

    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Checkout&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v4&lt;/span&gt;

      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Setup Python&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/setup-python@v5&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;python-version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;3.11"&lt;/span&gt;

      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Setup Java&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/setup-java@v4&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;distribution&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;temurin&lt;/span&gt;
          &lt;span class="na"&gt;java-version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;17"&lt;/span&gt;

      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Install dependencies&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;python -m pip install --upgrade pip&lt;/span&gt;
          &lt;span class="s"&gt;pip install pyspark==3.5.5 pytest&lt;/span&gt;

      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Run tests&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;pytest -q&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse exemplo utiliza uma combinação de versões para fins didáticos.&lt;/p&gt;

&lt;p&gt;Em um projeto real, devemos fixar e validar a matriz de compatibilidade, preferencialmente por arquivos de dependências e imagens versionadas.&lt;/p&gt;

&lt;p&gt;Também é importante revisar periodicamente as versões das Actions e as políticas de segurança da organização.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Build once, deploy many
&lt;/h2&gt;

&lt;p&gt;Uma boa prática é construir um artefato versionado e promovê-lo entre ambientes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source Code
     |
     v
Build Artifact
     |
     +----&amp;gt; Development
     |
     +----&amp;gt; Staging
     |
     +----&amp;gt; Production
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso reduz diferenças entre ambientes.&lt;/p&gt;

&lt;p&gt;O mesmo código pode receber configurações específicas para:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Credenciais.&lt;/li&gt;
&lt;li&gt;Endpoints.&lt;/li&gt;
&lt;li&gt;Buckets.&lt;/li&gt;
&lt;li&gt;Catálogos.&lt;/li&gt;
&lt;li&gt;Recursos.&lt;/li&gt;
&lt;li&gt;Logs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A configuração deve ser externa ao artefato sempre que apropriado.&lt;/p&gt;

&lt;p&gt;Não devemos incorporar senhas ou tokens diretamente no código.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. Ambientes Dev, Staging e Prod
&lt;/h2&gt;

&lt;p&gt;Uma plataforma madura normalmente separa ambientes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Development&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Utilizado para desenvolvimento e experimentação.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Staging&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Utilizado para validações de integração e testes próximos da produção.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Production&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Utilizado para cargas reais e consumidores finais.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DEV
 |
 v
TEST
 |
 v
STAGING
 |
 v
PRODUCTION
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A quantidade de ambientes depende da organização.&lt;/p&gt;

&lt;p&gt;Mas precisamos controlar diferenças de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Configuração.&lt;/li&gt;
&lt;li&gt;Dados.&lt;/li&gt;
&lt;li&gt;Permissões.&lt;/li&gt;
&lt;li&gt;Capacidade.&lt;/li&gt;
&lt;li&gt;Dependências.&lt;/li&gt;
&lt;li&gt;Versões.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Um pipeline que funciona localmente não está automaticamente pronto para produção.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. Configuração por ambiente
&lt;/h2&gt;

&lt;p&gt;Evite utilizar caminhos fixos no código.&lt;/p&gt;

&lt;p&gt;Exemplo inadequado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/home/user/projeto/output&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Uma alternativa é receber parâmetros.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;argparse&lt;/span&gt;

&lt;span class="n"&gt;parser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;argparse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ArgumentParser&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--processing-date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--output-path&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse_args&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora o pipeline pode ser executado com diferentes configurações.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;spark-submit &lt;span class="se"&gt;\&lt;/span&gt;
  process_sales.py &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--processing-date&lt;/span&gt; 2026-11-05 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--output-path&lt;/span&gt; /data/curated/sales
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em produção, os parâmetros devem ser validados antes do processamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Secrets Management
&lt;/h2&gt;

&lt;p&gt;Credenciais não devem ficar armazenadas em arquivos versionados.&lt;/p&gt;

&lt;p&gt;Evite:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;password&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;minha_senha&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Prefira mecanismos apropriados ao ambiente, como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Secret managers.&lt;/li&gt;
&lt;li&gt;Backends de secrets do Airflow.&lt;/li&gt;
&lt;li&gt;Identidades de workload.&lt;/li&gt;
&lt;li&gt;Credenciais temporárias.&lt;/li&gt;
&lt;li&gt;Variáveis de ambiente controladas, quando apropriado.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O objetivo é reduzir exposição e facilitar rotação.&lt;/p&gt;

&lt;p&gt;Também devemos evitar registrar secrets em logs, exceções ou parâmetros de linha de comando quando isso puder expô-los.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Observabilidade de pipelines
&lt;/h2&gt;

&lt;p&gt;Uma DAG verde não significa necessariamente que os dados estão corretos.&lt;/p&gt;

&lt;p&gt;Precisamos acompanhar três dimensões.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Orchestration Health&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Estados das tarefas.&lt;/li&gt;
&lt;li&gt;Retries.&lt;/li&gt;
&lt;li&gt;Falhas.&lt;/li&gt;
&lt;li&gt;Duração.&lt;/li&gt;
&lt;li&gt;Atrasos de agendamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Processing Health&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Duração do Spark Job.&lt;/li&gt;
&lt;li&gt;Shuffle.&lt;/li&gt;
&lt;li&gt;Spill.&lt;/li&gt;
&lt;li&gt;Memória.&lt;/li&gt;
&lt;li&gt;CPU.&lt;/li&gt;
&lt;li&gt;Falhas de Executors.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Data Health&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Freshness.&lt;/li&gt;
&lt;li&gt;Volume.&lt;/li&gt;
&lt;li&gt;Completude.&lt;/li&gt;
&lt;li&gt;Duplicidade.&lt;/li&gt;
&lt;li&gt;Contratos.&lt;/li&gt;
&lt;li&gt;Quality Gates.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 OBSERVABILITY
                       |
           +-----------+-----------+
           |           |           |
           v           v           v
        AIRFLOW       SPARK       DATA
         HEALTH       HEALTH      QUALITY
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essas informações precisam ser correlacionadas.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. Logs estruturados
&lt;/h2&gt;

&lt;p&gt;Logs são fundamentais para investigação.&lt;/p&gt;

&lt;p&gt;Mas mensagens genéricas como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Process failed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;oferecem pouco contexto.&lt;/p&gt;

&lt;p&gt;Uma mensagem estruturada pode incluir:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pipeline"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"daily_sales"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"run_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"scheduled__example"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"processing_date"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-11-05"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"stage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"transform"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"failed"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"error_type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"DataQualityError"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em produção, os valores devem ser obtidos do contexto real da execução.&lt;/p&gt;

&lt;p&gt;Logs estruturados facilitam busca, correlação e análise de incidentes.&lt;/p&gt;

&lt;p&gt;Também devemos evitar registrar dados sensíveis desnecessariamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. Alertas e incidentes
&lt;/h2&gt;

&lt;p&gt;Imagine que um pipeline crítico falhe às 2h da manhã.&lt;/p&gt;

&lt;p&gt;Precisamos definir:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Quem recebe o alerta.&lt;/li&gt;
&lt;li&gt;Qual é a criticidade.&lt;/li&gt;
&lt;li&gt;Qual é o SLA.&lt;/li&gt;
&lt;li&gt;Como investigar.&lt;/li&gt;
&lt;li&gt;Como recuperar.&lt;/li&gt;
&lt;li&gt;Quando escalar.&lt;/li&gt;
&lt;li&gt;Como comunicar consumidores.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma estratégia pode classificar incidentes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WARNING
   |
   v
Non-critical Degradation

ERROR
   |
   v
Pipeline Failure

CRITICAL
   |
   v
Business Data Unavailable
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A classificação deve refletir o impacto real.&lt;/p&gt;

&lt;p&gt;Nem toda falha técnica exige o mesmo nível de resposta.&lt;/p&gt;




&lt;h2&gt;
  
  
  31. SLO e SLA
&lt;/h2&gt;

&lt;p&gt;Dois conceitos aparecem frequentemente em operação.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SLO — Service Level Objective&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Objetivo interno de desempenho ou confiabilidade.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SLA — Service Level Agreement&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Compromisso formal de serviço, quando aplicável.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Dataset: daily_sales

SLO:
99% das publicações concluídas
até 07:00.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos acompanhar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Disponibilidade.&lt;/li&gt;
&lt;li&gt;Freshness.&lt;/li&gt;
&lt;li&gt;Tempo de processamento.&lt;/li&gt;
&lt;li&gt;Taxa de falhas.&lt;/li&gt;
&lt;li&gt;Tempo de recuperação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O importante é definir indicadores verificáveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Backfill seguro
&lt;/h2&gt;

&lt;p&gt;Vamos imaginar que precisamos reprocessar sete dias.&lt;/p&gt;

&lt;p&gt;Uma abordagem insegura seria disparar todas as execuções sem avaliar concorrência.&lt;/p&gt;

&lt;p&gt;Uma abordagem controlada considera:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Validate Historical Range
          |
          v
Estimate Processing Cost
          |
          v
Check Source Availability
          |
          v
Control Concurrency
          |
          v
Execute Backfill
          |
          v
Validate Results
          |
          v
Publish
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Também devemos considerar que os dados de origem podem ter mudado desde a execução original.&lt;/p&gt;

&lt;p&gt;Reprocessar uma data histórica não garante reproduzir exatamente o mesmo resultado se as fontes não forem versionadas ou preservadas.&lt;/p&gt;

&lt;p&gt;Essa é uma razão importante para manter histórico e snapshots quando necessário.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. Deploy e rollback
&lt;/h2&gt;

&lt;p&gt;Imagine que uma nova versão do pipeline produz resultados incorretos.&lt;/p&gt;

&lt;p&gt;Precisamos ter uma estratégia de recuperação.&lt;/p&gt;

&lt;p&gt;Isso pode envolver:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Reverter o código.&lt;/li&gt;
&lt;li&gt;Reimplantar o artefato anterior.&lt;/li&gt;
&lt;li&gt;Recuperar uma versão de tabela.&lt;/li&gt;
&lt;li&gt;Reprocessar períodos afetados.&lt;/li&gt;
&lt;li&gt;Validar o resultado.&lt;/li&gt;
&lt;li&gt;Comunicar consumidores.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas rollback de código e rollback de dados são operações diferentes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CODE ROLLBACK
      |
      v
Restore Previous Application Version
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DATA RECOVERY
      |
      v
Restore / Rebuild Correct Dataset
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Uma plataforma confiável precisa planejar ambos.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Orquestração de pipelines streaming
&lt;/h2&gt;

&lt;p&gt;Nem todo pipeline Spark termina após processar um intervalo.&lt;/p&gt;

&lt;p&gt;Uma aplicação Structured Streaming pode permanecer ativa por longos períodos.&lt;/p&gt;

&lt;p&gt;Nesse cenário, precisamos definir:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Como iniciar a aplicação.&lt;/li&gt;
&lt;li&gt;Como detectar falhas.&lt;/li&gt;
&lt;li&gt;Como reiniciar.&lt;/li&gt;
&lt;li&gt;Como preservar checkpoints.&lt;/li&gt;
&lt;li&gt;Como atualizar versões.&lt;/li&gt;
&lt;li&gt;Como realizar shutdown controlado.&lt;/li&gt;
&lt;li&gt;Como monitorar lag e estado.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Airflow pode participar dessa arquitetura, mas não precisa ser o supervisor principal de um serviço streaming de longa duração.&lt;/p&gt;

&lt;p&gt;Em determinados ambientes, Kubernetes ou outro mecanismo de gerenciamento de aplicações pode assumir essa responsabilidade.&lt;/p&gt;

&lt;p&gt;A decisão depende do modelo operacional.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. Arquitetura corporativa de referência
&lt;/h2&gt;

&lt;p&gt;Vamos conectar tudo o que estudamos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                    DATA SOURCES
                         |
                         v
                      INGESTION
                         |
                         v
                    RAW STORAGE
                         |
                         v
                  SPARK PROCESSING
                         |
                         v
                  DATA QUALITY
                         |
                         v
                  LAKEHOUSE TABLES
                         |
                         v
                    ANALYTICS


              ORCHESTRATION LAYER
                       |
                       v
                    AIRFLOW


                 DELIVERY LAYER
                       |
                       v
                  GIT + CI/CD


               GOVERNANCE LAYER
                       |
                       v
             CONTRACTS + LINEAGE


              OBSERVABILITY LAYER
                       |
                       v
              LOGS + METRICS + ALERTS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As camadas de orquestração, entrega, governança e observabilidade atuam transversalmente sobre os pipelines.&lt;/p&gt;

&lt;p&gt;Essa arquitetura separa responsabilidades e facilita manutenção.&lt;/p&gt;




&lt;h2&gt;
  
  
  36. Checklist para produção
&lt;/h2&gt;

&lt;p&gt;Antes de colocar um pipeline Spark em produção, eu verificaria:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Área&lt;/th&gt;
&lt;th&gt;Verificação&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Código&lt;/td&gt;
&lt;td&gt;Versionado e revisado&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Testes&lt;/td&gt;
&lt;td&gt;Unitários e integração&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Contratos&lt;/td&gt;
&lt;td&gt;Schemas e regras definidos&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qualidade&lt;/td&gt;
&lt;td&gt;Quality Gates configurados&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Orquestração&lt;/td&gt;
&lt;td&gt;Dependências explícitas&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retries&lt;/td&gt;
&lt;td&gt;Seguros e controlados&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Idempotência&lt;/td&gt;
&lt;td&gt;Reprocessamento validado&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Backfill&lt;/td&gt;
&lt;td&gt;Estratégia documentada&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Concorrência&lt;/td&gt;
&lt;td&gt;Limites definidos&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Segurança&lt;/td&gt;
&lt;td&gt;Secrets protegidos&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Deploy&lt;/td&gt;
&lt;td&gt;Artefatos versionados&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Observabilidade&lt;/td&gt;
&lt;td&gt;Logs, métricas e alertas&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recuperação&lt;/td&gt;
&lt;td&gt;Procedimento testado&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Custos&lt;/td&gt;
&lt;td&gt;Recursos e execução monitorados&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ownership&lt;/td&gt;
&lt;td&gt;Responsáveis definidos&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Esse checklist não substitui uma avaliação de arquitetura.&lt;/p&gt;

&lt;p&gt;Mas ajuda a identificar lacunas antes do deployment.&lt;/p&gt;




&lt;h2&gt;
  
  
  37. Erros comuns
&lt;/h2&gt;

&lt;p&gt;Alguns erros aparecem repetidamente.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Colocar toda a lógica dentro da DAG.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso dificulta testes e manutenção.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Executar Spark durante o parse da DAG.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode gerar processamento inesperado.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Utilizar &lt;code&gt;date.today()&lt;/code&gt; como referência de processamento.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode quebrar backfills e retries.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Configurar retries sem idempotência.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode produzir duplicidades.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não controlar concorrência.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode causar conflitos no destino.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não versionar dependências.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode produzir comportamentos diferentes entre ambientes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não separar código de configuração.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Dificulta deploy e manutenção.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tratar DAG verde como garantia de qualidade.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Dados incorretos podem ser publicados.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não planejar rollback de dados.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Reverter código pode não corrigir resultados já gravados.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não monitorar custos.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pipelines podem continuar funcionando enquanto se tornam economicamente inviáveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  38. Como eu estruturaria um projeto Spark + Airflow
&lt;/h2&gt;

&lt;p&gt;Para uma plataforma corporativa, começaria com uma organização simples.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;data-platform/
|
+-- airflow/
|   +-- dags/
|
+-- spark/
|   +-- jobs/
|   +-- transformations/
|   +-- quality/
|
+-- contracts/
|
+-- tests/
|   +-- unit/
|   +-- integration/
|
+-- configs/
|   +-- dev/
|   +-- staging/
|   +-- prod/
|
+-- infrastructure/
|
+-- observability/
|
+-- docs/
|
+-- .github/
|   +-- workflows/
|
+-- pyproject.toml
+-- README.md
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa estrutura não é obrigatória.&lt;/p&gt;

&lt;p&gt;Ela representa uma forma de separar responsabilidades.&lt;/p&gt;

&lt;p&gt;O objetivo é permitir que código, infraestrutura, contratos e operação evoluam de maneira controlada.&lt;/p&gt;




&lt;h2&gt;
  
  
  39. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;Até aqui, aprendemos como o Spark processa dados.&lt;/p&gt;

&lt;p&gt;Agora entendemos que um pipeline corporativo precisa de um ciclo de vida completo.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DEVELOP
   |
   v
TEST
   |
   v
VERSION
   |
   v
DEPLOY
   |
   v
ORCHESTRATE
   |
   v
MONITOR
   |
   v
RECOVER
   |
   v
IMPROVE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O princípio mais importante é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Um Job Spark que funciona é apenas o começo. Engenharia de Dados em produção exige automação, testes, rastreabilidade, observabilidade e recuperação.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Airflow ajuda a coordenar workflows.&lt;/p&gt;

&lt;p&gt;CI/CD ajuda a entregar alterações com segurança.&lt;/p&gt;

&lt;p&gt;DataOps conecta essas práticas ao ciclo de vida dos dados.&lt;/p&gt;




&lt;h2&gt;
  
  
  40. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando avalio um pipeline Spark, não pergunto apenas:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;O processamento terminou?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Quero saber:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qual versão do código foi executada?&lt;/li&gt;
&lt;li&gt;Qual intervalo de dados foi processado?&lt;/li&gt;
&lt;li&gt;Quais contratos foram validados?&lt;/li&gt;
&lt;li&gt;O resultado passou pelos Quality Gates?&lt;/li&gt;
&lt;li&gt;Como o pipeline reage a falhas?&lt;/li&gt;
&lt;li&gt;Os retries são seguros?&lt;/li&gt;
&lt;li&gt;Como executamos backfills?&lt;/li&gt;
&lt;li&gt;Como controlamos concorrência?&lt;/li&gt;
&lt;li&gt;Como monitoramos performance?&lt;/li&gt;
&lt;li&gt;Como recuperamos uma versão incorreta?&lt;/li&gt;
&lt;li&gt;Quanto custa manter esse pipeline?&lt;/li&gt;
&lt;li&gt;Quem é responsável por ele?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Essas perguntas representam a diferença entre um script funcional e um produto de dados operável.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora que entendemos como orquestrar, testar e operar pipelines Spark, vamos aprofundar um dos temas mais importantes de plataformas corporativas:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Performance Tuning e otimização avançada de Apache Spark.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Vamos explorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Catalyst Optimizer.&lt;/li&gt;
&lt;li&gt;Tungsten.&lt;/li&gt;
&lt;li&gt;Adaptive Query Execution.&lt;/li&gt;
&lt;li&gt;Query Plans.&lt;/li&gt;
&lt;li&gt;Broadcast Joins.&lt;/li&gt;
&lt;li&gt;Skew Join Optimization.&lt;/li&gt;
&lt;li&gt;Shuffle Partitions.&lt;/li&gt;
&lt;li&gt;Dynamic Partition Pruning.&lt;/li&gt;
&lt;li&gt;Caching.&lt;/li&gt;
&lt;li&gt;Memory Tuning.&lt;/li&gt;
&lt;li&gt;Spark UI.&lt;/li&gt;
&lt;li&gt;Métricas de execução.&lt;/li&gt;
&lt;li&gt;Benchmarking.&lt;/li&gt;
&lt;li&gt;Estratégias de otimização em produção.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central será:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como identificar gargalos reais e otimizar aplicações Spark utilizando evidências, planos de execução e métricas?&lt;/strong&gt;&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 16:&lt;/strong&gt; Apache Kafka, CDC e pipelines incrementais&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 18 — Performance Tuning avançado: Catalyst, AQE, Spark UI e otimização de Jobs&lt;/p&gt;

</description>
      <category>python</category>
      <category>sql</category>
      <category>bigdata</category>
      <category>spark</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 16: Apache Kafka, CDC e pipelines incrementais</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 20:36:02 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-16-apache-kafka-cdc-e-pipelines-incrementais-2bjn</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-16-apache-kafka-cdc-e-pipelines-incrementais-2bjn</guid>
      <description>&lt;h1&gt;
  
  
  Como aprendi Apache Spark — Parte 16: Apache Kafka, CDC e pipelines incrementais
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Nota histórica: esta série revisita minha trajetória em Engenharia de Dados iniciada em 2012. Tecnologias como Structured Streaming, Debezium e os formatos Lakehouse modernos surgiram ou amadureceram posteriormente. Este capítulo conecta aquela experiência aos padrões atuais de integração de dados.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nos capítulos anteriores, construímos uma visão progressiva do Apache Spark.&lt;/p&gt;

&lt;p&gt;Começamos pela execução distribuída, passamos por partições, shuffle, memória, otimização de consultas, armazenamento colunar, Lakehouse e qualidade de dados.&lt;/p&gt;

&lt;p&gt;Na Parte 15, entramos no universo do Structured Streaming.&lt;/p&gt;

&lt;p&gt;Aprendemos como o Spark processa fluxos de eventos utilizando microbatches, checkpoints, watermarks e gerenciamento de estado.&lt;/p&gt;

&lt;p&gt;Agora vamos responder a uma pergunta bastante comum em projetos corporativos:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como manter uma plataforma analítica atualizada quando os dados estão mudando continuamente em sistemas transacionais?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Imagine um e-commerce.&lt;/p&gt;

&lt;p&gt;Seu banco PostgreSQL recebe milhares de operações:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Novos pedidos.&lt;/li&gt;
&lt;li&gt;Alterações de status.&lt;/li&gt;
&lt;li&gt;Atualizações cadastrais.&lt;/li&gt;
&lt;li&gt;Cancelamentos.&lt;/li&gt;
&lt;li&gt;Exclusões.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma estratégia tradicional seria executar periodicamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pedidos&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas, conforme a tabela cresce, reler todos os registros se torna cada vez mais caro.&lt;/p&gt;

&lt;p&gt;Além disso, essa abordagem pode não identificar corretamente exclusões ou alterações intermediárias.&lt;/p&gt;

&lt;p&gt;Precisamos de um mecanismo que permita capturar as mudanças.&lt;/p&gt;

&lt;p&gt;É aqui que entram &lt;strong&gt;Change Data Capture, Apache Kafka e Spark Structured Streaming&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Full Load vs. Incremental Load
&lt;/h2&gt;

&lt;p&gt;Vamos começar pelo problema.&lt;/p&gt;

&lt;p&gt;Imagine uma tabela com 100 milhões de registros.&lt;/p&gt;

&lt;p&gt;Todos os dias, apenas 200 mil registros são inseridos ou alterados.&lt;/p&gt;

&lt;p&gt;Uma carga completa processaria:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100,000,000 records
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Uma carga incremental poderia processar aproximadamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;200,000 changed records
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A diferença potencial é significativa.&lt;/p&gt;

&lt;p&gt;Mas precisamos entender como identificar as alterações.&lt;/p&gt;

&lt;p&gt;Uma abordagem comum utiliza uma coluna como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;updated_at
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pedidos&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;updated_at&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;ultimo_processamento&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse mecanismo pode funcionar em determinados sistemas.&lt;/p&gt;

&lt;p&gt;Entretanto, possui limitações:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Depende da atualização correta do timestamp.&lt;/li&gt;
&lt;li&gt;Pode perder exclusões físicas.&lt;/li&gt;
&lt;li&gt;Exige tratamento cuidadoso de empates e limites temporais.&lt;/li&gt;
&lt;li&gt;Pode perder alterações se o controle incremental for implementado incorretamente.&lt;/li&gt;
&lt;li&gt;Pode exigir consultas frequentes ao banco operacional.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma alternativa é capturar mudanças a partir do log transacional.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. O que é Change Data Capture?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Change Data Capture (CDC)&lt;/strong&gt; é um conjunto de técnicas para identificar alterações realizadas em uma fonte de dados.&lt;/p&gt;

&lt;p&gt;Essas alterações podem incluir:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;INSERT
UPDATE
DELETE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em vez de consultar repetidamente a tabela inteira, um mecanismo CDC pode observar as mudanças registradas pelo banco.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;              POSTGRESQL
                  |
                  v
           TRANSACTION LOG
                  |
                  v
              CDC ENGINE
                  |
                  v
             CHANGE EVENTS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Existem diferentes implementações de CDC.&lt;/p&gt;

&lt;p&gt;Algumas utilizam consultas periódicas.&lt;/p&gt;

&lt;p&gt;Outras utilizam logs de transação.&lt;/p&gt;

&lt;p&gt;Neste artigo, vamos concentrar nossa atenção em CDC baseado em logs, especialmente com PostgreSQL e Debezium.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. PostgreSQL e Write-Ahead Log
&lt;/h2&gt;

&lt;p&gt;O PostgreSQL utiliza um mecanismo chamado &lt;strong&gt;Write-Ahead Logging (WAL)&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;O WAL registra informações necessárias à recuperação e à durabilidade das transações.&lt;/p&gt;

&lt;p&gt;Ele também pode ser utilizado, por meio de decodificação lógica, para disponibilizar alterações a sistemas externos.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SQL TRANSACTION
      |
      v
POSTGRESQL
      |
      v
     WAL
      |
      v
LOGICAL DECODING
      |
      v
CHANGE EVENTS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A captura lógica exige configurações e permissões apropriadas.&lt;/p&gt;

&lt;p&gt;No PostgreSQL, isso envolve recursos como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Logical Replication.&lt;/li&gt;
&lt;li&gt;Replication Slots.&lt;/li&gt;
&lt;li&gt;Publications.&lt;/li&gt;
&lt;li&gt;Configurações de WAL.&lt;/li&gt;
&lt;li&gt;Controle de retenção.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma preocupação operacional importante é que replication slots podem reter WAL quando o consumidor não avança.&lt;/p&gt;

&lt;p&gt;Se esse comportamento não for monitorado, o armazenamento do banco pode crescer significativamente.&lt;/p&gt;

&lt;p&gt;Portanto, CDC não é apenas uma configuração de integração.&lt;/p&gt;

&lt;p&gt;É também uma responsabilidade operacional.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. O papel do Debezium
&lt;/h2&gt;

&lt;p&gt;O &lt;strong&gt;Debezium&lt;/strong&gt; é uma plataforma de captura de alterações de bancos de dados.&lt;/p&gt;

&lt;p&gt;Uma arquitetura comum utiliza Debezium com Kafka Connect.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  POSTGRESQL
                      |
                      v
                 LOGICAL WAL
                      |
                      v
                   DEBEZIUM
                      |
                      v
                 KAFKA CONNECT
                      |
                      v
                  KAFKA TOPIC
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Debezium interpreta alterações da origem e publica eventos estruturados.&lt;/p&gt;

&lt;p&gt;Dependendo da configuração, também pode executar um snapshot inicial dos dados antes de acompanhar as mudanças contínuas.&lt;/p&gt;

&lt;p&gt;Isso é importante.&lt;/p&gt;

&lt;p&gt;Um novo pipeline CDC frequentemente precisa carregar o estado existente da tabela e depois continuar aplicando alterações.&lt;/p&gt;

&lt;p&gt;Precisamos garantir que essa transição seja consistente.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. O que é Apache Kafka?
&lt;/h2&gt;

&lt;p&gt;O Apache Kafka é uma plataforma distribuída de streaming de eventos.&lt;/p&gt;

&lt;p&gt;Ele permite que produtores publiquem registros e consumidores os leiam.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PRODUCERS
    |
    v
 KAFKA
    |
    +------ CONSUMER A
    |
    +------ CONSUMER B
    |
    +------ CONSUMER C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Kafka oferece mecanismos para:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Persistência de eventos.&lt;/li&gt;
&lt;li&gt;Organização em tópicos.&lt;/li&gt;
&lt;li&gt;Particionamento.&lt;/li&gt;
&lt;li&gt;Replicação.&lt;/li&gt;
&lt;li&gt;Leitura por offsets.&lt;/li&gt;
&lt;li&gt;Consumo independente por diferentes aplicações.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas Kafka não é simplesmente uma fila tradicional.&lt;/p&gt;

&lt;p&gt;Seu modelo permite que consumidores acompanhem posições no log e, dentro das condições de retenção, releiam eventos.&lt;/p&gt;

&lt;p&gt;Essa característica é especialmente útil em arquiteturas de dados.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Topics, Partitions e Offsets
&lt;/h2&gt;

&lt;p&gt;Um tópico Kafka pode ser dividido em partições.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TOPIC: pedidos

Partition 0
[0] [1] [2] [3] [4]

Partition 1
[0] [1] [2] [3]

Partition 2
[0] [1] [2] [3] [4] [5]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cada partição possui sua própria sequência de offsets.&lt;/p&gt;

&lt;p&gt;O offset identifica uma posição dentro daquela partição.&lt;/p&gt;

&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;partition = 0
offset = 10
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;partition = 1
offset = 10
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;representam posições diferentes.&lt;/p&gt;

&lt;p&gt;Uma propriedade importante é que Kafka preserva a ordem dos registros &lt;strong&gt;dentro de uma mesma partição&lt;/strong&gt;, não uma ordem global entre todas as partições.&lt;/p&gt;

&lt;p&gt;Essa distinção influencia diretamente o processamento CDC.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. A importância da chave de particionamento
&lt;/h2&gt;

&lt;p&gt;Imagine eventos de um pedido:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Pedido 101 → CRIADO
Pedido 101 → PAGO
Pedido 101 → ENVIADO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Queremos preservar a sequência de alterações desse pedido.&lt;/p&gt;

&lt;p&gt;Uma estratégia comum é utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pedido_id
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;como chave Kafka.&lt;/p&gt;

&lt;p&gt;Isso normalmente faz com que eventos com a mesma chave sejam encaminhados à mesma partição, considerando um particionador e uma configuração estáveis.&lt;/p&gt;

&lt;p&gt;Mas existem ressalvas.&lt;/p&gt;

&lt;p&gt;Mudanças no número de partições, no particionador ou na estratégia de chave podem afetar essa propriedade ao longo do tempo.&lt;/p&gt;

&lt;p&gt;Além disso, a ordem do Kafka não substitui uma regra de versionamento dos eventos no destino.&lt;/p&gt;

&lt;p&gt;Em pipelines CDC, precisamos considerar tanto a ordem de transporte quanto a ordem lógica das alterações na origem.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Consumer Groups
&lt;/h2&gt;

&lt;p&gt;Kafka permite organizar consumidores em grupos.&lt;/p&gt;

&lt;p&gt;Dentro de um consumer group, as partições são distribuídas entre os consumidores participantes.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TOPIC: pedidos
     |
     +-- Partition 0 --&amp;gt; Consumer A
     |
     +-- Partition 1 --&amp;gt; Consumer B
     |
     +-- Partition 2 --&amp;gt; Consumer C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso permite paralelismo.&lt;/p&gt;

&lt;p&gt;Entretanto, o número de partições limita o paralelismo efetivo de leitura por consumidores de um mesmo grupo.&lt;/p&gt;

&lt;p&gt;Se um tópico possui três partições, adicionar dez consumidores ao mesmo grupo não significa que todos receberão partições ativas simultaneamente.&lt;/p&gt;

&lt;p&gt;Também precisamos distinguir esse modelo da integração Spark.&lt;/p&gt;

&lt;p&gt;O conector Kafka do Structured Streaming gerencia a leitura e o progresso da consulta utilizando seus próprios mecanismos de offsets e checkpoints.&lt;/p&gt;

&lt;p&gt;Não devemos assumir que o gerenciamento de offsets de uma aplicação Spark seja idêntico ao de um consumidor Kafka convencional.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Como é um evento CDC?
&lt;/h2&gt;

&lt;p&gt;Vamos considerar um evento simplificado inspirado na estrutura de envelopes Debezium.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"before"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"pedido_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"CRIADO"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"valor"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;150.00&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"after"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"pedido_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PAGO"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"valor"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;150.00&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"op"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"u"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"ts_ms"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1793500000000&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O evento possui:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;before&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Estado anterior do registro, quando disponível.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;after&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Estado posterior do registro, quando aplicável.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;op&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Tipo de operação.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;ts_ms&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Timestamp associado ao evento, conforme a semântica do campo.&lt;/p&gt;

&lt;p&gt;Em envelopes Debezium reais, também podemos encontrar informações adicionais sobre origem, transação e snapshots.&lt;/p&gt;

&lt;p&gt;Os campos exatos dependem do conector, versão e configuração.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Operações CDC
&lt;/h2&gt;

&lt;p&gt;Em um envelope Debezium típico, podemos encontrar códigos como:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Código&lt;/th&gt;
&lt;th&gt;Significado&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;c&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Create / Insert&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;u&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Update&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;d&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Delete&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;r&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Read de snapshot&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;O evento de snapshot merece atenção.&lt;/p&gt;

&lt;p&gt;Ele representa uma leitura do estado existente durante a inicialização ou uma operação de snapshot.&lt;/p&gt;

&lt;p&gt;Não deve ser tratado automaticamente como uma nova inserção de negócio.&lt;/p&gt;

&lt;p&gt;A aplicação downstream precisa compreender a semântica desses eventos.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. DELETE e Tombstone
&lt;/h2&gt;

&lt;p&gt;Exclusões exigem cuidado especial.&lt;/p&gt;

&lt;p&gt;Um evento de delete pode conter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"before"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"pedido_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PAGO"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"after"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"op"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"d"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Dependendo da configuração Debezium/Kafka, também pode ser emitido um registro tombstone, com a chave do registro e valor Kafka nulo.&lt;/p&gt;

&lt;p&gt;Esse tombstone está relacionado à semântica de compactação de logs do Kafka.&lt;/p&gt;

&lt;p&gt;Ele não deve ser confundido com o evento CDC de exclusão.&lt;/p&gt;

&lt;p&gt;Uma aplicação que simplesmente descarta mensagens de valor nulo precisa compreender o impacto dessa decisão.&lt;/p&gt;

&lt;p&gt;E uma aplicação que ignora eventos &lt;code&gt;d&lt;/code&gt; pode manter registros que já foram excluídos da origem.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. Arquitetura end-to-end
&lt;/h2&gt;

&lt;p&gt;Vamos reunir os componentes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  POSTGRESQL
                      |
                      v
                     WAL
                      |
                      v
                   DEBEZIUM
                      |
                      v
                 KAFKA CONNECT
                      |
                      v
                 KAFKA TOPIC
                      |
                      v
             SPARK STRUCTURED
                 STREAMING
                      |
                      v
               SCHEMA PARSING
                      |
                      v
               CDC NORMALIZATION
                      |
                      v
               DATA QUALITY
                      |
                      v
               INCREMENTAL MERGE
                      |
                      v
                LAKEHOUSE TABLE
                      |
                      v
                  SQL / BI / ML
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cada camada possui responsabilidades específicas.&lt;/p&gt;

&lt;p&gt;O Debezium captura alterações.&lt;/p&gt;

&lt;p&gt;Kafka transporta e retém eventos conforme sua configuração.&lt;/p&gt;

&lt;p&gt;Spark interpreta, transforma e valida.&lt;/p&gt;

&lt;p&gt;O formato Lakehouse gerencia o estado da tabela analítica.&lt;/p&gt;

&lt;p&gt;Essa separação facilita a evolução da arquitetura.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. Lendo Kafka com Spark Structured Streaming
&lt;/h2&gt;

&lt;p&gt;Vamos utilizar PySpark.&lt;/p&gt;

&lt;p&gt;A integração exige o conector Kafka compatível com a versão do Spark e do Scala.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SparkKafkaCDC&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;eventos_kafka&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;readStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;kafka&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;kafka.bootstrap.servers&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;localhost:9092&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;subscribe&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dbserver1.public.pedidos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;startingOffsets&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;earliest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O endereço, o tópico e as configurações de segurança precisam corresponder ao ambiente real.&lt;/p&gt;

&lt;p&gt;O parâmetro &lt;code&gt;startingOffsets&lt;/code&gt; define a posição inicial para uma nova consulta.&lt;/p&gt;

&lt;p&gt;Ao reiniciar uma consulta com checkpoint válido, o progresso persistido normalmente prevalece sobre essa configuração inicial.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. Interpretando o payload
&lt;/h2&gt;

&lt;p&gt;Kafka disponibiliza o valor da mensagem como binário.&lt;/p&gt;

&lt;p&gt;Podemos convertê-lo para string:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;functions&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;

&lt;span class="n"&gt;mensagens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;eventos_kafka&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;cast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;string&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;kafka_key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;cast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;string&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;payload&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;topic&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;partition&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;offset&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora precisamos interpretar o JSON.&lt;/p&gt;

&lt;p&gt;Vamos definir um schema simplificado.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.types&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;DecimalType&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;schema_registro&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;DecimalType&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;schema_cdc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;before&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;schema_registro&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;after&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;schema_registro&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;op&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ts_ms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse schema corresponde ao envelope simplificado apresentado anteriormente.&lt;/p&gt;

&lt;p&gt;Em uma integração real, devemos ajustar o parsing à serialização utilizada, incluindo eventuais envelopes adicionais de Kafka Connect, formatos Avro ou Protobuf e integração com Schema Registry.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Normalizando eventos CDC
&lt;/h2&gt;

&lt;p&gt;Agora aplicamos &lt;code&gt;from_json&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;eventos_parseados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;mensagens&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cdc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;payload&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="n"&gt;schema_cdc&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Precisamos obter a chave do registro.&lt;/p&gt;

&lt;p&gt;Para INSERT e UPDATE, ela normalmente estará em &lt;code&gt;after&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Para DELETE, pode estar em &lt;code&gt;before&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;eventos_normalizados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos_parseados&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cdc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNotNull&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;coalesce&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cdc.after.pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cdc.before.pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cdc.after.status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cdc.after.valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cdc.op&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;operacao&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cdc.ts_ms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_ts_ms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;topic&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;partition&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;offset&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora temos uma representação mais adequada para processamento incremental.&lt;/p&gt;

&lt;p&gt;Entretanto, essa normalização ainda não resolve a ordenação lógica dos eventos.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. Por que timestamp não é suficiente para ordenar CDC?
&lt;/h2&gt;

&lt;p&gt;Imagine duas alterações do mesmo pedido:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Event A
pedido_id = 101
status = PAGO

Event B
pedido_id = 101
status = CANCELADO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se utilizarmos apenas um timestamp de baixa precisão, os eventos podem ter o mesmo valor temporal.&lt;/p&gt;

&lt;p&gt;Além disso, timestamps de processamento não representam necessariamente a ordem de commit na origem.&lt;/p&gt;

&lt;p&gt;Para garantir a aplicação correta das alterações, devemos utilizar metadados de ordenação apropriados ao conector e ao banco.&lt;/p&gt;

&lt;p&gt;No PostgreSQL, informações relacionadas à posição no WAL, como LSN, podem fazer parte dessa estratégia.&lt;/p&gt;

&lt;p&gt;Mas é importante validar a semântica dos campos expostos pelo Debezium.&lt;/p&gt;

&lt;p&gt;Não devemos assumir que um único campo de timestamp seja suficiente para reconstruir a ordem transacional em qualquer cenário.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. O problema dos eventos fora de ordem
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source Order:

1. CRIADO
2. PAGO
3. CANCELADO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas o destino recebe ou aplica:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. CRIADO
2. CANCELADO
3. PAGO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se aplicarmos simplesmente a última mensagem recebida, o estado final ficará incorreto.&lt;/p&gt;

&lt;p&gt;O pedido poderá aparecer como PAGO quando deveria estar CANCELADO.&lt;/p&gt;

&lt;p&gt;Por isso, pipelines CDC precisam definir:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Chave de negócio.&lt;/li&gt;
&lt;li&gt;Ordenação lógica.&lt;/li&gt;
&lt;li&gt;Estratégia de deduplicação.&lt;/li&gt;
&lt;li&gt;Tratamento de eventos antigos.&lt;/li&gt;
&lt;li&gt;Política de reprocessamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A arquitetura precisa preservar ou reconstruir a sequência correta de alterações.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. Deduplicação por chave e versão
&lt;/h2&gt;

&lt;p&gt;Vamos imaginar um dataset normalizado com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pedido_id
status
valor
source_version
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Aqui, &lt;code&gt;source_version&lt;/code&gt; representa uma versão lógica de origem confiável, já normalizada e comparável dentro do domínio de cada pedido.&lt;/p&gt;

&lt;p&gt;Não estamos assumindo que esse campo exista diretamente no envelope Debezium.&lt;/p&gt;

&lt;p&gt;Ele precisa ser construído conforme a semântica real da origem.&lt;/p&gt;

&lt;p&gt;Podemos selecionar a versão mais recente de cada pedido em um conjunto finito de eventos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Window&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;functions&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;

&lt;span class="n"&gt;janela&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;Window&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;partitionBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;orderBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source_version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;eventos_recentes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_eventos&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rn&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;row_number&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;over&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;janela&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rn&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;drop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rn&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse exemplo é adequado para um DataFrame batch, como o recebido dentro de &lt;code&gt;foreachBatch&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Para resultados determinísticos, a ordenação precisa ser total ou possuir um critério de desempate definido.&lt;/p&gt;

&lt;p&gt;Também precisamos tratar versões repetidas e eventos de delete.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. &lt;code&gt;foreachBatch&lt;/code&gt; para processamento incremental
&lt;/h2&gt;

&lt;p&gt;Uma estratégia comum é consumir eventos com Structured Streaming e aplicar cada microbatch ao destino.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;processar_microbatch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch_df&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;batch_id&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Processando microbatch: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;batch_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Validar dados
&lt;/span&gt;    &lt;span class="c1"&gt;# Normalizar operações
&lt;/span&gt;    &lt;span class="c1"&gt;# Deduplicar eventos
&lt;/span&gt;    &lt;span class="c1"&gt;# Aplicar MERGE transacional
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Iniciamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos_normalizados&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writeStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;foreachBatch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;processar_microbatch&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;checkpointLocation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/checkpoints/pedidos_cdc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O checkpoint deve utilizar armazenamento persistente e adequado ao ambiente.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Importante:&lt;/strong&gt; &lt;code&gt;foreachBatch&lt;/code&gt; oferece semântica de execução at-least-once para a função.&lt;/p&gt;

&lt;p&gt;Se ocorrer uma falha, o mesmo microbatch pode ser executado novamente.&lt;/p&gt;

&lt;p&gt;Portanto, precisamos tornar a aplicação no destino idempotente.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Upsert com MERGE INTO
&lt;/h2&gt;

&lt;p&gt;Em um Lakehouse compatível, podemos aplicar atualizações incrementais com &lt;code&gt;MERGE&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Exemplo conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="n"&gt;MERGE&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;lakehouse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sales&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedidos&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;destino&lt;/span&gt;
&lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="n"&gt;pedidos_incrementais&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;
&lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedido_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedido_id&lt;/span&gt;

&lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;MATCHED&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;operacao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'d'&lt;/span&gt;
&lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="k"&gt;DELETE&lt;/span&gt;

&lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;MATCHED&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;operacao&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'c'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'u'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'r'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt;
    &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt;

&lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="n"&gt;MATCHED&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;operacao&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'c'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'u'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'r'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;pedido_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;valor&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedido_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse exemplo demonstra a lógica básica de upsert e delete.&lt;/p&gt;

&lt;p&gt;Mas ainda existe um problema.&lt;/p&gt;

&lt;p&gt;Se um evento antigo chegar depois de um evento novo, o MERGE poderá sobrescrever o estado correto.&lt;/p&gt;

&lt;p&gt;Uma implementação robusta deve comparar versões de origem e persistir a versão aplicada.&lt;/p&gt;

&lt;p&gt;Além disso, uma exclusão precisa ser protegida contra a ressurreição indevida de registros por eventos antigos.&lt;/p&gt;

&lt;p&gt;Dependendo dos requisitos, isso pode exigir tombstones lógicos ou uma tabela de controle de versões.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. Idempotência: não basta repetir o MERGE
&lt;/h2&gt;

&lt;p&gt;Imagine que um microbatch seja aplicado com sucesso.&lt;/p&gt;

&lt;p&gt;Antes de o Spark confirmar o progresso, ocorre uma falha.&lt;/p&gt;

&lt;p&gt;Na recuperação, o mesmo microbatch é executado novamente.&lt;/p&gt;

&lt;p&gt;Se a operação for idempotente, o estado final permanecerá correto.&lt;/p&gt;

&lt;p&gt;Mas precisamos avaliar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Se a fonte possui identificadores estáveis.&lt;/li&gt;
&lt;li&gt;Se a versão da alteração é preservada.&lt;/li&gt;
&lt;li&gt;Se operações antigas são ignoradas.&lt;/li&gt;
&lt;li&gt;Se deletes são protegidos contra reintrodução.&lt;/li&gt;
&lt;li&gt;Se o destino oferece transações adequadas.&lt;/li&gt;
&lt;li&gt;Se existem efeitos colaterais externos.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Um &lt;code&gt;MERGE&lt;/code&gt; baseado apenas em chave pode não ser suficiente.&lt;/p&gt;

&lt;p&gt;A regra correta deve considerar a versão lógica da alteração.&lt;/p&gt;




&lt;h2&gt;
  
  
  22. Exemplo de MERGE protegido por versão
&lt;/h2&gt;

&lt;p&gt;Imagine uma tabela com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pedido_id
status
valor
source_version
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos utilizar uma condição adicional:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="n"&gt;MERGE&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;lakehouse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sales&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedidos&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;destino&lt;/span&gt;
&lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="n"&gt;pedidos_incrementais&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;
&lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedido_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedido_id&lt;/span&gt;

&lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;MATCHED&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;source_version&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;source_version&lt;/span&gt;
&lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt;
    &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;source_version&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;source_version&lt;/span&gt;

&lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="n"&gt;MATCHED&lt;/span&gt;
&lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;pedido_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;source_version&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedido_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;source_version&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa estratégia demonstra a proteção contra atualizações antigas.&lt;/p&gt;

&lt;p&gt;Entretanto, o exemplo pressupõe eventos de inserção ou atualização e uma versão de origem confiável.&lt;/p&gt;

&lt;p&gt;Deletes precisam de tratamento adicional, porque remover fisicamente o registro também remove a versão que impediria sua reintrodução por um evento atrasado.&lt;/p&gt;

&lt;p&gt;Em produção, a estratégia de exclusão deve ser projetada explicitamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. Snapshot inicial e streaming contínuo
&lt;/h2&gt;

&lt;p&gt;Um desafio importante é iniciar um pipeline CDC sobre uma tabela que já contém dados.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Existing Table:
10 million records
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Precisamos carregar esse estado inicial.&lt;/p&gt;

&lt;p&gt;Depois, precisamos acompanhar as alterações que ocorrem enquanto o snapshot está sendo produzido.&lt;/p&gt;

&lt;p&gt;Uma arquitetura CDC deve garantir uma transição consistente entre:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;INITIAL SNAPSHOT
       |
       v
CONTINUOUS CHANGES
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Debezium oferece mecanismos específicos para esse cenário.&lt;/p&gt;

&lt;p&gt;Mas o comportamento depende do conector, do modo de snapshot e da configuração.&lt;/p&gt;

&lt;p&gt;Não devemos implementar uma carga inicial independente e simplesmente iniciar a leitura do WAL depois, sem estabelecer uma fronteira consistente entre as duas operações.&lt;/p&gt;

&lt;p&gt;Isso pode causar perdas ou duplicidades.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Schema Evolution em CDC
&lt;/h2&gt;

&lt;p&gt;Agora imagine que o banco operacional adiciona uma coluna:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;pedidos&lt;/span&gt;
&lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;canal_venda&lt;/span&gt; &lt;span class="nb"&gt;VARCHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O evento CDC poderá passar a conter essa informação.&lt;/p&gt;

&lt;p&gt;Mas o pipeline Spark possui um schema definido.&lt;/p&gt;

&lt;p&gt;Se não houver uma estratégia de evolução, o novo campo poderá ser ignorado ou causar incompatibilidades, dependendo da implementação.&lt;/p&gt;

&lt;p&gt;Precisamos planejar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Evolução do schema na origem.&lt;/li&gt;
&lt;li&gt;Serialização dos eventos.&lt;/li&gt;
&lt;li&gt;Compatibilidade no Schema Registry, quando utilizado.&lt;/li&gt;
&lt;li&gt;Evolução do parsing Spark.&lt;/li&gt;
&lt;li&gt;Evolução da tabela de destino.&lt;/li&gt;
&lt;li&gt;Contratos com consumidores.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma alteração no banco operacional pode afetar toda a cadeia analítica.&lt;/p&gt;

&lt;p&gt;Esse é um exemplo claro da importância dos Data Contracts estudados na Parte 14.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. Schema Registry
&lt;/h2&gt;

&lt;p&gt;Em arquiteturas de eventos, um Schema Registry pode ajudar a gerenciar schemas e regras de compatibilidade.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;EVENT PRODUCER
      |
      v
SCHEMA VALIDATION
      |
      v
KAFKA TOPIC
      |
      v
SPARK CONSUMER
      |
      v
SCHEMA RESOLUTION
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Dependendo do ecossistema, podemos utilizar formatos como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Avro.&lt;/li&gt;
&lt;li&gt;Protobuf.&lt;/li&gt;
&lt;li&gt;JSON Schema.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas o Schema Registry não garante automaticamente que todos os consumidores estejam semanticamente compatíveis.&lt;/p&gt;

&lt;p&gt;Ele ajuda a gerenciar a estrutura e determinadas regras de evolução.&lt;/p&gt;

&lt;p&gt;As regras de negócio continuam exigindo contratos e validações próprias.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. Replay e recuperação
&lt;/h2&gt;

&lt;p&gt;Uma vantagem importante do Kafka é permitir releitura de eventos ainda disponíveis conforme as políticas de retenção.&lt;/p&gt;

&lt;p&gt;Imagine que uma transformação Spark possuía um bug.&lt;/p&gt;

&lt;p&gt;Depois da correção, talvez precisemos reprocessar os eventos.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;KAFKA EVENT LOG
       |
       v
REPLAY
       |
       v
CORRECTED SPARK PIPELINE
       |
       v
REBUILT ANALYTICAL TABLE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas replay exige planejamento.&lt;/p&gt;

&lt;p&gt;Precisamos saber:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Quais offsets representam o intervalo desejado.&lt;/li&gt;
&lt;li&gt;Se os eventos ainda estão disponíveis.&lt;/li&gt;
&lt;li&gt;Se o destino deve ser reconstruído ou atualizado.&lt;/li&gt;
&lt;li&gt;Como evitar duplicidades.&lt;/li&gt;
&lt;li&gt;Como preservar a ordem.&lt;/li&gt;
&lt;li&gt;Como validar o resultado.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Também não devemos apagar checkpoints indiscriminadamente para forçar um replay.&lt;/p&gt;

&lt;p&gt;Isso pode causar efeitos inesperados no destino.&lt;/p&gt;

&lt;p&gt;Uma estratégia segura pode utilizar uma nova consulta, um checkpoint separado e uma área de reconstrução controlada.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Retenção Kafka não é retenção analítica
&lt;/h2&gt;

&lt;p&gt;Kafka pode reter eventos por tempo, tamanho ou políticas específicas.&lt;/p&gt;

&lt;p&gt;Mas não devemos tratá-lo automaticamente como armazenamento histórico permanente.&lt;/p&gt;

&lt;p&gt;Imagine que o tópico possui retenção de sete dias.&lt;/p&gt;

&lt;p&gt;Se um pipeline ficar indisponível por dez dias, determinados eventos podem não estar mais disponíveis.&lt;/p&gt;

&lt;p&gt;Isso pode comprometer a recuperação.&lt;/p&gt;

&lt;p&gt;Portanto, precisamos alinhar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;KAFKA RETENTION
       |
       v
MAXIMUM RECOVERY WINDOW
       |
       v
PIPELINE SLA
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Para requisitos de histórico e auditoria, podemos persistir eventos brutos em uma camada de armazenamento durável.&lt;/p&gt;

&lt;p&gt;Essa camada pode servir como base para reprocessamentos posteriores.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Observabilidade de CDC
&lt;/h2&gt;

&lt;p&gt;Uma arquitetura CDC possui vários pontos de falha.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;POSTGRESQL
    |
    v
DEBEZIUM
    |
    v
KAFKA
    |
    v
SPARK
    |
    v
LAKEHOUSE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Precisamos monitorar cada camada.&lt;/p&gt;

&lt;h3&gt;
  
  
  PostgreSQL
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Retenção de WAL.&lt;/li&gt;
&lt;li&gt;Replication slots.&lt;/li&gt;
&lt;li&gt;Lag de replicação.&lt;/li&gt;
&lt;li&gt;Uso de armazenamento.&lt;/li&gt;
&lt;li&gt;Disponibilidade.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Debezium / Kafka Connect
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Estado do conector.&lt;/li&gt;
&lt;li&gt;Erros de captura.&lt;/li&gt;
&lt;li&gt;Progresso do snapshot.&lt;/li&gt;
&lt;li&gt;Lag.&lt;/li&gt;
&lt;li&gt;Falhas de serialização.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Kafka
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Retenção.&lt;/li&gt;
&lt;li&gt;Disponibilidade de brokers.&lt;/li&gt;
&lt;li&gt;Partições.&lt;/li&gt;
&lt;li&gt;Produção e consumo.&lt;/li&gt;
&lt;li&gt;Lag, conforme o consumidor.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Spark
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Duração dos microbatches.&lt;/li&gt;
&lt;li&gt;Taxa de entrada.&lt;/li&gt;
&lt;li&gt;Taxa de processamento.&lt;/li&gt;
&lt;li&gt;Checkpoints.&lt;/li&gt;
&lt;li&gt;Falhas.&lt;/li&gt;
&lt;li&gt;Backlog.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Lakehouse
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Tempo de MERGE.&lt;/li&gt;
&lt;li&gt;Quantidade de arquivos.&lt;/li&gt;
&lt;li&gt;Compactação.&lt;/li&gt;
&lt;li&gt;Qualidade.&lt;/li&gt;
&lt;li&gt;Consistência.&lt;/li&gt;
&lt;li&gt;Freshness.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma aplicação Spark saudável não significa que toda a cadeia CDC esteja saudável.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. Latência ponta a ponta
&lt;/h2&gt;

&lt;p&gt;Em uma arquitetura CDC, a latência relevante geralmente começa na alteração da origem.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Database Commit
      |
      v
CDC Capture
      |
      v
Kafka Publish
      |
      v
Spark Processing
      |
      v
Lakehouse Commit
      |
      v
Consumer Availability
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos definir:&lt;/p&gt;

&lt;p&gt;[&lt;/p&gt;

&lt;h1&gt;
  
  
  \text{End-to-End Latency}
&lt;/h1&gt;

&lt;h2&gt;
  
  
  \text{Availability Time}
&lt;/h2&gt;

&lt;p&gt;\text{Source Commit Time}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Essa métrica precisa utilizar timestamps confiáveis e semântica bem definida.&lt;/p&gt;

&lt;p&gt;O timestamp de processamento do Spark, isoladamente, não mede toda a latência da cadeia.&lt;/p&gt;

&lt;p&gt;Também precisamos distinguir latência média de percentis como P95 e P99.&lt;/p&gt;

&lt;p&gt;Uma média baixa pode esconder eventos que demoram muito mais do que o esperado.&lt;/p&gt;


&lt;h2&gt;
  
  
  30. Data Quality em eventos CDC
&lt;/h2&gt;

&lt;p&gt;Nem todo evento recebido deve ser aplicado automaticamente.&lt;/p&gt;

&lt;p&gt;Precisamos validar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Identificador do registro.&lt;/li&gt;
&lt;li&gt;Tipo de operação.&lt;/li&gt;
&lt;li&gt;Estrutura do payload.&lt;/li&gt;
&lt;li&gt;Versão da alteração.&lt;/li&gt;
&lt;li&gt;Campos obrigatórios.&lt;/li&gt;
&lt;li&gt;Regras de negócio.&lt;/li&gt;
&lt;li&gt;Compatibilidade de schema.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;eventos_validos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;eventos_normalizados&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNotNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;operacao&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;c&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;u&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa é apenas uma validação inicial.&lt;/p&gt;

&lt;p&gt;Em produção, precisamos diferenciar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Eventos válidos.&lt;/li&gt;
&lt;li&gt;Eventos inválidos.&lt;/li&gt;
&lt;li&gt;Tombstones.&lt;/li&gt;
&lt;li&gt;Eventos de controle.&lt;/li&gt;
&lt;li&gt;Alterações incompatíveis.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Também precisamos preservar os dados necessários para investigação.&lt;/p&gt;

&lt;p&gt;Descartar silenciosamente eventos CDC pode comprometer a consistência do destino.&lt;/p&gt;




&lt;h2&gt;
  
  
  31. Laboratório local: simulando CDC sem Kafka
&lt;/h2&gt;

&lt;p&gt;Agora vamos construir um exemplo executável em PySpark.&lt;/p&gt;

&lt;p&gt;O objetivo é compreender a lógica de aplicação incremental sem depender de PostgreSQL, Kafka ou Debezium.&lt;/p&gt;

&lt;p&gt;Vamos simular eventos com:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Identificador.&lt;/li&gt;
&lt;li&gt;Operação.&lt;/li&gt;
&lt;li&gt;Status.&lt;/li&gt;
&lt;li&gt;Valor.&lt;/li&gt;
&lt;li&gt;Versão de origem.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Primeiro, criamos os eventos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;functions&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Window&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.types&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;DecimalType&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;decimal&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Decimal&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;master&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;local[*]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CDCLab&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;operacao&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;DecimalType&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source_version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;eventos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;c&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;150.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;u&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;150.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;102&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;c&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;230.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;u&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ENVIADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;150.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;102&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;103&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;c&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;99.90&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;u&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;150.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;df_eventos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createDataFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df_eventos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;truncate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Observe que incluímos propositalmente um evento antigo repetido do pedido 101.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Selecionando o estado mais recente
&lt;/h2&gt;

&lt;p&gt;Agora queremos identificar o último evento lógico de cada pedido.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;janela&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;Window&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;partitionBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;orderBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source_version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;eventos_recentes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_eventos&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rn&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;row_number&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;over&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;janela&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rn&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;drop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rn&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;eventos_recentes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;truncate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Como os eventos duplicados do pedido 101 possuem a mesma versão e o mesmo conteúdo, não há conflito de estado nesse exemplo.&lt;/p&gt;

&lt;p&gt;Em produção, versões iguais com conteúdos diferentes precisam ser tratadas como inconsistência ou resolvidas por uma regra explícita.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. Aplicando exclusões
&lt;/h2&gt;

&lt;p&gt;Agora separamos os registros cujo último evento representa delete.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;registros_ativos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos_recentes&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;operacao&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source_version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;registros_excluidos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos_recentes&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;operacao&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source_version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos visualizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;registros_ativos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;registros_excluidos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O estado ativo esperado é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pedido_id = 101
status = ENVIADO

pedido_id = 103
status = CRIADO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O pedido 102 foi excluído logicamente da visão ativa.&lt;/p&gt;

&lt;p&gt;Esse laboratório reconstrói o estado final a partir de todo o conjunto de eventos disponível.&lt;/p&gt;

&lt;p&gt;Ele não representa, sozinho, uma implementação incremental transacional de produção.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Por que manter o histórico de alterações?
&lt;/h2&gt;

&lt;p&gt;Uma tabela de estado atual responde:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Qual é a situação do pedido agora?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Mas não responde necessariamente:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Como o pedido chegou a esse estado?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Por isso, muitas arquiteturas mantêm duas representações.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Event History&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Preserva as alterações recebidas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Current State&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Representa o estado mais recente de cada entidade.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  CDC EVENTS
                      |
             +--------+--------+
             |                 |
             v                 v
       EVENT HISTORY       CURRENT STATE
             |                 |
             v                 v
          AUDIT           ANALYTICS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa separação pode facilitar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Auditoria.&lt;/li&gt;
&lt;li&gt;Replay.&lt;/li&gt;
&lt;li&gt;Investigação.&lt;/li&gt;
&lt;li&gt;Reconstrução de tabelas.&lt;/li&gt;
&lt;li&gt;Análises temporais.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas o histórico precisa de políticas de retenção, privacidade e acesso.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. Uma arquitetura de referência para produção
&lt;/h2&gt;

&lt;p&gt;Vamos reunir as principais decisões.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  OPERATIONAL SYSTEM
                         |
                         v
                     POSTGRESQL
                         |
                         v
                    LOGICAL WAL
                         |
                         v
                      DEBEZIUM
                         |
                         v
                    KAFKA TOPICS
                         |
                         v
                 SPARK STREAMING
                         |
                         v
                 CONTRACT VALIDATION
                         |
                         v
                   CDC NORMALIZATION
                         |
                         v
                 ORDERING / DEDUP
                         |
                         v
                    QUALITY GATES
                         |
              +----------+----------+
              |                     |
              v                     v
          VALID EVENTS         QUARANTINE
              |
              v
         RAW EVENT HISTORY
              |
              v
         INCREMENTAL MERGE
              |
              v
         LAKEHOUSE TABLE
              |
              v
          SQL / BI / ML
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em uma implementação real, algumas dessas responsabilidades podem ser reorganizadas.&lt;/p&gt;

&lt;p&gt;Por exemplo, podemos persistir o histórico bruto antes das validações de negócio, preservando eventos para auditoria e reprocessamento.&lt;/p&gt;

&lt;p&gt;O desenho depende dos requisitos de segurança, recuperação e consistência.&lt;/p&gt;




&lt;h2&gt;
  
  
  36. Quando CDC não é necessário?
&lt;/h2&gt;

&lt;p&gt;CDC é poderoso, mas introduz complexidade.&lt;/p&gt;

&lt;p&gt;Talvez não seja necessário quando:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A tabela é pequena.&lt;/li&gt;
&lt;li&gt;A atualização é pouco frequente.&lt;/li&gt;
&lt;li&gt;O SLA permite cargas diárias.&lt;/li&gt;
&lt;li&gt;Não existem deletes relevantes.&lt;/li&gt;
&lt;li&gt;O sistema de origem oferece APIs incrementais confiáveis.&lt;/li&gt;
&lt;li&gt;O custo operacional do CDC supera seus benefícios.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Em alguns cenários, uma carga incremental por timestamp bem implementada pode ser suficiente.&lt;/p&gt;

&lt;p&gt;A decisão deve considerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Volume.&lt;/li&gt;
&lt;li&gt;Frequência de mudanças.&lt;/li&gt;
&lt;li&gt;Latência.&lt;/li&gt;
&lt;li&gt;Custo.&lt;/li&gt;
&lt;li&gt;Complexidade.&lt;/li&gt;
&lt;li&gt;Garantias de consistência.&lt;/li&gt;
&lt;li&gt;Capacidade operacional da equipe.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A arquitetura mais sofisticada nem sempre é a melhor arquitetura.&lt;/p&gt;




&lt;h2&gt;
  
  
  37. Erros comuns em pipelines CDC
&lt;/h2&gt;

&lt;p&gt;Alguns erros merecem atenção especial.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Utilizar timestamp como única garantia de ordenação.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pode não representar a sequência real das alterações.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ignorar eventos de delete.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;O destino pode manter registros inexistentes na origem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Confundir tombstone Kafka com delete CDC.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;São conceitos relacionados, mas diferentes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Aplicar MERGE sem controle de versão.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Eventos antigos podem sobrescrever estados novos.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não monitorar replication slots.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;O WAL pode crescer e comprometer o PostgreSQL.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não planejar snapshot inicial.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Podem ocorrer lacunas ou duplicidades entre carga inicial e streaming.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não definir política de replay.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Falhas podem exigir reconstruções difíceis.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Assumir exactly-once em qualquer destino.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;As garantias dependem da arquitetura completa.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não versionar schemas.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Mudanças na origem podem quebrar consumidores.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não preservar histórico quando necessário.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso dificulta auditoria e reprocessamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  38. Como eu avaliaria uma arquitetura CDC
&lt;/h2&gt;

&lt;p&gt;Antes de colocar um pipeline em produção, faria perguntas como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual é o banco de origem?
          |
          v
Como as alterações são capturadas?
          |
          v
Como funciona o snapshot inicial?
          |
          v
Qual é a chave do registro?
          |
          v
Como preservamos a ordem?
          |
          v
Como tratamos duplicidades?
          |
          v
Como tratamos DELETE?
          |
          v
Como controlamos versões?
          |
          v
Como garantimos idempotência?
          |
          v
Como fazemos replay?
          |
          v
Qual é a retenção necessária?
          |
          v
Como monitoramos lag?
          |
          v
Como validamos a consistência final?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essas perguntas precisam ser respondidas antes de considerar a arquitetura pronta.&lt;/p&gt;




&lt;h2&gt;
  
  
  39. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;Apache Kafka, Debezium e Spark Structured Streaming permitem construir pipelines incrementais capazes de acompanhar mudanças em sistemas transacionais.&lt;/p&gt;

&lt;p&gt;Mas a complexidade principal não está apenas em transportar eventos.&lt;/p&gt;

&lt;p&gt;Ela está em preservar corretamente a semântica das alterações.&lt;/p&gt;

&lt;p&gt;Precisamos compreender:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Origem das mudanças.&lt;/li&gt;
&lt;li&gt;Ordenação.&lt;/li&gt;
&lt;li&gt;Chaves.&lt;/li&gt;
&lt;li&gt;Duplicidades.&lt;/li&gt;
&lt;li&gt;Deletes.&lt;/li&gt;
&lt;li&gt;Snapshots.&lt;/li&gt;
&lt;li&gt;Versões.&lt;/li&gt;
&lt;li&gt;Checkpoints.&lt;/li&gt;
&lt;li&gt;Idempotência.&lt;/li&gt;
&lt;li&gt;Recuperação.&lt;/li&gt;
&lt;li&gt;Qualidade.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O princípio mais importante é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;CDC não é apenas capturar mudanças. É reconstruir e manter corretamente o estado dos dados a partir dessas mudanças.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  40. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando avalio uma plataforma que promete dados em tempo quase real, não pergunto apenas:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Qual é a latência do Kafka?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Quero saber:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Como as mudanças são capturadas?&lt;/li&gt;
&lt;li&gt;Qual é a consistência do snapshot inicial?&lt;/li&gt;
&lt;li&gt;Como eventos fora de ordem são tratados?&lt;/li&gt;
&lt;li&gt;Como o destino evita atualizações antigas?&lt;/li&gt;
&lt;li&gt;Como deletes são preservados?&lt;/li&gt;
&lt;li&gt;O que acontece após uma falha?&lt;/li&gt;
&lt;li&gt;Como os eventos podem ser reprocessados?&lt;/li&gt;
&lt;li&gt;Como monitoramos a cadeia inteira?&lt;/li&gt;
&lt;li&gt;Como validamos a consistência entre origem e destino?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Essas perguntas representam a diferença entre um pipeline que transporta mensagens e uma arquitetura incremental confiável.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora entendemos como capturar e processar mudanças de sistemas transacionais.&lt;/p&gt;

&lt;p&gt;Na próxima parte, vamos aprofundar a construção de pipelines Spark para ambientes corporativos.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Orquestração, Airflow, CI/CD e DataOps com Apache Spark.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Vamos estudar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Orquestração vs. processamento.&lt;/li&gt;
&lt;li&gt;Apache Airflow.&lt;/li&gt;
&lt;li&gt;DAGs.&lt;/li&gt;
&lt;li&gt;Dependências.&lt;/li&gt;
&lt;li&gt;Retries.&lt;/li&gt;
&lt;li&gt;Backfills.&lt;/li&gt;
&lt;li&gt;Idempotência.&lt;/li&gt;
&lt;li&gt;Testes automatizados.&lt;/li&gt;
&lt;li&gt;CI/CD.&lt;/li&gt;
&lt;li&gt;Versionamento.&lt;/li&gt;
&lt;li&gt;Deploy.&lt;/li&gt;
&lt;li&gt;Observabilidade.&lt;/li&gt;
&lt;li&gt;Estratégias de produção.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central será:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como transformar Jobs Spark isolados em pipelines automatizados, testáveis, observáveis e confiáveis?&lt;/strong&gt;&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 15:&lt;/strong&gt; Structured Streaming, Watermarks e Checkpoints&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 17 — Orquestração, Airflow, CI/CD e DataOps com Apache Spark&lt;/p&gt;

</description>
      <category>python</category>
      <category>spark</category>
      <category>bigdata</category>
      <category>sql</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 15: Structured Streaming, Watermarks e Checkpoints</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 20:14:42 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-15-structured-streaming-watermarks-e-checkpoints-i48</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-15-structured-streaming-watermarks-e-checkpoints-i48</guid>
      <description>&lt;h1&gt;
  
  
  Como aprendi Apache Spark — Parte 15: Structured Streaming, Watermarks e Checkpoints
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Nota histórica: esta série revisita minha trajetória em Engenharia de Dados iniciada em 2012. O Structured Streaming foi introduzido posteriormente, na evolução do Apache Spark 2.x. Este capítulo conecta os fundamentos do processamento distribuído às arquiteturas modernas de streaming.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nos capítulos anteriores, estudamos como o Apache Spark distribui processamento, organiza Tasks, gerencia memória, executa shuffle e otimiza consultas.&lt;/p&gt;

&lt;p&gt;Também exploramos formatos de armazenamento, arquiteturas Lakehouse, qualidade de dados e governança.&lt;/p&gt;

&lt;p&gt;Agora vamos enfrentar um problema diferente.&lt;/p&gt;

&lt;p&gt;Até aqui, muitos exemplos partiam de um dataset finito.&lt;/p&gt;

&lt;p&gt;Líamos os dados, aplicávamos transformações e gravávamos o resultado.&lt;/p&gt;

&lt;p&gt;Mas e quando os dados não param de chegar?&lt;/p&gt;

&lt;p&gt;Imagine uma plataforma que recebe continuamente:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Transações financeiras.&lt;/li&gt;
&lt;li&gt;Pedidos de e-commerce.&lt;/li&gt;
&lt;li&gt;Eventos de aplicativos.&lt;/li&gt;
&lt;li&gt;Telemetria de equipamentos.&lt;/li&gt;
&lt;li&gt;Atualizações de sistemas operacionais.&lt;/li&gt;
&lt;li&gt;Mensagens de dispositivos IoT.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Não queremos necessariamente esperar até o dia seguinte para processar tudo.&lt;/p&gt;

&lt;p&gt;Queremos transformar eventos em informações úteis conforme eles chegam.&lt;/p&gt;

&lt;p&gt;É nesse contexto que entra o &lt;strong&gt;Apache Spark Structured Streaming&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A pergunta central deste capítulo é:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como processar dados potencialmente infinitos utilizando um mecanismo distribuído, com controle de estado, recuperação de falhas e semântica temporal?&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Batch vs. Streaming
&lt;/h2&gt;

&lt;p&gt;No processamento batch, trabalhamos normalmente com um conjunto finito de dados.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 BATCH PROCESSING

Data Source
     |
     v
Finite Dataset
     |
     v
Spark Job
     |
     v
Result
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas/2026-11-05&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;categoria&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/resultados&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A aplicação processa os dados disponíveis e termina.&lt;/p&gt;

&lt;p&gt;No streaming, a fonte pode continuar produzindo eventos indefinidamente.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;               STREAM PROCESSING

Event 1 ----+
Event 2 ----+
Event 3 ----+----&amp;gt; Spark Streaming Query
Event 4 ----+              |
Event 5 ----+              v
   ...                Continuous Results
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O processamento acompanha a evolução da fonte.&lt;/p&gt;

&lt;p&gt;Isso não significa que cada evento seja necessariamente processado individualmente e imediatamente.&lt;/p&gt;

&lt;p&gt;A estratégia de execução depende do mecanismo utilizado.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. O que é Structured Streaming?
&lt;/h2&gt;

&lt;p&gt;Structured Streaming é a API de processamento de fluxos do Apache Spark baseada no modelo estruturado de DataFrames e Spark SQL.&lt;/p&gt;

&lt;p&gt;Sua principal proposta é permitir que desenvolvedores utilizem operações familiares para trabalhar com dados que chegam continuamente.&lt;/p&gt;

&lt;p&gt;Em batch:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/eventos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em streaming:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;readStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/eventos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A diferença parece pequena.&lt;/p&gt;

&lt;p&gt;Mas o modelo de execução muda.&lt;/p&gt;

&lt;p&gt;O DataFrame streaming representa uma consulta sobre uma fonte que pode receber novos dados.&lt;/p&gt;

&lt;p&gt;O Spark passa a executar essa consulta incrementalmente, conforme a fonte disponibiliza informações.&lt;/p&gt;

&lt;p&gt;Nem todas as operações permitidas em DataFrames batch são suportadas em consultas streaming, e algumas exigem modos de saída ou configurações específicos.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. O modelo de tabela ilimitada
&lt;/h2&gt;

&lt;p&gt;Uma forma útil de entender Structured Streaming é imaginar um fluxo como uma tabela que cresce continuamente.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TIME T1

+----------+--------+
| event_id | value  |
+----------+--------+
| 1        | 100    |
+----------+--------+

TIME T2

+----------+--------+
| event_id | value  |
+----------+--------+
| 1        | 100    |
| 2        | 200    |
+----------+--------+

TIME T3

+----------+--------+
| event_id | value  |
+----------+--------+
| 1        | 100    |
| 2        | 200    |
| 3        | 300    |
+----------+--------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Spark mantém uma consulta lógica sobre essa tabela.&lt;/p&gt;

&lt;p&gt;À medida que novos dados chegam, o mecanismo atualiza o resultado de acordo com a operação e o modo de execução.&lt;/p&gt;

&lt;p&gt;Esse modelo ajuda a conectar streaming ao conhecimento de Spark SQL que já construímos.&lt;/p&gt;

&lt;p&gt;Mas existe um desafio.&lt;/p&gt;

&lt;p&gt;Se a tabela nunca para de crescer, não podemos simplesmente recalcular todo o histórico a cada evento.&lt;/p&gt;

&lt;p&gt;Precisamos de execução incremental.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Microbatch: o modelo mais utilizado
&lt;/h2&gt;

&lt;p&gt;O mecanismo padrão do Structured Streaming é o processamento por &lt;strong&gt;microbatches&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;EVENT STREAM
     |
     v
+------------+
| Microbatch |
|     1      |
+------------+
     |
     v
Spark Execution

+------------+
| Microbatch |
|     2      |
+------------+
     |
     v
Spark Execution

+------------+
| Microbatch |
|     3      |
+------------+
     |
     v
Spark Execution
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cada microbatch processa um conjunto de dados novos identificado pelo mecanismo de streaming.&lt;/p&gt;

&lt;p&gt;O Spark reutiliza componentes de seu motor distribuído para executar as transformações.&lt;/p&gt;

&lt;p&gt;Isso permite aproveitar conceitos já conhecidos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Driver.&lt;/li&gt;
&lt;li&gt;Executors.&lt;/li&gt;
&lt;li&gt;Tasks.&lt;/li&gt;
&lt;li&gt;Partições.&lt;/li&gt;
&lt;li&gt;Shuffle.&lt;/li&gt;
&lt;li&gt;Spark SQL.&lt;/li&gt;
&lt;li&gt;Planejamento físico.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Structured Streaming não elimina esses fundamentos.&lt;/p&gt;

&lt;p&gt;Ele os utiliza em um modelo de execução contínua.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Microbatch não significa latência zero
&lt;/h2&gt;

&lt;p&gt;Imagine uma consulta configurada para iniciar novos microbatches aproximadamente a cada cinco segundos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trigger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;processingTime&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;5 seconds&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso não garante que todos os eventos serão processados em exatamente cinco segundos.&lt;/p&gt;

&lt;p&gt;A latência depende de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Disponibilidade dos dados.&lt;/li&gt;
&lt;li&gt;Tempo de processamento.&lt;/li&gt;
&lt;li&gt;Scheduling.&lt;/li&gt;
&lt;li&gt;Capacidade do cluster.&lt;/li&gt;
&lt;li&gt;Shuffle.&lt;/li&gt;
&lt;li&gt;Escrita no destino.&lt;/li&gt;
&lt;li&gt;Backpressure.&lt;/li&gt;
&lt;li&gt;Comportamento da fonte.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Se um microbatch demora mais do que o intervalo configurado, a consulta pode acumular atraso.&lt;/p&gt;

&lt;p&gt;Portanto, precisamos distinguir:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Trigger Interval&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Frequência desejada para verificar e iniciar processamento.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Processing Time&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Tempo necessário para executar um microbatch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;End-to-End Latency&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Tempo entre a ocorrência do evento e sua disponibilidade no destino.&lt;/p&gt;

&lt;p&gt;Essas métricas não são equivalentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Sources e Sinks
&lt;/h2&gt;

&lt;p&gt;Uma consulta Structured Streaming possui três componentes conceituais:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SOURCE
   |
   v
TRANSFORMATION
   |
   v
SINK
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A &lt;strong&gt;Source&lt;/strong&gt; fornece os dados.&lt;/p&gt;

&lt;p&gt;Exemplos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Kafka.&lt;/li&gt;
&lt;li&gt;Arquivos.&lt;/li&gt;
&lt;li&gt;Rate source para testes.&lt;/li&gt;
&lt;li&gt;Outras fontes compatíveis.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As &lt;strong&gt;Transformations&lt;/strong&gt; representam as operações Spark.&lt;/p&gt;

&lt;p&gt;Exemplos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Filter.&lt;/li&gt;
&lt;li&gt;Select.&lt;/li&gt;
&lt;li&gt;Aggregation.&lt;/li&gt;
&lt;li&gt;Join.&lt;/li&gt;
&lt;li&gt;Window.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O &lt;strong&gt;Sink&lt;/strong&gt; recebe o resultado.&lt;/p&gt;

&lt;p&gt;Exemplos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Console.&lt;/li&gt;
&lt;li&gt;Arquivos.&lt;/li&gt;
&lt;li&gt;Tabelas compatíveis.&lt;/li&gt;
&lt;li&gt;Kafka.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;foreachBatch&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As combinações permitidas dependem das operações, do modo de saída e das capacidades de cada conector.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Primeiro laboratório: streaming sem Kafka
&lt;/h2&gt;

&lt;p&gt;Vamos começar com um exemplo executável localmente.&lt;/p&gt;

&lt;p&gt;Utilizaremos a fonte &lt;code&gt;rate&lt;/code&gt;, que gera linhas contendo um timestamp e um valor numérico.&lt;/p&gt;

&lt;p&gt;Ela é útil para aprendizado e testes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;master&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;local[2]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;StructuredStreamingLab&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;eventos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;readStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rowsPerSecond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa fonte disponibiliza duas colunas principais:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;timestamp
value
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora podemos transformar os dados:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;functions&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;

&lt;span class="n"&gt;eventos_transformados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;categoria&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAR&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;otherwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;IMPAR&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ainda não iniciamos a execução.&lt;/p&gt;

&lt;p&gt;Assim como nos DataFrames batch, definimos um plano lógico de transformação.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Iniciando uma streaming query
&lt;/h2&gt;

&lt;p&gt;Para executar o processamento, utilizamos &lt;code&gt;writeStream&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos_transformados&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writeStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;console&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;outputMode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;append&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;truncate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trigger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;processingTime&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;5 seconds&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Spark inicia uma consulta streaming.&lt;/p&gt;

&lt;p&gt;O console passa a apresentar novos registros processados em microbatches.&lt;/p&gt;

&lt;p&gt;Para encerrar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em uma aplicação de longa duração, podemos utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;awaitTermination&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse método bloqueia a execução aguardando o encerramento da consulta.&lt;/p&gt;

&lt;p&gt;No laboratório, execute &lt;code&gt;stop()&lt;/code&gt; em outra célula ou utilize um encerramento programado para não manter o processo indefinidamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. O papel dos Output Modes
&lt;/h2&gt;

&lt;p&gt;Structured Streaming oferece modos de saída para determinadas consultas.&lt;/p&gt;

&lt;p&gt;Os três principais são:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Append&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Emite novas linhas de resultado quando elas podem ser consideradas finais para a operação suportada.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Emite linhas que foram atualizadas desde o último trigger, conforme a consulta e o sink.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Complete&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Emite a tabela de resultados completa a cada trigger.&lt;/p&gt;

&lt;p&gt;A disponibilidade desses modos depende do tipo de operação.&lt;/p&gt;

&lt;p&gt;Por exemplo, uma agregação streaming sem watermark pode precisar manter resultados atualizáveis continuamente.&lt;/p&gt;

&lt;p&gt;Nesse cenário, &lt;code&gt;append&lt;/code&gt; não é automaticamente apropriado.&lt;/p&gt;

&lt;p&gt;É importante validar a compatibilidade do modo com a consulta.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Event Time vs. Processing Time
&lt;/h2&gt;

&lt;p&gt;Agora chegamos a um dos conceitos mais importantes de streaming.&lt;/p&gt;

&lt;p&gt;Imagine um evento que ocorreu às 10h00.&lt;/p&gt;

&lt;p&gt;Mas chegou ao Spark às 10h07.&lt;/p&gt;

&lt;p&gt;Temos dois tempos diferentes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Event Time&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;O momento em que o evento aconteceu.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Processing Time&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;O momento em que o sistema o processou.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;EVENT OCCURS
   10:00
     |
     |
     | Network / Queue / Delay
     |
     v
EVENT PROCESSED
   10:07
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa diferença é fundamental para análises temporais.&lt;/p&gt;

&lt;p&gt;Imagine calcular o número de transações por minuto.&lt;/p&gt;

&lt;p&gt;Devemos agrupar pela hora em que a transação ocorreu ou pela hora em que foi processada?&lt;/p&gt;

&lt;p&gt;A resposta depende do requisito de negócio.&lt;/p&gt;

&lt;p&gt;Para muitas análises de eventos, queremos utilizar Event Time.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. Eventos fora de ordem
&lt;/h2&gt;

&lt;p&gt;Em sistemas distribuídos, eventos nem sempre chegam na mesma ordem em que ocorreram.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Arrival Order:

Event A → 10:05
Event B → 10:03
Event C → 10:04
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O evento B chegou depois de A, embora tenha ocorrido antes.&lt;/p&gt;

&lt;p&gt;Isso pode acontecer por:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Latência de rede.&lt;/li&gt;
&lt;li&gt;Retries.&lt;/li&gt;
&lt;li&gt;Buffers.&lt;/li&gt;
&lt;li&gt;Partições de mensageria.&lt;/li&gt;
&lt;li&gt;Falhas temporárias.&lt;/li&gt;
&lt;li&gt;Processamento assíncrono.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Se a aplicação assumir que os eventos chegam ordenados, os resultados podem ficar incorretos.&lt;/p&gt;

&lt;p&gt;Precisamos de uma estratégia para lidar com atrasos e desordenação.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. Window Aggregations
&lt;/h2&gt;

&lt;p&gt;Structured Streaming permite realizar agregações por janelas temporais.&lt;/p&gt;

&lt;p&gt;Imagine contar eventos em intervalos de um minuto.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;contagem_por_janela&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1 minute&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa consulta cria grupos baseados no timestamp dos eventos.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10:00 ───────── 10:01
       Window 1

10:01 ───────── 10:02
       Window 2

10:02 ───────── 10:03
       Window 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os eventos são associados às janelas correspondentes.&lt;/p&gt;

&lt;p&gt;Mas surge uma pergunta.&lt;/p&gt;

&lt;p&gt;Quando podemos considerar que uma janela está completa?&lt;/p&gt;

&lt;p&gt;Se eventos atrasados podem chegar indefinidamente, o resultado também pode continuar mudando.&lt;/p&gt;

&lt;p&gt;É aqui que entram Watermarks.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. O que é Watermark?
&lt;/h2&gt;

&lt;p&gt;Uma &lt;strong&gt;Watermark&lt;/strong&gt; é um mecanismo utilizado pelo Spark para acompanhar o progresso do tempo dos eventos e estabelecer limites para o tratamento de dados atrasados em operações compatíveis.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;eventos_com_watermark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withWatermark&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;10 minutes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso não significa simplesmente:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Descartar todo evento que chegou mais de dez minutos depois de acontecer.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A semântica é mais específica.&lt;/p&gt;

&lt;p&gt;O mecanismo acompanha o maior Event Time observado e utiliza o atraso configurado para determinar o avanço da watermark.&lt;/p&gt;

&lt;p&gt;De forma conceitual, em uma consulta com uma única watermark:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Watermark ≈
Maximum Observed Event Time
-
Allowed Delay
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O comportamento exato depende da operação, da execução e da política de watermark utilizada.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. Como a Watermark afeta as janelas?
&lt;/h2&gt;

&lt;p&gt;Imagine uma janela:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10:00 até 10:05
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E uma watermark que já avançou para além do encerramento dessa janela.&lt;/p&gt;

&lt;p&gt;Em uma agregação por janela compatível, o Spark pode considerar que não precisa mais manter aquele estado.&lt;/p&gt;

&lt;p&gt;Isso permite liberar recursos.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Event-Time Timeline

10:00      10:05      10:10      10:15
  |----------|----------|----------|
   Window A

                            ^
                         Watermark
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Quando a watermark ultrapassa o fim da janela, o resultado pode ser finalizado e o estado correspondente removido.&lt;/p&gt;

&lt;p&gt;Esse mecanismo é importante porque streams podem durar meses ou anos.&lt;/p&gt;

&lt;p&gt;Sem políticas adequadas, determinadas operações stateful poderiam acumular estado indefinidamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Exemplo de agregação com Watermark
&lt;/h2&gt;

&lt;p&gt;Vamos construir uma consulta que conta eventos por janelas de um minuto.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;eventos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;readStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rowsPerSecond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withWatermark&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2 minutes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1 minute&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora iniciamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;resultado&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writeStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;console&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;outputMode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;update&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;truncate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trigger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;processingTime&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;5 seconds&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Utilizamos &lt;code&gt;update&lt;/code&gt; para observar resultados parciais das janelas.&lt;/p&gt;

&lt;p&gt;Se utilizássemos &lt;code&gt;append&lt;/code&gt;, os resultados seriam emitidos quando as janelas estivessem prontas para finalização, conforme o avanço da watermark.&lt;/p&gt;

&lt;p&gt;Isso pode produzir uma espera perceptível.&lt;/p&gt;

&lt;p&gt;Para encerrar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  16. Watermark não garante aceitar todos os eventos dentro do atraso
&lt;/h2&gt;

&lt;p&gt;Existe uma sutileza importante.&lt;/p&gt;

&lt;p&gt;O atraso configurado é utilizado para controlar a evolução da watermark e a retenção de estado.&lt;/p&gt;

&lt;p&gt;Ele não representa uma garantia absoluta de que todo evento com atraso inferior ao intervalo será aceito em qualquer circunstância.&lt;/p&gt;

&lt;p&gt;O comportamento depende do avanço do tempo de evento observado e da operação executada.&lt;/p&gt;

&lt;p&gt;Além disso, o Spark pode descartar registros considerados atrasados para determinadas operações stateful.&lt;/p&gt;

&lt;p&gt;Por isso, a escolha da watermark deve considerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Distribuição real de atrasos.&lt;/li&gt;
&lt;li&gt;Requisitos de completude.&lt;/li&gt;
&lt;li&gt;Memória disponível.&lt;/li&gt;
&lt;li&gt;Latência desejada.&lt;/li&gt;
&lt;li&gt;Tolerância a correções posteriores.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Watermarks representam um compromisso entre completude, latência e consumo de estado.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Stateful vs. Stateless Processing
&lt;/h2&gt;

&lt;p&gt;Nem toda transformação streaming precisa manter informações de eventos anteriores.&lt;/p&gt;

&lt;p&gt;Uma transformação stateless processa registros sem depender de estado acumulado entre microbatches.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;filtrado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;eventos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Já uma agregação precisa manter informações entre microbatches.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;agregado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1 minute&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;STATELESS

Event → Transformation → Output
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;STATEFUL

Event → Transformation
              |
              v
          State Store
              |
              v
            Output
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Operações stateful são mais complexas porque exigem gerenciamento de estado e recuperação consistente.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. State Store
&lt;/h2&gt;

&lt;p&gt;O Structured Streaming utiliza mecanismos de armazenamento de estado para determinadas operações.&lt;/p&gt;

&lt;p&gt;O estado pode representar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Contagens.&lt;/li&gt;
&lt;li&gt;Agregações.&lt;/li&gt;
&lt;li&gt;Informações para deduplicação.&lt;/li&gt;
&lt;li&gt;Dados necessários a joins streaming.&lt;/li&gt;
&lt;li&gt;Outros valores mantidos entre microbatches.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Microbatch 1
     |
     v
State Update
     |
     v
State Store

Microbatch 2
     |
     v
Read Previous State
     |
     v
Update State
     |
     v
State Store
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Spark oferece diferentes implementações e opções de configuração para armazenamento de estado, dependendo da versão.&lt;/p&gt;

&lt;p&gt;O volume de estado influencia diretamente:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Memória.&lt;/li&gt;
&lt;li&gt;Disco.&lt;/li&gt;
&lt;li&gt;Checkpoint.&lt;/li&gt;
&lt;li&gt;Tempo de processamento.&lt;/li&gt;
&lt;li&gt;Recuperação de falhas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por isso, operações stateful exigem atenção especial em produção.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. Checkpoints: a base da recuperação
&lt;/h2&gt;

&lt;p&gt;Imagine que uma consulta streaming esteja processando eventos continuamente.&lt;/p&gt;

&lt;p&gt;De repente, o processo falha.&lt;/p&gt;

&lt;p&gt;Como saber quais dados já foram processados?&lt;/p&gt;

&lt;p&gt;Como recuperar o estado de uma agregação?&lt;/p&gt;

&lt;p&gt;Como evitar reprocessamentos inconsistentes?&lt;/p&gt;

&lt;p&gt;O Structured Streaming utiliza &lt;strong&gt;checkpoints&lt;/strong&gt; para persistir informações necessárias à recuperação de consultas compatíveis.&lt;/p&gt;

&lt;p&gt;Um checkpoint pode conter informações sobre:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Progresso da consulta.&lt;/li&gt;
&lt;li&gt;Offsets ou posições da fonte.&lt;/li&gt;
&lt;li&gt;Commits.&lt;/li&gt;
&lt;li&gt;Estado de operações stateful.&lt;/li&gt;
&lt;li&gt;Metadados da execução.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A estrutura exata depende da versão e do tipo de consulta.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Configurando Checkpoint Location
&lt;/h2&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos_transformados&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writeStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;path&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/streaming/output&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;checkpointLocation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/streaming/checkpoint&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;outputMode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;append&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O diretório de checkpoint deve ser persistente e acessível ao ambiente de execução.&lt;/p&gt;

&lt;p&gt;Em produção, não devemos depender de um diretório temporário local que possa desaparecer após a reinicialização do container ou nó.&lt;/p&gt;

&lt;p&gt;Também devemos evitar compartilhar o mesmo checkpoint entre consultas independentes.&lt;/p&gt;

&lt;p&gt;O checkpoint representa o estado de uma consulta específica.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. O que acontece após uma falha?
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Microbatch 1 → Completed
Microbatch 2 → Completed
Microbatch 3 → Failure
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A aplicação é reiniciada com uma configuração compatível e o mesmo checkpoint.&lt;/p&gt;

&lt;p&gt;O Spark utiliza as informações persistidas para recuperar o progresso e o estado da consulta.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;STREAMING QUERY
      |
      v
CHECKPOINT
      |
      v
PROCESS FAILURE
      |
      v
RESTART
      |
      v
RECOVER PROGRESS
      |
      v
CONTINUE PROCESSING
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas essa recuperação depende de condições importantes.&lt;/p&gt;

&lt;p&gt;A fonte precisa permitir a recuperação das posições necessárias.&lt;/p&gt;

&lt;p&gt;O sink precisa oferecer garantias compatíveis.&lt;/p&gt;

&lt;p&gt;E determinadas alterações na consulta ou no schema de estado podem impedir a recuperação com o mesmo checkpoint.&lt;/p&gt;

&lt;p&gt;Checkpoint não significa que qualquer modificação no código seja segura durante um restart.&lt;/p&gt;




&lt;h2&gt;
  
  
  22. Exactly-once: o que realmente significa?
&lt;/h2&gt;

&lt;p&gt;Um dos termos mais utilizados em streaming é &lt;strong&gt;exactly-once&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Mas precisamos ser precisos.&lt;/p&gt;

&lt;p&gt;Existem diferentes semânticas de processamento e entrega.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;At-most-once&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Um evento pode ser perdido, mas não é necessariamente reprocessado.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;At-least-once&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Um evento pode ser processado novamente após falhas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Exactly-once&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Busca garantir que o efeito lógico do processamento seja aplicado uma única vez, dentro das condições e limites da arquitetura.&lt;/p&gt;

&lt;p&gt;Structured Streaming pode oferecer garantias end-to-end de exactly-once em determinadas combinações de fontes, operações e sinks.&lt;/p&gt;

&lt;p&gt;Mas não podemos generalizar essa garantia para qualquer destino externo.&lt;/p&gt;

&lt;p&gt;Por exemplo, um &lt;code&gt;foreachBatch&lt;/code&gt; que envia mensagens para uma API pode produzir duplicidades se a aplicação falhar após a chamada externa e antes de registrar o progresso.&lt;/p&gt;

&lt;p&gt;Nesse caso, precisamos implementar idempotência.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. &lt;code&gt;foreachBatch&lt;/code&gt; e idempotência
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;foreachBatch&lt;/code&gt; permite aplicar uma função a cada microbatch.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;processar_batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch_df&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;batch_id&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Processando batch &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;batch_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;batch_df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;append&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/streaming/destino&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Iniciamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writeStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;foreachBatch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;processar_batch&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;checkpointLocation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/streaming/checkpoint_foreach&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Atenção:&lt;/strong&gt; esse exemplo demonstra a API, mas não implementa exactly-once no destino Parquet.&lt;/p&gt;

&lt;p&gt;Se ocorrer uma falha entre a escrita e a confirmação do progresso, o microbatch pode ser executado novamente e gerar dados duplicados.&lt;/p&gt;

&lt;p&gt;Uma estratégia de produção pode utilizar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Identificador de batch.&lt;/li&gt;
&lt;li&gt;Escrita transacional.&lt;/li&gt;
&lt;li&gt;Controle de commits.&lt;/li&gt;
&lt;li&gt;Chaves idempotentes.&lt;/li&gt;
&lt;li&gt;Deduplicação.&lt;/li&gt;
&lt;li&gt;Operações MERGE.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O desenho precisa considerar as garantias reais do destino.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Kafka e Structured Streaming
&lt;/h2&gt;

&lt;p&gt;Agora vamos conectar o Spark a uma arquitetura de eventos.&lt;/p&gt;

&lt;p&gt;Kafka é frequentemente utilizado como plataforma de mensageria e streaming.&lt;/p&gt;

&lt;p&gt;Uma arquitetura comum:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;APPLICATIONS
     |
     v
   KAFKA
     |
     v
SPARK STRUCTURED STREAMING
     |
     v
TRANSFORMATIONS
     |
     v
LAKEHOUSE / ANALYTICS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Kafka organiza mensagens em tópicos e partições.&lt;/p&gt;

&lt;p&gt;O Spark pode consumir esses dados por meio de um conector compatível.&lt;/p&gt;

&lt;p&gt;A integração exige dependências específicas, além de configuração de rede, segurança e permissões.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. Exemplo de leitura Kafka
&lt;/h2&gt;

&lt;p&gt;Com o conector Spark Kafka instalado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;eventos_kafka&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;readStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;kafka&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;kafka.bootstrap.servers&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;localhost:9092&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;subscribe&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedidos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;startingOffsets&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;earliest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O DataFrame disponibiliza campos como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;key
value
topic
partition
offset
timestamp
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As colunas &lt;code&gt;key&lt;/code&gt; e &lt;code&gt;value&lt;/code&gt; são representadas como dados binários.&lt;/p&gt;

&lt;p&gt;Podemos convertê-las:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;mensagens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;eventos_kafka&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;cast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;string&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;cast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;string&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;payload&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;topic&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;partition&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;offset&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois podemos interpretar o payload conforme o contrato de dados.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. Interpretando eventos JSON do Kafka
&lt;/h2&gt;

&lt;p&gt;Imagine mensagens com a seguinte estrutura:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pedido_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"cliente_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"valor"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;150.75&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PAGO"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Definimos o schema:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.types&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;DecimalType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StringType&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;schema_pedido&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;DecimalType&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Aplicamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;pedidos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;mensagens&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dados&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;payload&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="n"&gt;schema_pedido&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dados.*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;topic&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;partition&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;offset&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora temos um DataFrame estruturado.&lt;/p&gt;

&lt;p&gt;Mas precisamos tratar payloads inválidos, campos ausentes e alterações de schema.&lt;/p&gt;

&lt;p&gt;A integração Kafka–Spark não elimina as responsabilidades de Data Quality e Data Contracts estudadas na Parte 14.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Deduplicação em streaming
&lt;/h2&gt;

&lt;p&gt;Eventos duplicados podem surgir por diferentes motivos.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Event ID 100
Event ID 101
Event ID 100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se cada evento representar uma transação, processar duplicidades pode distorcer os resultados.&lt;/p&gt;

&lt;p&gt;Structured Streaming oferece operações de deduplicação, mas a estratégia precisa considerar estado e tempo.&lt;/p&gt;

&lt;p&gt;Uma abordagem com suporte nas versões compatíveis é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;eventos_unicos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos_com_id&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withWatermark&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_time&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;10 minutes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dropDuplicatesWithinWatermark&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;dropDuplicatesWithinWatermark&lt;/code&gt; está disponível em versões modernas do Spark, incluindo Spark 3.5 ou posterior.&lt;/p&gt;

&lt;p&gt;Esse mecanismo permite limitar o estado de deduplicação utilizando a watermark.&lt;/p&gt;

&lt;p&gt;Mas não oferece deduplicação eterna de qualquer evento.&lt;/p&gt;

&lt;p&gt;Se o requisito exigir unicidade global por tempo indefinido, precisaremos de uma estratégia diferente.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Streaming Joins
&lt;/h2&gt;

&lt;p&gt;Também podemos combinar dados de diferentes fontes.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Orders Stream
      |
      +------ JOIN ------+
      |                  |
      v                  v
Orders Data         Customers Data
      |                  |
      +--------+---------+
               |
               v
         Enriched Orders
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Existem diferentes tipos de joins em Structured Streaming.&lt;/p&gt;

&lt;p&gt;Um join entre stream e tabela estática possui características diferentes de um join entre dois streams.&lt;/p&gt;

&lt;p&gt;Em joins stream-stream, frequentemente precisamos controlar o crescimento do estado utilizando restrições temporais e watermarks.&lt;/p&gt;

&lt;p&gt;Sem limites adequados, o estado pode crescer significativamente.&lt;/p&gt;

&lt;p&gt;Por isso, a escolha do join deve considerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Semântica temporal.&lt;/li&gt;
&lt;li&gt;Atrasos.&lt;/li&gt;
&lt;li&gt;Cardinalidade.&lt;/li&gt;
&lt;li&gt;Volume.&lt;/li&gt;
&lt;li&gt;Retenção de estado.&lt;/li&gt;
&lt;li&gt;Requisitos de completude.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  29. Backpressure e atraso de processamento
&lt;/h2&gt;

&lt;p&gt;Imagine que Kafka produz eventos mais rapidamente do que Spark consegue processar.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Incoming Events: 100,000/s

Spark Capacity:   60,000/s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A diferença pode produzir acúmulo de dados pendentes.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PRODUCER RATE
      |
      v
   EVENT QUEUE
      |
      v
SPARK PROCESSING
      |
      v
CONSUMER OUTPUT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se a capacidade de processamento for inferior à taxa de entrada durante muito tempo, a latência tende a aumentar.&lt;/p&gt;

&lt;p&gt;Precisamos acompanhar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Taxa de entrada.&lt;/li&gt;
&lt;li&gt;Taxa de processamento.&lt;/li&gt;
&lt;li&gt;Duração dos microbatches.&lt;/li&gt;
&lt;li&gt;Atraso na fonte.&lt;/li&gt;
&lt;li&gt;Estado acumulado.&lt;/li&gt;
&lt;li&gt;Capacidade dos Executors.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Em determinadas fontes, podemos limitar a quantidade de dados processados por trigger.&lt;/p&gt;

&lt;p&gt;Por exemplo, o conector Kafka oferece configurações como &lt;code&gt;maxOffsetsPerTrigger&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Mas limitar a ingestão não resolve a falta estrutural de capacidade.&lt;/p&gt;

&lt;p&gt;Apenas controla quanto trabalho é admitido por microbatch.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. Observabilidade de Streaming
&lt;/h2&gt;

&lt;p&gt;Em batch, normalmente acompanhamos a duração de cada Job.&lt;/p&gt;

&lt;p&gt;Em streaming, precisamos monitorar uma aplicação de longa duração.&lt;/p&gt;

&lt;p&gt;Structured Streaming disponibiliza métricas de progresso.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lastProgress&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Também podemos consultar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;recentProgress&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Entre os indicadores relevantes estão:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;inputRowsPerSecond&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;processedRowsPerSecond&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;numInputRows&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Duração dos triggers.&lt;/li&gt;
&lt;li&gt;Informações de estado.&lt;/li&gt;
&lt;li&gt;Progresso das fontes.&lt;/li&gt;
&lt;li&gt;Informações do sink.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Essas métricas devem ser interpretadas em conjunto.&lt;/p&gt;

&lt;p&gt;Uma taxa de processamento elevada não garante baixa latência se a fonte estiver acumulando backlog.&lt;/p&gt;




&lt;h2&gt;
  
  
  31. Exemplo de monitoramento
&lt;/h2&gt;

&lt;p&gt;Podemos inspecionar o progresso de uma consulta:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;progresso&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lastProgress&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;progresso&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Input rows/s:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;progresso&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;inputRowsPerSecond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Processed rows/s:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;progresso&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;processedRowsPerSecond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Batch duration:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;progresso&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;durationMs&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em produção, essas informações podem ser enviadas para uma plataforma de observabilidade.&lt;/p&gt;

&lt;p&gt;Também devemos monitorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Falhas.&lt;/li&gt;
&lt;li&gt;Reinicializações.&lt;/li&gt;
&lt;li&gt;Crescimento do checkpoint.&lt;/li&gt;
&lt;li&gt;Uso de memória.&lt;/li&gt;
&lt;li&gt;Estado.&lt;/li&gt;
&lt;li&gt;Latência ponta a ponta.&lt;/li&gt;
&lt;li&gt;Qualidade dos eventos.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A observabilidade precisa abranger tanto o Spark quanto a fonte e o destino.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Streaming e Lakehouse
&lt;/h2&gt;

&lt;p&gt;Agora podemos conectar as Partes 13, 14 e 15.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OPERATIONAL DATABASE
        |
        v
       CDC
        |
        v
      KAFKA
        |
        v
SPARK STRUCTURED STREAMING
        |
        v
DATA CONTRACT VALIDATION
        |
        v
DATA QUALITY RULES
        |
        v
LAKEHOUSE TABLE
        |
        v
BI / ANALYTICS / ML
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa arquitetura pode oferecer ingestão incremental e disponibilização contínua de dados analíticos.&lt;/p&gt;

&lt;p&gt;Mas existem responsabilidades importantes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Idempotência.&lt;/li&gt;
&lt;li&gt;Evolução de schema.&lt;/li&gt;
&lt;li&gt;Controle de estado.&lt;/li&gt;
&lt;li&gt;Checkpoints.&lt;/li&gt;
&lt;li&gt;Qualidade.&lt;/li&gt;
&lt;li&gt;Consistência transacional.&lt;/li&gt;
&lt;li&gt;Monitoramento.&lt;/li&gt;
&lt;li&gt;Recuperação de falhas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A combinação de Spark, Kafka e um formato Lakehouse não garante automaticamente que todos esses aspectos estejam resolvidos.&lt;/p&gt;

&lt;p&gt;Precisamos projetá-los.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. Streaming não é sempre a melhor escolha
&lt;/h2&gt;

&lt;p&gt;Existe uma tendência de associar streaming a arquiteturas mais modernas.&lt;/p&gt;

&lt;p&gt;Mas nem todo problema exige processamento contínuo.&lt;/p&gt;

&lt;p&gt;Imagine um relatório financeiro atualizado uma vez por dia.&lt;/p&gt;

&lt;p&gt;Um pipeline batch pode ser mais simples, barato e confiável.&lt;/p&gt;

&lt;p&gt;Agora imagine uma plataforma que precisa detectar transações suspeitas em poucos segundos.&lt;/p&gt;

&lt;p&gt;Nesse caso, streaming pode ser necessário.&lt;/p&gt;

&lt;p&gt;A decisão deve considerar:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Critério&lt;/th&gt;
&lt;th&gt;Batch&lt;/th&gt;
&lt;th&gt;Streaming&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Latência&lt;/td&gt;
&lt;td&gt;Normalmente maior&lt;/td&gt;
&lt;td&gt;Potencialmente menor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Complexidade operacional&lt;/td&gt;
&lt;td&gt;Geralmente menor&lt;/td&gt;
&lt;td&gt;Geralmente maior&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Estado contínuo&lt;/td&gt;
&lt;td&gt;Não necessariamente&lt;/td&gt;
&lt;td&gt;Pode ser necessário&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recuperação&lt;/td&gt;
&lt;td&gt;Por execução&lt;/td&gt;
&lt;td&gt;Por progresso e checkpoint&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Monitoramento&lt;/td&gt;
&lt;td&gt;Jobs e SLAs&lt;/td&gt;
&lt;td&gt;Progresso contínuo e backlog&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Custo&lt;/td&gt;
&lt;td&gt;Depende da frequência&lt;/td&gt;
&lt;td&gt;Depende da operação contínua&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Streaming não é automaticamente mais rápido ou mais barato.&lt;/p&gt;

&lt;p&gt;Ele atende a requisitos diferentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Microbatch vs. processamento de baixa latência
&lt;/h2&gt;

&lt;p&gt;O Structured Streaming oferece diferentes possibilidades de execução, mas precisamos evitar confundir os conceitos.&lt;/p&gt;

&lt;p&gt;O microbatch é o mecanismo predominante e mais amplamente utilizado.&lt;/p&gt;

&lt;p&gt;Existem recursos específicos para execução contínua em determinadas versões e operações, com limitações próprias.&lt;/p&gt;

&lt;p&gt;Além disso, tecnologias especializadas em streaming podem oferecer modelos diferentes de processamento e gerenciamento de estado.&lt;/p&gt;

&lt;p&gt;A escolha entre Spark e outras engines deve considerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Latência exigida.&lt;/li&gt;
&lt;li&gt;Volume.&lt;/li&gt;
&lt;li&gt;Semântica de processamento.&lt;/li&gt;
&lt;li&gt;Complexidade de estado.&lt;/li&gt;
&lt;li&gt;Operações necessárias.&lt;/li&gt;
&lt;li&gt;Ecossistema.&lt;/li&gt;
&lt;li&gt;Custo operacional.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Se o requisito for latência de poucos milissegundos, não devemos assumir que uma arquitetura Spark baseada em microbatches seja a melhor solução.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. Um laboratório completo de streaming local
&lt;/h2&gt;

&lt;p&gt;Vamos finalizar com um exemplo independente que gera eventos, transforma os dados e grava arquivos Parquet.&lt;/p&gt;

&lt;p&gt;Esse laboratório não exige Kafka.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;functions&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sleep&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;master&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;local[2]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;StreamingEndToEnd&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;fonte&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;readStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rowsPerSecond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;eventos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;fonte&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_time&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;categoria_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1.5&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;eventos&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writeStream&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;outputMode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;append&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;path&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/spark-streaming-lab/output&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;checkpointLocation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/spark-streaming-lab/checkpoint&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trigger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;processingTime&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;5 seconds&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/spark-streaming-lab/output&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;truncate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Registros gravados:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse exemplo demonstra:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Criação de uma fonte streaming.&lt;/li&gt;
&lt;li&gt;Transformação com DataFrames.&lt;/li&gt;
&lt;li&gt;Execução por microbatches.&lt;/li&gt;
&lt;li&gt;Escrita em Parquet.&lt;/li&gt;
&lt;li&gt;Configuração de checkpoint.&lt;/li&gt;
&lt;li&gt;Encerramento da consulta.&lt;/li&gt;
&lt;li&gt;Leitura batch dos resultados.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A quantidade de registros produzidos pode variar conforme o tempo de inicialização e a execução dos microbatches.&lt;/p&gt;

&lt;p&gt;Para repetir o laboratório desde o início, utilize novos diretórios de saída e checkpoint ou limpe os diretórios de teste de maneira controlada.&lt;/p&gt;

&lt;p&gt;Em produção, nunca elimine checkpoints de uma consulta ativa ou persistente sem avaliar as consequências para recuperação e reprocessamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  36. Erros comuns em Structured Streaming
&lt;/h2&gt;

&lt;p&gt;Alguns problemas aparecem frequentemente.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Acreditar que streaming significa tempo real instantâneo.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A latência depende do mecanismo e da infraestrutura.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não configurar checkpoints persistentes.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso pode comprometer a recuperação.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Confundir Event Time com Processing Time.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso pode produzir agregações temporalmente incorretas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Definir watermarks sem analisar atrasos reais.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso pode causar perda de eventos relevantes nas operações afetadas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ignorar crescimento do estado.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Operações stateful podem consumir recursos crescentes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Assumir exactly-once para qualquer sink.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;As garantias dependem da arquitetura completa.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não implementar idempotência em &lt;code&gt;foreachBatch&lt;/code&gt;.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Retries podem produzir efeitos duplicados.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não monitorar backlog.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Uma consulta ativa pode estar cada vez mais atrasada.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Utilizar streaming sem necessidade de baixa latência.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso pode aumentar complexidade e custo desnecessariamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  37. Como eu projetaria um pipeline streaming em produção
&lt;/h2&gt;

&lt;p&gt;Antes de implementar, começaria pelas perguntas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual é a latência exigida?
          |
          v
Qual é a fonte?
          |
          v
Qual é a taxa de eventos?
          |
          v
Existe Event Time?
          |
          v
Qual é a distribuição de atrasos?
          |
          v
Existem operações stateful?
          |
          v
Qual é o limite de estado?
          |
          v
Como funcionam checkpoints?
          |
          v
Qual é a garantia do sink?
          |
          v
Como implementar idempotência?
          |
          v
Como monitorar backlog?
          |
          v
Como recuperar falhas?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Só depois definiria a arquitetura e as configurações.&lt;/p&gt;

&lt;p&gt;Essa abordagem reduz o risco de construir um pipeline tecnicamente funcional, mas operacionalmente frágil.&lt;/p&gt;




&lt;h2&gt;
  
  
  38. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;Structured Streaming amplia o modelo de processamento do Spark para dados que chegam continuamente.&lt;/p&gt;

&lt;p&gt;Mas streaming confiável exige mais do que utilizar &lt;code&gt;readStream&lt;/code&gt; e &lt;code&gt;writeStream&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Precisamos compreender:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Microbatches.&lt;/li&gt;
&lt;li&gt;Event Time.&lt;/li&gt;
&lt;li&gt;Processing Time.&lt;/li&gt;
&lt;li&gt;Watermarks.&lt;/li&gt;
&lt;li&gt;Estado.&lt;/li&gt;
&lt;li&gt;Checkpoints.&lt;/li&gt;
&lt;li&gt;Semântica de entrega.&lt;/li&gt;
&lt;li&gt;Idempotência.&lt;/li&gt;
&lt;li&gt;Observabilidade.&lt;/li&gt;
&lt;li&gt;Recuperação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O princípio mais importante é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Streaming não é apenas processar eventos rapidamente. É manter resultados corretos e recuperáveis enquanto os dados continuam chegando.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  39. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando avalio uma arquitetura streaming, não pergunto apenas:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Quantos eventos por segundo ela consegue processar?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Também quero saber:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qual é a latência ponta a ponta?&lt;/li&gt;
&lt;li&gt;Como eventos atrasados são tratados?&lt;/li&gt;
&lt;li&gt;Qual é o tamanho do estado?&lt;/li&gt;
&lt;li&gt;O que acontece após uma falha?&lt;/li&gt;
&lt;li&gt;Quais garantias o destino oferece?&lt;/li&gt;
&lt;li&gt;Como evitamos duplicidades?&lt;/li&gt;
&lt;li&gt;Como identificamos backlog?&lt;/li&gt;
&lt;li&gt;Como reprocessamos eventos?&lt;/li&gt;
&lt;li&gt;Como validamos contratos e qualidade?&lt;/li&gt;
&lt;li&gt;Qual é o custo operacional?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Essas perguntas representam a diferença entre uma demonstração de streaming e uma plataforma de dados confiável.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora entendemos os fundamentos do Structured Streaming.&lt;/p&gt;

&lt;p&gt;Na próxima parte, vamos aprofundar um dos cenários mais importantes de integração de dados:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Apache Kafka, CDC e pipelines incrementais com Spark.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Vamos explorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Arquitetura Kafka.&lt;/li&gt;
&lt;li&gt;Topics, Partitions e Offsets.&lt;/li&gt;
&lt;li&gt;Consumer Groups.&lt;/li&gt;
&lt;li&gt;CDC com Debezium.&lt;/li&gt;
&lt;li&gt;Integração com PostgreSQL.&lt;/li&gt;
&lt;li&gt;Eventos de INSERT, UPDATE e DELETE.&lt;/li&gt;
&lt;li&gt;Ordenação e deduplicação.&lt;/li&gt;
&lt;li&gt;Evolução de schema.&lt;/li&gt;
&lt;li&gt;Upserts em Lakehouse.&lt;/li&gt;
&lt;li&gt;Idempotência.&lt;/li&gt;
&lt;li&gt;Replay e recuperação.&lt;/li&gt;
&lt;li&gt;Observabilidade.&lt;/li&gt;
&lt;li&gt;Arquitetura end-to-end.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central será:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como capturar mudanças de sistemas transacionais e disponibilizá-las em uma plataforma analítica sem depender de cargas completas?&lt;/strong&gt;&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 14:&lt;/strong&gt; Data Quality, Data Contracts e Governança&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 16 — Apache Kafka, CDC e pipelines incrementais com Spark&lt;/p&gt;

</description>
      <category>bigdata</category>
      <category>python</category>
      <category>sql</category>
      <category>apache</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 14: Data Quality, Data Contracts e Governança</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 20:06:57 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-14-data-quality-data-contracts-e-governanca-3j9f</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-14-data-quality-data-contracts-e-governanca-3j9f</guid>
      <description>&lt;h1&gt;
  
  
  Como aprendi Apache Spark — Parte 14: Data Quality, Data Contracts e Governança
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Nota histórica: esta série revisita minha trajetória em Engenharia de Dados iniciada em 2012 e conecta os fundamentos daquela experiência às práticas e tecnologias modernas. Data Contracts, observabilidade e governança evoluíram significativamente desde então.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nos primeiros capítulos desta série, concentramos nossa atenção em como o Apache Spark executa o processamento distribuído.&lt;/p&gt;

&lt;p&gt;Estudamos Driver, Executors, Jobs, Stages, Tasks, partições, shuffle, memória e tolerância a falhas.&lt;/p&gt;

&lt;p&gt;Depois avançamos para dimensionamento, formatos de armazenamento e arquiteturas Lakehouse.&lt;/p&gt;

&lt;p&gt;Mas existe uma questão que considero tão importante quanto a performance:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O que acontece quando um pipeline executa perfeitamente, mas entrega dados errados?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Imagine um processamento que termina em cinco minutos, utiliza poucos recursos, não apresenta falhas e grava todos os arquivos esperados.&lt;/p&gt;

&lt;p&gt;Tecnicamente, a execução foi bem-sucedida.&lt;/p&gt;

&lt;p&gt;Entretanto, o dataset final contém pedidos duplicados, valores negativos indevidos, datas inconsistentes e registros sem identificadores.&lt;/p&gt;

&lt;p&gt;O Spark executou corretamente as instruções recebidas.&lt;/p&gt;

&lt;p&gt;Mas o produto de dados está incorreto.&lt;/p&gt;

&lt;p&gt;Essa diferença nos leva a três disciplinas fundamentais:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Data Quality:&lt;/strong&gt; verificar se os dados atendem aos requisitos definidos.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Data Contracts:&lt;/strong&gt; formalizar expectativas e responsabilidades entre produtores e consumidores.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Data Governance:&lt;/strong&gt; estabelecer políticas, responsabilidades, controles e rastreabilidade sobre os dados.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Vamos entender como essas disciplinas se conectam e como implementá-las em pipelines Spark.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Pipeline bem-sucedido não significa dado confiável
&lt;/h2&gt;

&lt;p&gt;Considere um pipeline simples:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Operational Database
        |
        v
    Ingestion
        |
        v
      Spark
        |
        v
     Parquet
        |
        v
   BI Dashboard
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O processamento termina sem erros.&lt;/p&gt;

&lt;p&gt;Mas a origem enviou registros duplicados.&lt;/p&gt;

&lt;p&gt;O Spark pode gravá-los normalmente se não existir nenhuma regra para identificá-los.&lt;/p&gt;

&lt;p&gt;Agora imagine que um dashboard utilize esses registros para calcular faturamento.&lt;/p&gt;

&lt;p&gt;O resultado estará incorreto.&lt;/p&gt;

&lt;p&gt;Esse é um problema de qualidade de dados, não necessariamente de execução distribuída.&lt;/p&gt;

&lt;p&gt;Precisamos distinguir:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Technical Success&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A aplicação executou conforme as instruções.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data Quality Success&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Os dados resultantes satisfazem as regras de qualidade estabelecidas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Business Success&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Os dados permitem decisões corretas e atendem às necessidades dos consumidores.&lt;/p&gt;

&lt;p&gt;Uma plataforma madura precisa avaliar essas três dimensões.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. As principais dimensões de Data Quality
&lt;/h2&gt;

&lt;p&gt;Existem diferentes modelos de classificação de qualidade de dados. Para aplicações Spark, considero especialmente úteis estas dimensões:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Dimensão&lt;/th&gt;
&lt;th&gt;Pergunta&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Completude&lt;/td&gt;
&lt;td&gt;Os campos obrigatórios estão preenchidos?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unicidade&lt;/td&gt;
&lt;td&gt;Existem duplicidades indevidas?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Validade&lt;/td&gt;
&lt;td&gt;Os valores respeitam os domínios e formatos definidos?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consistência&lt;/td&gt;
&lt;td&gt;Os dados respeitam regras entre campos e sistemas?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Atualidade&lt;/td&gt;
&lt;td&gt;Os dados estão disponíveis dentro da janela esperada?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Acurácia&lt;/td&gt;
&lt;td&gt;Os valores representam corretamente a realidade?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Integridade referencial&lt;/td&gt;
&lt;td&gt;As referências apontam para entidades válidas?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Essas dimensões exigem mecanismos diferentes.&lt;/p&gt;

&lt;p&gt;Por exemplo, podemos verificar se um campo está preenchido com Spark.&lt;/p&gt;

&lt;p&gt;Mas determinar se um endereço realmente existe pode exigir uma fonte de referência externa.&lt;/p&gt;

&lt;p&gt;Portanto, qualidade não é apenas executar expressões &lt;code&gt;isNotNull()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;É definir critérios verificáveis para cada contexto.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Construindo um laboratório de qualidade com PySpark
&lt;/h2&gt;

&lt;p&gt;Vamos utilizar um dataset sintético de pedidos.&lt;/p&gt;

&lt;p&gt;O exemplo foi projetado para funcionar em uma instalação PySpark compatível, sem exigir Iceberg, Delta Lake ou bibliotecas externas.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;functions&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.types&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;DecimalType&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;decimal&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Decimal&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;master&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;local[*]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SparkDataQuality&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;DecimalType&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data_pedido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;registros&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;150.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2026-11-01&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;102&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;230.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2026-11-02&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;102&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;230.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2026-11-02&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;99.90&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2026-11-03&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;104&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;DESCONHECIDO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;50.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2026-11-04&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;105&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-100.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2026-11-05&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;106&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;80.00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data-invalida&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createDataFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;registros&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;truncate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse dataset contém problemas intencionais:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Pedido duplicado.&lt;/li&gt;
&lt;li&gt;Cliente ausente.&lt;/li&gt;
&lt;li&gt;Status não permitido.&lt;/li&gt;
&lt;li&gt;Valor negativo.&lt;/li&gt;
&lt;li&gt;Data inválida.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Vamos construir regras para identificá-los.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Definindo regras de validação
&lt;/h2&gt;

&lt;p&gt;Primeiro, precisamos estabelecer o contrato lógico dos pedidos.&lt;/p&gt;

&lt;p&gt;Para este laboratório, vamos considerar:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;pedido_id&lt;/code&gt; obrigatório.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;cliente_id&lt;/code&gt; obrigatório.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;status&lt;/code&gt; pertencente a um domínio conhecido.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;valor&lt;/code&gt; maior ou igual a zero.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;data_pedido&lt;/code&gt; representando uma data válida.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pedido_id&lt;/code&gt; único dentro do dataset.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Essas são regras de exemplo, não universais.&lt;/p&gt;

&lt;p&gt;Em um sistema real, valores negativos podem ser legítimos para estornos ou ajustes.&lt;/p&gt;

&lt;p&gt;A regra deve refletir a semântica do domínio.&lt;/p&gt;

&lt;p&gt;Vamos implementar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Window&lt;/span&gt;

&lt;span class="n"&gt;status_validos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CANCELADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;janela_pedido&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Window&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;partitionBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df_validado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data_convertida&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;try_cast(data_pedido AS DATE)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;qtd_por_pedido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;over&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;janela_pedido&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_cliente_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status_validos&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data_convertida&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_duplicidade&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;qtd_por_pedido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Utilizamos &lt;code&gt;try_cast&lt;/code&gt; para tratar datas inválidas como nulas sem depender de exceções de conversão. O suporte deve ser confirmado na versão de Spark SQL utilizada.&lt;/p&gt;

&lt;p&gt;Para o laboratório, assumimos Spark 3.5 ou posterior.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Criando uma classificação de qualidade
&lt;/h2&gt;

&lt;p&gt;Agora queremos saber se cada registro é válido.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;colunas_erro&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_cliente_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro_duplicidade&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;condicao_invalida&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;coluna&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;colunas_erro&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;condicao_invalida&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;condicao_invalida&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;coluna&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df_classificado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_validado&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registro_valido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="n"&gt;condicao_invalida&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;quality_status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registro_valido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;VALID&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;otherwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;INVALID&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df_classificado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;quality_status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora cada registro possui uma classificação explícita.&lt;/p&gt;

&lt;p&gt;Mas precisamos tomar cuidado com a interpretação.&lt;/p&gt;

&lt;p&gt;Um registro pode falhar em mais de uma regra.&lt;/p&gt;

&lt;p&gt;Portanto, o número de violações não será necessariamente igual ao número de registros inválidos.&lt;/p&gt;

&lt;p&gt;Essa distinção é importante para métricas de qualidade.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Gerando métricas de Data Quality
&lt;/h2&gt;

&lt;p&gt;Vamos calcular indicadores agregados.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;metricas&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_classificado&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;agg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;total_registros&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registro_valido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                &lt;span class="mi"&gt;1&lt;/span&gt;
            &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;otherwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registros_validos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registro_valido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                &lt;span class="mi"&gt;1&lt;/span&gt;
            &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;otherwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registros_invalidos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;quality_score&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;total_registros&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registros_validos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;total_registros&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;metricas&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O resultado esperado para o dataset de exemplo é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Total de registros: 7
Registros válidos: 2
Registros inválidos: 5
Quality Score: aproximadamente 0,2857
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A duplicidade torna as duas ocorrências do pedido 2 inválidas.&lt;/p&gt;

&lt;p&gt;Esse score é uma métrica operacional simplificada: a proporção de registros que passaram por todas as regras.&lt;/p&gt;

&lt;p&gt;Não representa uma medida universal de qualidade.&lt;/p&gt;

&lt;p&gt;Em produção, podemos precisar de indicadores separados por dimensão, criticidade e impacto de negócio.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Data Quality por regra
&lt;/h2&gt;

&lt;p&gt;Um score agregado pode esconder problemas importantes.&lt;/p&gt;

&lt;p&gt;Imagine que uma tabela tenha 99% de registros válidos.&lt;/p&gt;

&lt;p&gt;Isso parece excelente.&lt;/p&gt;

&lt;p&gt;Mas e se o 1% inválido representar justamente os pedidos de maior valor?&lt;/p&gt;

&lt;p&gt;Por isso, devemos medir cada regra separadamente.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;expressoes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;total_registros&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;coluna&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;colunas_erro&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;expressoes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;coluna&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;otherwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;violacoes_&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;coluna&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;metricas_regras&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df_classificado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;agg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;expressoes&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;metricas_regras&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;truncate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora conseguimos identificar quais regras concentram problemas.&lt;/p&gt;

&lt;p&gt;Podemos acompanhar tendências como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Null Customer IDs
Duplicate Orders
Invalid Status
Negative Values
Invalid Dates
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essas métricas são muito mais úteis para diagnóstico do que apenas uma classificação geral.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Quarentena de registros
&lt;/h2&gt;

&lt;p&gt;O que devemos fazer com registros inválidos?&lt;/p&gt;

&lt;p&gt;Uma possibilidade é interromper todo o pipeline.&lt;/p&gt;

&lt;p&gt;Outra é separar os registros inválidos para investigação.&lt;/p&gt;

&lt;p&gt;Essa segunda estratégia é frequentemente chamada de &lt;strong&gt;quarentena&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 SOURCE DATA
                      |
                      v
                 SPARK VALIDATION
                      |
             +--------+--------+
             |                 |
             v                 v
         VALID DATA        INVALID DATA
             |                 |
             v                 v
       CURATED LAYER      QUARANTINE
             |                 |
             v                 v
         ANALYTICS         INVESTIGATION
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em PySpark:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_validos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df_classificado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registro_valido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df_quarentena&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df_classificado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registro_valido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos persistir os resultados:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_validos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/dq/pedidos_validos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_quarentena&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/dq/pedidos_quarentena&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esses caminhos são apenas para o laboratório.&lt;/p&gt;

&lt;p&gt;Em produção, a quarentena deve ter políticas de acesso, retenção, rastreabilidade e reprocessamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Quarentena não é descarte
&lt;/h2&gt;

&lt;p&gt;Uma decisão importante é preservar informações suficientes para investigar o problema.&lt;/p&gt;

&lt;p&gt;Idealmente, um registro em quarentena deve permitir responder:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qual fonte produziu o dado?&lt;/li&gt;
&lt;li&gt;Quando ele foi recebido?&lt;/li&gt;
&lt;li&gt;Qual pipeline o processou?&lt;/li&gt;
&lt;li&gt;Qual regra falhou?&lt;/li&gt;
&lt;li&gt;Qual era o valor original?&lt;/li&gt;
&lt;li&gt;O registro pode ser corrigido?&lt;/li&gt;
&lt;li&gt;Como será reprocessado?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Podemos adicionar metadados:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_quarentena&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_quarentena&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;quarantine_timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;current_timestamp&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pipeline_name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;orders_ingestion&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;validation_version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1.0.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em ambientes corporativos, também devemos registrar um identificador de execução e, quando aplicável, a versão do contrato.&lt;/p&gt;

&lt;p&gt;Isso ajuda a conectar qualidade, observabilidade e auditoria.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Quando interromper o pipeline?
&lt;/h2&gt;

&lt;p&gt;Nem toda violação deve produzir o mesmo comportamento.&lt;/p&gt;

&lt;p&gt;Imagine três cenários.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cenário A — Campo opcional ausente&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Podemos aceitar o registro e registrar uma advertência.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cenário B — Pedido sem identificador&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Talvez o registro deva ser enviado para quarentena.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cenário C — Volume de dados muito abaixo do esperado&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Talvez seja necessário interromper a publicação do dataset inteiro.&lt;/p&gt;

&lt;p&gt;Podemos definir políticas como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WARN
  |
  v
Log and Continue

QUARANTINE
  |
  v
Isolate Invalid Records

FAIL
  |
  v
Stop Pipeline / Block Publication
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa classificação deve fazer parte do desenho do produto de dados.&lt;/p&gt;

&lt;p&gt;Não deve depender de decisões improvisadas durante incidentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. Quality Gates
&lt;/h2&gt;

&lt;p&gt;Um &lt;strong&gt;Quality Gate&lt;/strong&gt; é uma condição que precisa ser satisfeita antes que os dados avancem para determinada etapa.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A publicação será bloqueada se menos de 99% dos registros atenderem às regras críticas.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Podemos implementar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado_metricas&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;metricas&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;first&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;quality_score&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resultado_metricas&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;quality_score&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;limite_minimo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.99&lt;/span&gt;

&lt;span class="nf"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;quality_score&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;quality_score&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;limite_minimo&lt;/span&gt;
&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Quality Gate reprovado: &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;score=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;quality_score&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No nosso laboratório, a execução será interrompida porque o score está abaixo de 99%.&lt;/p&gt;

&lt;p&gt;Isso é esperado.&lt;/p&gt;

&lt;p&gt;Entretanto, em produção, um Quality Gate não deve ser a única proteção.&lt;/p&gt;

&lt;p&gt;Precisamos garantir que os dados não sejam publicados parcialmente antes da validação.&lt;/p&gt;

&lt;p&gt;Uma estratégia é validar em uma área de staging e promover o resultado somente após aprovação.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. O que são Data Contracts?
&lt;/h2&gt;

&lt;p&gt;Até aqui, definimos regras dentro do pipeline.&lt;/p&gt;

&lt;p&gt;Mas existe um problema.&lt;/p&gt;

&lt;p&gt;Como o sistema produtor sabe quais regras precisa respeitar?&lt;/p&gt;

&lt;p&gt;E como os consumidores sabem o que esperar?&lt;/p&gt;

&lt;p&gt;É aqui que entram os &lt;strong&gt;Data Contracts&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Um Data Contract formaliza expectativas sobre um conjunto de dados e suas interfaces.&lt;/p&gt;

&lt;p&gt;Pode incluir:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Identificação do dataset.&lt;/li&gt;
&lt;li&gt;Responsável pelo dado.&lt;/li&gt;
&lt;li&gt;Schema.&lt;/li&gt;
&lt;li&gt;Tipos.&lt;/li&gt;
&lt;li&gt;Campos obrigatórios.&lt;/li&gt;
&lt;li&gt;Regras de qualidade.&lt;/li&gt;
&lt;li&gt;Semântica.&lt;/li&gt;
&lt;li&gt;Frequência de atualização.&lt;/li&gt;
&lt;li&gt;SLA.&lt;/li&gt;
&lt;li&gt;Política de evolução.&lt;/li&gt;
&lt;li&gt;Classificação de dados.&lt;/li&gt;
&lt;li&gt;Responsabilidades.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Um contrato transforma expectativas implícitas em requisitos explícitos e verificáveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. Exemplo de Data Contract
&lt;/h2&gt;

&lt;p&gt;Considere um contrato conceitual em YAML:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;contract&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;orders&lt;/span&gt;
  &lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1.0.0"&lt;/span&gt;
  &lt;span class="na"&gt;owner&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sales-data-team&lt;/span&gt;

&lt;span class="na"&gt;dataset&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;curated.orders&lt;/span&gt;
  &lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Pedidos comerciais consolidados&lt;/span&gt;

&lt;span class="na"&gt;schema&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;fields&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pedido_id&lt;/span&gt;
      &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;long&lt;/span&gt;
      &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
      &lt;span class="na"&gt;unique&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;cliente_id&lt;/span&gt;
      &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;long&lt;/span&gt;
      &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;status&lt;/span&gt;
      &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;string&lt;/span&gt;
      &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
      &lt;span class="na"&gt;allowed_values&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;CRIADO&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;PAGO&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;CANCELADO&lt;/span&gt;

    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;valor&lt;/span&gt;
      &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;decimal(12,2)&lt;/span&gt;
      &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
      &lt;span class="na"&gt;minimum&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;

    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;data_pedido&lt;/span&gt;
      &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;date&lt;/span&gt;
      &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

&lt;span class="na"&gt;service_level&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;freshness_minutes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;60&lt;/span&gt;

&lt;span class="na"&gt;quality&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;minimum_valid_ratio&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0.99&lt;/span&gt;
  &lt;span class="na"&gt;on_failure&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;block_publication&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse YAML é um exemplo didático, não uma implementação validada de um padrão específico de Data Contracts.&lt;/p&gt;

&lt;p&gt;Para produção, devemos escolher uma especificação, definir sua versão e implementar validadores compatíveis.&lt;/p&gt;

&lt;p&gt;O ponto principal é que as regras deixem de existir apenas como conhecimento informal da equipe.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. Schema Enforcement vs. Data Contract
&lt;/h2&gt;

&lt;p&gt;Esses conceitos estão relacionados, mas não são equivalentes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Schema Enforcement&lt;/strong&gt; verifica se os dados respeitam determinadas regras estruturais.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pedido_id → LONG
valor     → DECIMAL(12,2)
status    → STRING
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Data Contract&lt;/strong&gt; pode incluir regras adicionais:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pedido_id → obrigatório e único
valor     → não negativo
status    → domínio permitido
freshness → até 60 minutos
owner     → equipe comercial
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DATA CONTRACT
      |
      +-- Schema
      +-- Semantics
      +-- Quality Rules
      +-- SLA / SLO
      +-- Ownership
      +-- Evolution Policy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Um schema correto é necessário em muitos cenários, mas não é suficiente para garantir a qualidade de um produto de dados.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Versionamento e evolução de contratos
&lt;/h2&gt;

&lt;p&gt;Imagine que o produtor adiciona uma coluna:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;customer_segment
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Talvez essa alteração seja compatível com consumidores existentes.&lt;/p&gt;

&lt;p&gt;Agora imagine que o produtor renomeia:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pedido_id
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;para:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;order_identifier
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa mudança pode quebrar pipelines, dashboards e modelos downstream.&lt;/p&gt;

&lt;p&gt;Precisamos de uma política de evolução.&lt;/p&gt;

&lt;p&gt;Uma abordagem é classificar alterações em:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Compatíveis.&lt;/li&gt;
&lt;li&gt;Potencialmente incompatíveis.&lt;/li&gt;
&lt;li&gt;Incompatíveis.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Também podemos utilizar versionamento semântico como convenção de comunicação, desde que as regras sejam explicitamente definidas.&lt;/p&gt;

&lt;p&gt;O importante é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Alterações de contrato devem ser avaliadas pelo impacto nos consumidores, não apenas pela facilidade de implementação no produtor.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  16. Contract Testing
&lt;/h2&gt;

&lt;p&gt;Um Data Contract só produz valor operacional quando suas regras são verificadas.&lt;/p&gt;

&lt;p&gt;Podemos criar testes automatizados para validar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Schema.&lt;/li&gt;
&lt;li&gt;Campos obrigatórios.&lt;/li&gt;
&lt;li&gt;Domínios.&lt;/li&gt;
&lt;li&gt;Unicidade.&lt;/li&gt;
&lt;li&gt;Volume.&lt;/li&gt;
&lt;li&gt;Atualidade.&lt;/li&gt;
&lt;li&gt;Compatibilidade entre versões.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Em um pipeline de CI/CD, podemos executar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Pull Request
     |
     v
Schema Validation
     |
     v
Contract Tests
     |
     v
Data Quality Tests
     |
     v
Integration Tests
     |
     v
Deployment
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso reduz a probabilidade de alterações incompatíveis chegarem à produção.&lt;/p&gt;

&lt;p&gt;Mas testes de CI/CD não substituem validações em runtime.&lt;/p&gt;

&lt;p&gt;Os dados reais podem apresentar problemas mesmo quando o código foi aprovado.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Testes unitários de transformações Spark
&lt;/h2&gt;

&lt;p&gt;Podemos testar uma transformação sem executar um pipeline inteiro.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;functions&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;F&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;filtrar_valores_validos&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNotNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Um teste simples:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;entrada&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createDataFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;100.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;50.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;filtrar_valores_validos&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;entrada&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;ids&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse teste verifica uma regra específica.&lt;/p&gt;

&lt;p&gt;Em produção, devemos incluir casos como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Valores nulos.&lt;/li&gt;
&lt;li&gt;Tipos inesperados.&lt;/li&gt;
&lt;li&gt;Duplicidades.&lt;/li&gt;
&lt;li&gt;Limites numéricos.&lt;/li&gt;
&lt;li&gt;Datas inválidas.&lt;/li&gt;
&lt;li&gt;Strings vazias.&lt;/li&gt;
&lt;li&gt;Evolução de schema.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  18. Data Observability
&lt;/h2&gt;

&lt;p&gt;Data Quality verifica propriedades dos dados.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data Observability&lt;/strong&gt; procura oferecer visibilidade contínua sobre o comportamento dos dados e pipelines.&lt;/p&gt;

&lt;p&gt;Isso pode incluir:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DATA PIPELINE
      |
      +-- Freshness
      +-- Volume
      +-- Schema Changes
      +-- Null Rates
      +-- Distribution Changes
      +-- Pipeline Failures
      +-- Quality Violations
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Imagine um pipeline que normalmente recebe 10 milhões de registros por dia.&lt;/p&gt;

&lt;p&gt;Hoje recebeu apenas 100 mil.&lt;/p&gt;

&lt;p&gt;Os registros podem passar por todas as validações individuais.&lt;/p&gt;

&lt;p&gt;Mesmo assim, existe um problema operacional importante.&lt;/p&gt;

&lt;p&gt;Precisamos monitorar o comportamento agregado do dataset.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. Freshness e SLA
&lt;/h2&gt;

&lt;p&gt;Imagine que um dashboard comercial precisa ser atualizado a cada hora.&lt;/p&gt;

&lt;p&gt;O pipeline termina com sucesso, mas os dados mais recentes possuem três dias.&lt;/p&gt;

&lt;p&gt;Tecnicamente, a execução pode ter funcionado.&lt;/p&gt;

&lt;p&gt;Operacionalmente, o serviço falhou.&lt;/p&gt;

&lt;p&gt;Podemos medir:&lt;/p&gt;

&lt;p&gt;[&lt;/p&gt;

&lt;h1&gt;
  
  
  \text{Data Age}
&lt;/h1&gt;

&lt;h2&gt;
  
  
  \text{Current Time}
&lt;/h2&gt;

&lt;p&gt;\text{Latest Event Time}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Por exemplo, se a última atualização válida ocorreu há 180 minutos e o limite é 60 minutos, temos uma violação.&lt;/p&gt;

&lt;p&gt;É importante distinguir:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Horário do evento.&lt;/li&gt;
&lt;li&gt;Horário de ingestão.&lt;/li&gt;
&lt;li&gt;Horário de processamento.&lt;/li&gt;
&lt;li&gt;Horário de publicação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cada timestamp responde a uma pergunta diferente.&lt;/p&gt;

&lt;p&gt;Um pipeline pode processar hoje eventos históricos legítimos.&lt;/p&gt;

&lt;p&gt;Por isso, a métrica de freshness deve ser definida conforme a semântica do produto de dados.&lt;/p&gt;


&lt;h2&gt;
  
  
  20. Monitoramento de volume
&lt;/h2&gt;

&lt;p&gt;Podemos acompanhar a quantidade de registros processados por execução.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;volume_atual&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas devemos lembrar que &lt;code&gt;count()&lt;/code&gt; é uma Action e pode exigir uma varredura distribuída.&lt;/p&gt;

&lt;p&gt;Em datasets grandes, precisamos avaliar o custo dessa medição.&lt;/p&gt;

&lt;p&gt;Em determinados cenários, métricas da própria execução ou metadados confiáveis podem ser alternativas mais eficientes.&lt;/p&gt;

&lt;p&gt;O objetivo é identificar anomalias como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Expected: 10 million rows
Observed: 100 thousand rows
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Uma variação pode indicar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Falha de ingestão.&lt;/li&gt;
&lt;li&gt;Filtro incorreto.&lt;/li&gt;
&lt;li&gt;Alteração na origem.&lt;/li&gt;
&lt;li&gt;Partição ausente.&lt;/li&gt;
&lt;li&gt;Problema de conectividade.&lt;/li&gt;
&lt;li&gt;Mudança legítima de comportamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O alerta deve iniciar uma investigação, não assumir automaticamente a causa.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. Data Lineage
&lt;/h2&gt;

&lt;p&gt;Agora chegamos a outro componente importante da governança.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data Lineage&lt;/strong&gt; representa a origem e as transformações pelas quais os dados passaram.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CRM Database
     |
     v
Raw Customers
     |
     v
Spark Cleaning
     |
     v
Curated Customers
     |
     v
Customer Analytics
     |
     v
Executive Dashboard
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se a coluna &lt;code&gt;customer_segment&lt;/code&gt; estiver incorreta, queremos saber:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qual sistema produziu o valor?&lt;/li&gt;
&lt;li&gt;Qual transformação o modificou?&lt;/li&gt;
&lt;li&gt;Quais tabelas dependem dele?&lt;/li&gt;
&lt;li&gt;Quais dashboards podem ter sido afetados?&lt;/li&gt;
&lt;li&gt;Quais pipelines precisam ser reexecutados?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Lineage ajuda a responder essas perguntas.&lt;/p&gt;




&lt;h2&gt;
  
  
  22. Lineage não é o mesmo que RDD Lineage
&lt;/h2&gt;

&lt;p&gt;Na Parte 8, estudamos o lineage interno do Spark.&lt;/p&gt;

&lt;p&gt;Ele representa dependências computacionais utilizadas, entre outras finalidades, na recuperação de partições.&lt;/p&gt;

&lt;p&gt;Agora estamos falando de lineage de dados em nível de plataforma.&lt;/p&gt;

&lt;p&gt;A diferença é importante.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;RDD Lineage&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD A
  |
  v
RDD B
  |
  v
RDD C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Data Lineage corporativo&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source System
     |
     v
Ingestion Job
     |
     v
Lakehouse Table
     |
     v
Business Metric
     |
     v
Dashboard
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O primeiro está relacionado à execução distribuída.&lt;/p&gt;

&lt;p&gt;O segundo está relacionado à rastreabilidade, impacto e governança.&lt;/p&gt;

&lt;p&gt;Ambos são importantes, mas resolvem problemas diferentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. OpenLineage e integração com pipelines
&lt;/h2&gt;

&lt;p&gt;Uma abordagem para padronizar metadados de lineage é utilizar o projeto OpenLineage.&lt;/p&gt;

&lt;p&gt;Ele define um modelo para representar eventos de execução, Jobs e datasets.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark Job
    |
    v
Lineage Events
    |
    v
Metadata Collector
    |
    v
Lineage Platform
    |
    v
Impact Analysis
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A integração depende das versões e componentes utilizados.&lt;/p&gt;

&lt;p&gt;Não devemos assumir que uma instalação padrão do PySpark já produz automaticamente todos os eventos de lineage corporativo.&lt;/p&gt;

&lt;p&gt;É necessário configurar mecanismos de coleta e integração compatíveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Catálogo de dados e metadados
&lt;/h2&gt;

&lt;p&gt;Uma plataforma corporativa precisa permitir que usuários descubram e compreendam os datasets disponíveis.&lt;/p&gt;

&lt;p&gt;Um catálogo pode registrar informações como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Nome.&lt;/li&gt;
&lt;li&gt;Descrição.&lt;/li&gt;
&lt;li&gt;Responsável.&lt;/li&gt;
&lt;li&gt;Schema.&lt;/li&gt;
&lt;li&gt;Classificação.&lt;/li&gt;
&lt;li&gt;Localização.&lt;/li&gt;
&lt;li&gt;Frequência de atualização.&lt;/li&gt;
&lt;li&gt;Regras de qualidade.&lt;/li&gt;
&lt;li&gt;Lineage.&lt;/li&gt;
&lt;li&gt;Termos de negócio.&lt;/li&gt;
&lt;li&gt;Políticas de acesso.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma visão conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                DATA CATALOG
                     |
        +------------+------------+
        |            |            |
        v            v            v
      Schema       Lineage      Ownership
        |            |            |
        +------------+------------+
                     |
                     v
               DATA DISCOVERY
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ferramentas como OpenMetadata e DataHub são exemplos de plataformas que podem apoiar esse tipo de capacidade.&lt;/p&gt;

&lt;p&gt;Mas o catálogo não substitui a definição de responsabilidades e processos de governança.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. Governança não é apenas catalogação
&lt;/h2&gt;

&lt;p&gt;É comum reduzir governança à instalação de uma ferramenta.&lt;/p&gt;

&lt;p&gt;Mas governança envolve decisões organizacionais.&lt;/p&gt;

&lt;p&gt;Precisamos definir:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data Owner&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Responsável pela decisão e pelo uso apropriado dos dados em determinado domínio.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data Steward&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Responsável por apoiar a gestão de definições, qualidade e políticas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data Engineering&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Responsável pela implementação e operação dos pipelines e estruturas técnicas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data Consumer&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Responsável por utilizar os dados de acordo com sua finalidade e restrições.&lt;/p&gt;

&lt;p&gt;Os papéis exatos variam entre organizações.&lt;/p&gt;

&lt;p&gt;O importante é evitar datasets sem responsáveis claros.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. Segurança, privacidade e LGPD
&lt;/h2&gt;

&lt;p&gt;Qualidade e governança também precisam considerar segurança e privacidade.&lt;/p&gt;

&lt;p&gt;Imagine um dataset contendo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;customer_id
customer_name
email
phone
document_number
purchase_history
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nem todos os consumidores precisam acessar todas essas informações.&lt;/p&gt;

&lt;p&gt;Precisamos avaliar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Classificação dos dados.&lt;/li&gt;
&lt;li&gt;Finalidade de tratamento.&lt;/li&gt;
&lt;li&gt;Base legal aplicável.&lt;/li&gt;
&lt;li&gt;Controle de acesso.&lt;/li&gt;
&lt;li&gt;Retenção.&lt;/li&gt;
&lt;li&gt;Minimização.&lt;/li&gt;
&lt;li&gt;Mascaramento ou pseudonimização.&lt;/li&gt;
&lt;li&gt;Auditoria.&lt;/li&gt;
&lt;li&gt;Exclusão ou anonimização quando cabível.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A LGPD não é implementada simplesmente ativando uma configuração do Spark.&lt;/p&gt;

&lt;p&gt;Ela exige medidas técnicas, organizacionais e jurídicas adequadas ao contexto.&lt;/p&gt;

&lt;p&gt;O Spark participa da arquitetura, mas não substitui uma política de proteção de dados.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Qualidade em arquiteturas Lakehouse
&lt;/h2&gt;

&lt;p&gt;Na Parte 13, estudamos formatos de tabela como Iceberg, Delta Lake e Hudi.&lt;/p&gt;

&lt;p&gt;Agora podemos conectar esses recursos à qualidade de dados.&lt;/p&gt;

&lt;p&gt;Uma arquitetura conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                   DATA SOURCES
                        |
                        v
                    RAW LAYER
                        |
                        v
                 CONTRACT CHECK
                        |
                        v
                DATA QUALITY RULES
                        |
             +----------+----------+
             |                     |
             v                     v
         APPROVED DATA         QUARANTINE
             |                     |
             v                     v
        LAKEHOUSE TABLE        INVESTIGATION
             |
             v
        CONSUMPTION LAYER
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O formato de tabela pode ajudar na publicação transacional.&lt;/p&gt;

&lt;p&gt;Mas precisamos garantir que as validações sejam realizadas antes de disponibilizar uma versão para os consumidores.&lt;/p&gt;

&lt;p&gt;A estratégia exata depende do formato, da engine e do mecanismo de publicação.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Idempotência e reprocessamento
&lt;/h2&gt;

&lt;p&gt;Imagine que um pipeline falhou depois de processar parte dos registros.&lt;/p&gt;

&lt;p&gt;O orquestrador executa uma nova tentativa.&lt;/p&gt;

&lt;p&gt;Se a escrita não for idempotente, podemos gerar duplicidades.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Batch 100
   |
   v
Write
   |
   v
Failure
   |
   v
Retry Batch 100
   |
   v
Duplicate Records
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Uma estratégia idempotente procura garantir que reprocessar a mesma unidade lógica de trabalho não produza efeitos adicionais indevidos.&lt;/p&gt;

&lt;p&gt;Isso pode envolver:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Chaves de negócio.&lt;/li&gt;
&lt;li&gt;Identificadores de batch.&lt;/li&gt;
&lt;li&gt;Operações MERGE.&lt;/li&gt;
&lt;li&gt;Deduplicação.&lt;/li&gt;
&lt;li&gt;Controle de commits.&lt;/li&gt;
&lt;li&gt;Escritas transacionais.&lt;/li&gt;
&lt;li&gt;Checkpoints.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A implementação precisa considerar o destino e a semântica de entrega.&lt;/p&gt;

&lt;p&gt;Não basta utilizar &lt;code&gt;dropDuplicates()&lt;/code&gt; indiscriminadamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. Data Quality em streaming
&lt;/h2&gt;

&lt;p&gt;Em pipelines streaming, a qualidade precisa ser avaliada continuamente.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Kafka
  |
  v
Spark Structured Streaming
  |
  v
Validation
  |
  +-- Valid Events
  |
  +-- Invalid Events
  |
  v
Lakehouse
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Existem desafios adicionais:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Eventos atrasados.&lt;/li&gt;
&lt;li&gt;Eventos fora de ordem.&lt;/li&gt;
&lt;li&gt;Duplicidades.&lt;/li&gt;
&lt;li&gt;Schema Evolution.&lt;/li&gt;
&lt;li&gt;Janelas temporais.&lt;/li&gt;
&lt;li&gt;Estado.&lt;/li&gt;
&lt;li&gt;Checkpoints.&lt;/li&gt;
&lt;li&gt;Reprocessamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por exemplo, uma regra de unicidade em streaming pode exigir manutenção de estado e políticas de retenção.&lt;/p&gt;

&lt;p&gt;Não podemos simplesmente aplicar uma janela global ilimitada sobre um fluxo infinito e esperar comportamento eficiente.&lt;/p&gt;

&lt;p&gt;Esses detalhes serão aprofundados quando chegarmos a Structured Streaming.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. Frameworks de Data Quality
&lt;/h2&gt;

&lt;p&gt;Nem toda validação precisa ser implementada manualmente.&lt;/p&gt;

&lt;p&gt;Existem ferramentas e bibliotecas que ajudam a estruturar regras, métricas e relatórios.&lt;/p&gt;

&lt;p&gt;Entre as opções conhecidas estão:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Great Expectations&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Framework de validação e documentação de expectativas sobre dados, com integrações que dependem da versão e do ambiente.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Deequ&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Biblioteca voltada à definição e análise de restrições de qualidade em datasets, construída sobre Spark.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Soda&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Ecossistema de ferramentas para verificações e monitoramento de qualidade de dados.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;PySpark nativo&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Permite implementar validações diretamente com DataFrames, Spark SQL e testes automatizados.&lt;/p&gt;

&lt;p&gt;A escolha depende de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Integração com a plataforma.&lt;/li&gt;
&lt;li&gt;Volume.&lt;/li&gt;
&lt;li&gt;Linguagem.&lt;/li&gt;
&lt;li&gt;Requisitos de observabilidade.&lt;/li&gt;
&lt;li&gt;Manutenção.&lt;/li&gt;
&lt;li&gt;Licenciamento.&lt;/li&gt;
&lt;li&gt;Complexidade operacional.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Em alguns projetos, um conjunto pequeno de regras Spark bem testadas pode ser suficiente.&lt;/p&gt;

&lt;p&gt;Em outros, uma plataforma dedicada traz vantagens importantes.&lt;/p&gt;




&lt;h2&gt;
  
  
  31. Arquitetura de referência
&lt;/h2&gt;

&lt;p&gt;Vamos reunir os conceitos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                      DATA PRODUCERS
                            |
                            v
                      DATA CONTRACTS
                            |
                            v
                       INGESTION
                            |
                            v
                        RAW LAYER
                            |
                            v
                    SPARK VALIDATION
                            |
                +-----------+-----------+
                |                       |
                v                       v
           VALID DATA              INVALID DATA
                |                       |
                v                       v
          LAKEHOUSE TABLE           QUARANTINE
                |
                v
          QUALITY METRICS
                |
                v
          DATA OBSERVABILITY
                |
                v
           DATA CATALOG
                |
                v
             LINEAGE
                |
                v
           DATA CONSUMERS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Na prática, observabilidade, catálogo, lineage e governança são capacidades transversais, não necessariamente etapas sequenciais do processamento.&lt;/p&gt;

&lt;p&gt;Essa arquitetura ajuda a visualizar responsabilidades.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Quality Gates e publicação transacional
&lt;/h2&gt;

&lt;p&gt;Uma estratégia robusta separa o processamento da publicação.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. Ingest Data
       |
       v
2. Transform
       |
       v
3. Validate
       |
       v
4. Calculate Quality Metrics
       |
       v
5. Evaluate Quality Gates
       |
       +-- FAIL --&amp;gt; Quarantine / Alert
       |
       v
6. Commit / Publish
       |
       v
7. Notify Consumers
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O objetivo é impedir que dados reprovados sejam disponibilizados inadvertidamente.&lt;/p&gt;

&lt;p&gt;Entretanto, a atomicidade entre múltiplas tabelas ou sistemas não deve ser presumida.&lt;/p&gt;

&lt;p&gt;Se uma publicação envolver vários destinos, precisaremos definir explicitamente a estratégia de consistência e recuperação.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. Indicadores para um produto de dados
&lt;/h2&gt;

&lt;p&gt;Além das métricas técnicas do Spark, considero importante acompanhar indicadores de qualidade e confiabilidade.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Indicador&lt;/th&gt;
&lt;th&gt;Objetivo&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Valid Record Ratio&lt;/td&gt;
&lt;td&gt;Proporção de registros aprovados&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Null Rate&lt;/td&gt;
&lt;td&gt;Completude de campos&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Duplicate Rate&lt;/td&gt;
&lt;td&gt;Ocorrência de duplicidades&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Freshness&lt;/td&gt;
&lt;td&gt;Atualidade do dataset&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Schema Compatibility&lt;/td&gt;
&lt;td&gt;Compatibilidade estrutural&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Volume Anomaly&lt;/td&gt;
&lt;td&gt;Variações inesperadas&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quality Gate Failures&lt;/td&gt;
&lt;td&gt;Reprovações de publicação&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Time to Detection&lt;/td&gt;
&lt;td&gt;Tempo até detectar um problema&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Time to Recovery&lt;/td&gt;
&lt;td&gt;Tempo até recuperar o serviço&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Contract Violations&lt;/td&gt;
&lt;td&gt;Descumprimentos do contrato&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Esses indicadores devem possuir definições claras.&lt;/p&gt;

&lt;p&gt;Por exemplo, o denominador de uma taxa de duplicidade precisa ser documentado.&lt;/p&gt;

&lt;p&gt;Sem definições consistentes, dashboards de qualidade podem gerar interpretações equivocadas.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Erros comuns
&lt;/h2&gt;

&lt;p&gt;Alguns problemas aparecem repetidamente em plataformas de dados.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Validar apenas o schema.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Tipos corretos não garantem valores corretos.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Descartar registros inválidos silenciosamente.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso compromete rastreabilidade e pode ocultar perdas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Utilizar um único Quality Score para tudo.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Problemas críticos podem ficar escondidos em uma média.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não versionar contratos.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Mudanças incompatíveis podem quebrar consumidores.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não monitorar freshness.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Um dataset estruturalmente correto pode estar desatualizado.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não definir ownership.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Incidentes ficam sem responsáveis claros.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Confundir lineage técnico com lineage corporativo.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A rastreabilidade precisa abranger fontes, transformações e consumidores.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Acreditar que ACID resolve qualidade.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Transações consistentes podem armazenar dados semanticamente incorretos.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. Como eu estruturaria a governança de um pipeline Spark
&lt;/h2&gt;

&lt;p&gt;Para um projeto corporativo, começaria com um conjunto mínimo de controles:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Identificação do dataset e de seus responsáveis.&lt;/li&gt;
&lt;li&gt;Contrato versionado.&lt;/li&gt;
&lt;li&gt;Schema explícito.&lt;/li&gt;
&lt;li&gt;Regras críticas de qualidade.&lt;/li&gt;
&lt;li&gt;Métricas por execução.&lt;/li&gt;
&lt;li&gt;Área de quarentena.&lt;/li&gt;
&lt;li&gt;Quality Gates antes da publicação.&lt;/li&gt;
&lt;li&gt;Logs e identificadores de execução.&lt;/li&gt;
&lt;li&gt;Política de reprocessamento.&lt;/li&gt;
&lt;li&gt;Monitoramento de freshness e volume.&lt;/li&gt;
&lt;li&gt;Registro de lineage.&lt;/li&gt;
&lt;li&gt;Catálogo e classificação.&lt;/li&gt;
&lt;li&gt;Política de acesso e retenção.&lt;/li&gt;
&lt;li&gt;Testes automatizados.&lt;/li&gt;
&lt;li&gt;Procedimentos de resposta a incidentes.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;O objetivo não é criar burocracia.&lt;/p&gt;

&lt;p&gt;É tornar a confiabilidade dos dados verificável e operacional.&lt;/p&gt;




&lt;h2&gt;
  
  
  36. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;Até aqui, estudamos como fazer o Spark executar corretamente.&lt;/p&gt;

&lt;p&gt;Agora precisamos ampliar nossa definição de sucesso.&lt;/p&gt;

&lt;p&gt;Um pipeline confiável não é apenas aquele que:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executes Successfully
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ele também precisa:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Respect Contracts
       |
       v
Validate Data
       |
       v
Track Quality
       |
       v
Preserve Lineage
       |
       v
Enforce Governance
       |
       v
Deliver Trusted Data
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa é a diferença entre simplesmente processar dados e entregar um produto de dados confiável.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Performance torna o processamento eficiente. Qualidade torna os resultados utilizáveis. Governança torna o uso sustentável e responsável.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  37. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando avalio uma plataforma de dados, não pergunto apenas:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Quanto tempo o Spark demora para processar?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Também quero saber:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Quem é responsável pelo dataset?&lt;/li&gt;
&lt;li&gt;Qual contrato define sua estrutura?&lt;/li&gt;
&lt;li&gt;Quais regras de qualidade são verificadas?&lt;/li&gt;
&lt;li&gt;Como identificamos registros inválidos?&lt;/li&gt;
&lt;li&gt;Como evitamos publicações incorretas?&lt;/li&gt;
&lt;li&gt;Como investigamos mudanças inesperadas?&lt;/li&gt;
&lt;li&gt;Como rastreamos a origem dos dados?&lt;/li&gt;
&lt;li&gt;Como sabemos se o dataset está atualizado?&lt;/li&gt;
&lt;li&gt;Como controlamos acesso e retenção?&lt;/li&gt;
&lt;li&gt;Como recuperamos uma execução com problemas?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Essas perguntas fazem parte da Engenharia de Dados tanto quanto CPU, memória e shuffle.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora que entendemos como validar e governar dados processados pelo Spark, vamos avançar para um dos temas mais importantes das arquiteturas modernas:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Apache Spark Structured Streaming.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Vamos explorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Batch vs. Streaming.&lt;/li&gt;
&lt;li&gt;Microbatch.&lt;/li&gt;
&lt;li&gt;Structured Streaming.&lt;/li&gt;
&lt;li&gt;Sources e Sinks.&lt;/li&gt;
&lt;li&gt;Triggers.&lt;/li&gt;
&lt;li&gt;Checkpoints.&lt;/li&gt;
&lt;li&gt;Event Time vs. Processing Time.&lt;/li&gt;
&lt;li&gt;Watermarks.&lt;/li&gt;
&lt;li&gt;Stateful Processing.&lt;/li&gt;
&lt;li&gt;Exactly-once e suas limitações.&lt;/li&gt;
&lt;li&gt;Integração com Kafka.&lt;/li&gt;
&lt;li&gt;Escrita em Lakehouse.&lt;/li&gt;
&lt;li&gt;Observabilidade e recuperação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central será:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como o Spark transforma fluxos contínuos de eventos em processamento distribuído confiável?&lt;/strong&gt;&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 13:&lt;/strong&gt; Iceberg, Delta Lake e Hudi — a evolução para Lakehouse&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 15 — Spark Structured Streaming: processando dados em tempo quase real&lt;/p&gt;

</description>
      <category>python</category>
      <category>bigdata</category>
      <category>spark</category>
      <category>apache</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 13: Iceberg, Delta Lake e Hudi — a evolução para Lakehouse</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 20:00:36 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-13-iceberg-delta-lake-e-hudi-a-evolucao-para-lakehouse-32g8</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-13-iceberg-delta-lake-e-hudi-a-evolucao-para-lakehouse-32g8</guid>
      <description>&lt;p&gt;&lt;em&gt;Nota histórica: esta série revisita minha trajetória em Engenharia de Dados iniciada em 2012. Apache Iceberg, Delta Lake e Apache Hudi surgiram posteriormente. Neste capítulo, conectamos os fundamentos daquela jornada às arquiteturas Lakehouse modernas.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nos capítulos anteriores, aprendemos como o Apache Spark executa processamento distribuído, organiza Tasks, utiliza memória, movimenta dados por shuffle e otimiza consultas.&lt;/p&gt;

&lt;p&gt;Na Parte 12, exploramos os formatos CSV, JSON, Parquet e ORC.&lt;/p&gt;

&lt;p&gt;Entendemos por que formatos colunares são importantes para workloads analíticos e como recursos como Column Pruning, Predicate Pushdown e Partition Pruning podem reduzir o custo de leitura.&lt;/p&gt;

&lt;p&gt;Mas existe uma limitação que merece atenção.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Um conjunto de arquivos Parquet não é, automaticamente, uma tabela transacional.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Imagine um Data Lake contendo milhares de arquivos Parquet.&lt;/p&gt;

&lt;p&gt;Precisamos atualizar pedidos, corrigir registros, receber novos eventos, consultar versões anteriores e permitir que diferentes aplicações leiam os dados enquanto outras escrevem.&lt;/p&gt;

&lt;p&gt;Como garantir consistência?&lt;/p&gt;

&lt;p&gt;Como controlar quais arquivos pertencem à versão atual da tabela?&lt;/p&gt;

&lt;p&gt;Como recuperar uma versão anterior?&lt;/p&gt;

&lt;p&gt;E como impedir que duas operações concorrentes deixem os dados em um estado inconsistente?&lt;/p&gt;

&lt;p&gt;Essas perguntas nos levam aos &lt;strong&gt;formatos de tabela&lt;/strong&gt; e à arquitetura &lt;strong&gt;Data Lakehouse&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Data Lake, Data Warehouse e Lakehouse
&lt;/h2&gt;

&lt;p&gt;Antes de comparar tecnologias, precisamos entender o problema arquitetural.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data Warehouse
&lt;/h3&gt;

&lt;p&gt;Um Data Warehouse tradicional oferece recursos como modelagem analítica, SQL, gerenciamento de tabelas, otimização de consultas e mecanismos de consistência.&lt;/p&gt;

&lt;p&gt;É projetado para facilitar o consumo estruturado de dados.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data Lake
&lt;/h3&gt;

&lt;p&gt;Um Data Lake permite armazenar dados em diferentes formatos e estruturas, frequentemente utilizando armazenamento distribuído ou de objetos.&lt;/p&gt;

&lt;p&gt;Pode receber:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Dados estruturados.&lt;/li&gt;
&lt;li&gt;Dados semiestruturados.&lt;/li&gt;
&lt;li&gt;Arquivos brutos.&lt;/li&gt;
&lt;li&gt;Eventos.&lt;/li&gt;
&lt;li&gt;Logs.&lt;/li&gt;
&lt;li&gt;Dados históricos.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O Data Lake oferece flexibilidade, mas arquivos isolados não fornecem automaticamente todas as garantias de um sistema de tabelas.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data Lakehouse
&lt;/h3&gt;

&lt;p&gt;A arquitetura Lakehouse procura combinar a flexibilidade do armazenamento de Data Lakes com capacidades de gerenciamento e consistência de tabelas analíticas.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  DATA SOURCES
                       |
                       v
                OBJECT STORAGE
                       |
                       v
               OPEN TABLE FORMAT
                       |
            +----------+----------+
            |          |          |
            v          v          v
           SQL         BI        ML / AI
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A proposta não é simplesmente substituir o Data Warehouse.&lt;/p&gt;

&lt;p&gt;É disponibilizar capacidades adicionais sobre dados armazenados em formatos abertos e acessíveis a diferentes mecanismos de processamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. O problema dos arquivos Parquet isolados
&lt;/h2&gt;

&lt;p&gt;Imagine uma tabela de pedidos representada por três arquivos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pedidos/
  part-00001.parquet
  part-00002.parquet
  part-00003.parquet
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora precisamos atualizar o status de um pedido.&lt;/p&gt;

&lt;p&gt;Em uma arquitetura baseada apenas em arquivos, uma abordagem seria:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Ler os dados existentes.&lt;/li&gt;
&lt;li&gt;Aplicar a atualização.&lt;/li&gt;
&lt;li&gt;Reescrever os arquivos afetados.&lt;/li&gt;
&lt;li&gt;Substituir os arquivos anteriores.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Parece simples.&lt;/p&gt;

&lt;p&gt;Mas imagine que outra aplicação esteja lendo a tabela enquanto os arquivos são substituídos.&lt;/p&gt;

&lt;p&gt;Dependendo da estratégia utilizada, ela pode observar um conjunto incompleto ou inconsistente de arquivos.&lt;/p&gt;

&lt;p&gt;Outro problema surge quando duas aplicações tentam modificar os mesmos dados simultaneamente.&lt;/p&gt;

&lt;p&gt;Precisamos de um mecanismo que controle &lt;strong&gt;qual conjunto de arquivos representa uma versão válida da tabela&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Essa é uma das principais responsabilidades dos formatos de tabela modernos.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Formato de arquivo não é formato de tabela
&lt;/h2&gt;

&lt;p&gt;Essa distinção é fundamental.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Parquet&lt;/strong&gt; define como os dados são armazenados fisicamente em arquivos.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Iceberg, Delta Lake e Hudi&lt;/strong&gt; adicionam mecanismos para gerenciar tabelas compostas por arquivos.&lt;/p&gt;

&lt;p&gt;Uma visão conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  TABLE
                    |
                    v
             TABLE METADATA
                    |
                    v
             TABLE SNAPSHOT
                    |
                    v
               DATA FILES
                    |
                    v
                  PARQUET
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os detalhes internos diferem entre as tecnologias.&lt;/p&gt;

&lt;p&gt;Mas a ideia geral é semelhante: precisamos de uma camada de gerenciamento acima dos arquivos.&lt;/p&gt;

&lt;p&gt;Essa camada pode oferecer:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Transações.&lt;/li&gt;
&lt;li&gt;Controle de versões.&lt;/li&gt;
&lt;li&gt;Metadados.&lt;/li&gt;
&lt;li&gt;Evolução de schema.&lt;/li&gt;
&lt;li&gt;Atualizações e exclusões.&lt;/li&gt;
&lt;li&gt;Operações concorrentes.&lt;/li&gt;
&lt;li&gt;Gerenciamento do ciclo de vida dos arquivos.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Isso transforma a maneira como trabalhamos com dados analíticos.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. ACID em tabelas analíticas
&lt;/h2&gt;

&lt;p&gt;ACID representa quatro propriedades tradicionalmente associadas a transações.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Atomicidade:&lt;/strong&gt; uma operação transacional deve ser aplicada integralmente ou não ser confirmada.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Consistência:&lt;/strong&gt; a transação deve preservar as regras e invariantes garantidos pelo sistema.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Isolamento:&lt;/strong&gt; operações concorrentes devem respeitar o modelo de isolamento oferecido.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Durabilidade:&lt;/strong&gt; uma transação confirmada deve permanecer registrada conforme as garantias do sistema.&lt;/p&gt;

&lt;p&gt;Em uma arquitetura Lakehouse, essas propriedades são implementadas por mecanismos específicos de metadados, logs, snapshots e controle de concorrência.&lt;/p&gt;

&lt;p&gt;É importante não presumir que todos os formatos ofereçam exatamente os mesmos níveis de isolamento ou comportamento em qualquer engine.&lt;/p&gt;

&lt;p&gt;As garantias precisam ser verificadas para cada combinação de formato, catálogo, armazenamento e mecanismo de processamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. O conceito de snapshot
&lt;/h2&gt;

&lt;p&gt;Imagine uma tabela com três arquivos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SNAPSHOT 100

File A
File B
File C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois, uma operação atualiza determinados registros.&lt;/p&gt;

&lt;p&gt;A nova versão pode referenciar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SNAPSHOT 101

File A
File C
File D
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O arquivo B deixou de fazer parte da versão atual e foi substituído por D.&lt;/p&gt;

&lt;p&gt;Observe que não precisamos necessariamente modificar o conteúdo dos arquivos antigos.&lt;/p&gt;

&lt;p&gt;Podemos representar a evolução da tabela por metadados que identificam os arquivos válidos em cada versão.&lt;/p&gt;

&lt;p&gt;Isso permite implementar recursos como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Leitura consistente.&lt;/li&gt;
&lt;li&gt;Histórico de versões.&lt;/li&gt;
&lt;li&gt;Rollback, quando suportado.&lt;/li&gt;
&lt;li&gt;Time Travel.&lt;/li&gt;
&lt;li&gt;Controle de mudanças.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Os detalhes de retenção e limpeza de arquivos são importantes: uma versão antiga só poderá ser consultada enquanto seus metadados e arquivos necessários estiverem disponíveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Apache Iceberg
&lt;/h2&gt;

&lt;p&gt;O &lt;strong&gt;Apache Iceberg&lt;/strong&gt; é um formato aberto de tabela desenvolvido para datasets analíticos de grande escala.&lt;/p&gt;

&lt;p&gt;Seu projeto enfatiza gerenciamento de metadados, evolução de tabelas e interoperabilidade entre mecanismos de processamento.&lt;/p&gt;

&lt;p&gt;Uma arquitetura conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  ICEBERG TABLE
                        |
                        v
                 CATALOG ENTRY
                        |
                        v
                 METADATA FILE
                        |
                        v
                   SNAPSHOT
                        |
                        v
                 MANIFEST LIST
                        |
                        v
                 MANIFEST FILES
                        |
                        v
                   DATA FILES
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os arquivos de manifesto descrevem informações sobre arquivos de dados e seus metadados.&lt;/p&gt;

&lt;p&gt;Essa organização permite que engines identifiquem os arquivos relevantes sem depender exclusivamente de uma listagem completa de diretórios.&lt;/p&gt;

&lt;p&gt;Iceberg também suporta arquivos de exclusão em cenários e versões compatíveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Evolução de schema no Iceberg
&lt;/h2&gt;

&lt;p&gt;Imagine uma tabela inicial:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;customer_id
customer_name
state
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora precisamos adicionar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;customer_segment
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em sistemas baseados apenas em arquivos, mudanças de schema podem exigir cuidados adicionais para manter a interpretação consistente dos dados históricos.&lt;/p&gt;

&lt;p&gt;O Iceberg oferece mecanismos de evolução de schema, incluindo identificação de campos por IDs.&lt;/p&gt;

&lt;p&gt;Isso ajuda a evitar determinadas ambiguidades associadas a mudanças de nomes e posições de colunas.&lt;/p&gt;

&lt;p&gt;Exemplo conceitual em Spark SQL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;lakehouse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;crm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clientes&lt;/span&gt;
&lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;customer_segment&lt;/span&gt; &lt;span class="n"&gt;STRING&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A operação depende de um catálogo Iceberg corretamente configurado.&lt;/p&gt;

&lt;p&gt;O recurso é especialmente importante em plataformas nas quais os contratos de dados evoluem continuamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Hidden Partitioning
&lt;/h2&gt;

&lt;p&gt;Um recurso conhecido do Iceberg é o &lt;strong&gt;Hidden Partitioning&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Em datasets particionados por diretórios tradicionais, frequentemente precisamos lidar explicitamente com colunas derivadas como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ano
mes
dia
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Iceberg permite definir transformações de particionamento nos metadados da tabela.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;lakehouse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sales&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedidos&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;pedido_id&lt;/span&gt; &lt;span class="nb"&gt;BIGINT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;cliente_id&lt;/span&gt; &lt;span class="nb"&gt;BIGINT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;data_pedido&lt;/span&gt; &lt;span class="nb"&gt;TIMESTAMP&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;valor&lt;/span&gt; &lt;span class="nb"&gt;DECIMAL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="n"&gt;iceberg&lt;/span&gt;
&lt;span class="n"&gt;PARTITIONED&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data_pedido&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nesse exemplo, a tabela é particionada por dia com base no timestamp.&lt;/p&gt;

&lt;p&gt;O consumidor pode consultar a coluna original:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;lakehouse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sales&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedidos&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;data_pedido&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nb"&gt;TIMESTAMP&lt;/span&gt; &lt;span class="s1"&gt;'2026-11-01 00:00:00'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;data_pedido&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nb"&gt;TIMESTAMP&lt;/span&gt; &lt;span class="s1"&gt;'2026-11-02 00:00:00'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O mecanismo pode utilizar os metadados de particionamento para reduzir os arquivos examinados.&lt;/p&gt;

&lt;p&gt;Isso diminui o acoplamento entre a lógica das consultas e a estrutura física do armazenamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Configurando Iceberg com Spark
&lt;/h2&gt;

&lt;p&gt;Uma integração Spark–Iceberg pode utilizar um catálogo configurado na SparkSession.&lt;/p&gt;

&lt;p&gt;Exemplo ilustrativo com catálogo Hadoop para desenvolvimento local:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;IcebergExample&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.sql.extensions&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.sql.catalog.lakehouse&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;org.apache.iceberg.spark.SparkCatalog&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.sql.catalog.lakehouse.type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hadoop&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spark.sql.catalog.lakehouse.warehouse&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;file:///tmp/iceberg-warehouse&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Importante:&lt;/strong&gt; esse exemplo pressupõe que os artefatos Iceberg compatíveis com a versão do Spark e do Scala estejam disponíveis no classpath. Apenas configurar a SparkSession não instala as dependências.&lt;/p&gt;

&lt;p&gt;O catálogo Hadoop local é adequado para experimentação controlada, não uma recomendação genérica para produção distribuída.&lt;/p&gt;

&lt;p&gt;Em produção, devemos avaliar catálogos e mecanismos de armazenamento compatíveis com os requisitos de concorrência e atomicidade.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Criando uma tabela Iceberg
&lt;/h2&gt;

&lt;p&gt;Com o ambiente configurado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;
CREATE NAMESPACE IF NOT EXISTS lakehouse.sales
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;
CREATE TABLE IF NOT EXISTS lakehouse.sales.pedidos (
    pedido_id BIGINT,
    cliente_id BIGINT,
    status STRING,
    valor DECIMAL(12,2)
)
USING iceberg
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Inserimos dados:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;
INSERT INTO lakehouse.sales.pedidos
VALUES
    (1, 101, &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;, 150.00),
    (2, 102, &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;, 230.00),
    (3, 103, &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;, 99.90)
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora podemos consultar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;
SELECT *
FROM lakehouse.sales.pedidos
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A tabela possui metadados gerenciados pelo formato Iceberg, em vez de ser apenas um diretório de arquivos Parquet sem gerenciamento transacional.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. MERGE INTO: atualizações incrementais
&lt;/h2&gt;

&lt;p&gt;Em pipelines modernos, frequentemente recebemos alterações incrementais.&lt;/p&gt;

&lt;p&gt;Imagine um fluxo de CDC — &lt;em&gt;Change Data Capture&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;O sistema de origem informa que o pedido 1 mudou de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CRIADO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;para:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PAGO
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Com suporte adequado do formato e da engine, podemos utilizar &lt;code&gt;MERGE INTO&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="n"&gt;MERGE&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;lakehouse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sales&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedidos&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;destino&lt;/span&gt;
&lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;SELECT&lt;/span&gt;
        &lt;span class="k"&gt;CAST&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="nb"&gt;BIGINT&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;pedido_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'PAGO'&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;
&lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedido_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pedido_id&lt;/span&gt;

&lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;MATCHED&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt;
    &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;destino&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;origem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse padrão é importante para:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ingestão incremental.&lt;/li&gt;
&lt;li&gt;Sincronização de sistemas.&lt;/li&gt;
&lt;li&gt;Atualização de dimensões.&lt;/li&gt;
&lt;li&gt;Correção de registros.&lt;/li&gt;
&lt;li&gt;Processamento de eventos de alteração.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas um MERGE não resolve automaticamente duplicidades ou eventos fora de ordem.&lt;/p&gt;

&lt;p&gt;Precisamos definir regras de negócio, chaves e ordenação das alterações.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. Delta Lake
&lt;/h2&gt;

&lt;p&gt;O &lt;strong&gt;Delta Lake&lt;/strong&gt; é outra tecnologia de gerenciamento de tabelas para Data Lakes.&lt;/p&gt;

&lt;p&gt;Sua arquitetura utiliza um log transacional, conhecido como &lt;strong&gt;Delta Log&lt;/strong&gt;, para controlar as versões e alterações da tabela.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                   DELTA TABLE
                        |
             +----------+----------+
             |                     |
             v                     v
        DATA FILES             _delta_log/
             |                     |
             v                     v
          PARQUET              COMMITS
                                   |
                                   v
                               CHECKPOINTS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O log registra as ações que compõem o estado da tabela.&lt;/p&gt;

&lt;p&gt;Checkpoints ajudam a tornar mais eficiente a reconstrução desse estado.&lt;/p&gt;

&lt;p&gt;O Delta Lake é amplamente utilizado em arquiteturas Lakehouse e possui integração importante com o ecossistema Spark.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. Escrevendo uma tabela Delta
&lt;/h2&gt;

&lt;p&gt;Em um ambiente com Delta Lake instalado e configurado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;dados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createDataFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;150.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;230.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CRIADO&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;99.9&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos escrever:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dados&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;delta&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/delta-pedidos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E ler:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_delta&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;delta&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/delta-pedidos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df_delta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse exemplo não funciona em uma instalação Spark pura sem os componentes Delta necessários.&lt;/p&gt;

&lt;p&gt;As versões do Delta Lake, Spark e Scala precisam ser compatíveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. Atualizações com Delta Lake
&lt;/h2&gt;

&lt;p&gt;Uma abordagem programática utiliza a API DeltaTable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;delta.tables&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;DeltaTable&lt;/span&gt;

&lt;span class="n"&gt;tabela&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;DeltaTable&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;forPath&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/delta-pedidos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;tabela&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;condition&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pedido_id = 1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"'&lt;/span&gt;&lt;span class="s"&gt;PAGO&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Também podemos realizar operações &lt;code&gt;MERGE&lt;/code&gt; quando as condições e a versão utilizada oferecerem suporte.&lt;/p&gt;

&lt;p&gt;Um aspecto importante é que essas operações não devem ser interpretadas como alterações diretas e arbitrárias de arquivos Parquet.&lt;/p&gt;

&lt;p&gt;Elas são gerenciadas pelo mecanismo transacional da tabela.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Time Travel
&lt;/h2&gt;

&lt;p&gt;Um recurso bastante útil em formatos de tabela modernos é consultar versões anteriores.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Version 0 → Pedidos originais
Version 1 → Atualização de status
Version 2 → Novos pedidos
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em Delta Lake, podemos consultar uma versão anterior:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;versao_anterior&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;delta&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;versionAsOf&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/delta-pedidos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode ajudar em:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Auditoria.&lt;/li&gt;
&lt;li&gt;Investigação de incidentes.&lt;/li&gt;
&lt;li&gt;Comparação de versões.&lt;/li&gt;
&lt;li&gt;Reprocessamento.&lt;/li&gt;
&lt;li&gt;Recuperação lógica.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas Time Travel não deve ser confundido com backup completo.&lt;/p&gt;

&lt;p&gt;Se arquivos antigos forem removidos por políticas de retenção ou manutenção, determinadas versões podem deixar de estar disponíveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. Apache Hudi
&lt;/h2&gt;

&lt;p&gt;O &lt;strong&gt;Apache Hudi&lt;/strong&gt; também oferece mecanismos de gerenciamento de tabelas sobre Data Lakes.&lt;/p&gt;

&lt;p&gt;Uma de suas características é o foco em workloads com ingestão incremental, atualizações e gerenciamento eficiente de registros.&lt;/p&gt;

&lt;p&gt;Hudi disponibiliza diferentes tipos de tabela, incluindo:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Copy-on-Write (CoW)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Atualizações podem resultar na reescrita de arquivos base.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Merge-on-Read (MoR)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Alterações podem ser registradas em estruturas adicionais e reconciliadas durante a leitura ou por operações de compactação, conforme a configuração.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 HUDI TABLE
                      |
               TABLE TIMELINE
                      |
             +--------+--------+
             |                 |
             v                 v
       BASE FILES          LOG FILES
             |                 |
             +--------+--------+
                      |
                      v
                QUERY RESULT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O comportamento exato depende do tipo de tabela, das operações e da versão utilizada.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Copy-on-Write vs. Merge-on-Read
&lt;/h2&gt;

&lt;p&gt;Considere um dataset que recebe muitas atualizações.&lt;/p&gt;

&lt;p&gt;Em uma abordagem Copy-on-Write:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Existing File
      |
      v
Update Records
      |
      v
Rewrite File
      |
      v
New File Version
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode favorecer determinados padrões de leitura, mas produzir amplificação de escrita.&lt;/p&gt;

&lt;p&gt;Em uma abordagem Merge-on-Read:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Base File
    |
    +-- Update Log
    |
    +-- Update Log
    |
    v
Merged Read
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos reduzir determinados custos imediatos de atualização, mas introduzir trabalho adicional de leitura ou manutenção.&lt;/p&gt;

&lt;p&gt;O trade-off depende de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Frequência de atualização.&lt;/li&gt;
&lt;li&gt;Latência exigida.&lt;/li&gt;
&lt;li&gt;Padrões de consulta.&lt;/li&gt;
&lt;li&gt;Estratégia de compactação.&lt;/li&gt;
&lt;li&gt;Volume de dados.&lt;/li&gt;
&lt;li&gt;Recursos disponíveis.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Não existe uma estratégia universalmente superior.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. Hudi e pipelines incrementais
&lt;/h2&gt;

&lt;p&gt;Imagine um sistema transacional que produz eventos de alteração.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Operational Database
        |
        v
       CDC
        |
        v
      Kafka
        |
        v
 Spark Processing
        |
        v
    Hudi Table
        |
        v
   Analytical Queries
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse tipo de arquitetura permite construir datasets analíticos atualizados incrementalmente.&lt;/p&gt;

&lt;p&gt;Mas precisamos resolver problemas como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Identificação única dos registros.&lt;/li&gt;
&lt;li&gt;Ordenação de eventos.&lt;/li&gt;
&lt;li&gt;Atualizações fora de ordem.&lt;/li&gt;
&lt;li&gt;Deduplicação.&lt;/li&gt;
&lt;li&gt;Exclusões.&lt;/li&gt;
&lt;li&gt;Reprocessamento.&lt;/li&gt;
&lt;li&gt;Consistência entre origem e destino.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O formato de tabela oferece mecanismos úteis, mas a correção do pipeline continua dependendo da engenharia da aplicação.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. Comparando Iceberg, Delta Lake e Hudi
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Critério&lt;/th&gt;
&lt;th&gt;Apache Iceberg&lt;/th&gt;
&lt;th&gt;Delta Lake&lt;/th&gt;
&lt;th&gt;Apache Hudi&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Organização central&lt;/td&gt;
&lt;td&gt;Snapshots e manifests&lt;/td&gt;
&lt;td&gt;Log transacional&lt;/td&gt;
&lt;td&gt;Timeline de operações&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Arquivos de dados&lt;/td&gt;
&lt;td&gt;Frequentemente Parquet&lt;/td&gt;
&lt;td&gt;Principalmente Parquet&lt;/td&gt;
&lt;td&gt;Frequentemente Parquet, com logs conforme o tipo&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Evolução de schema&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Operações incrementais&lt;/td&gt;
&lt;td&gt;Sim, conforme engine&lt;/td&gt;
&lt;td&gt;Sim, conforme engine&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Time Travel&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim, conforme recursos e retenção&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Atualizações e exclusões&lt;/td&gt;
&lt;td&gt;Sim, conforme engine&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Integração com Spark&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Característica de destaque&lt;/td&gt;
&lt;td&gt;Metadados e interoperabilidade&lt;/td&gt;
&lt;td&gt;Ecossistema Delta e integração Spark&lt;/td&gt;
&lt;td&gt;Ingestão e atualização incremental&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Essa tabela é uma visão conceitual.&lt;/p&gt;

&lt;p&gt;Recursos específicos, desempenho e interoperabilidade variam conforme versões, catálogos, engines e configurações.&lt;/p&gt;

&lt;p&gt;Portanto, não devemos escolher um formato apenas por uma comparação superficial de funcionalidades.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Como escolher entre os três?
&lt;/h2&gt;

&lt;p&gt;Minha avaliação começa pelos requisitos.&lt;/p&gt;

&lt;h3&gt;
  
  
  Quando avaliar Iceberg
&lt;/h3&gt;

&lt;p&gt;Iceberg merece atenção quando buscamos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Interoperabilidade entre engines.&lt;/li&gt;
&lt;li&gt;Gerenciamento de metadados em grande escala.&lt;/li&gt;
&lt;li&gt;Evolução de schema e particionamento.&lt;/li&gt;
&lt;li&gt;Integração com catálogos.&lt;/li&gt;
&lt;li&gt;Arquiteturas analíticas abertas.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Quando avaliar Delta Lake
&lt;/h3&gt;

&lt;p&gt;Delta Lake merece atenção quando buscamos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Integração com Spark.&lt;/li&gt;
&lt;li&gt;Operações transacionais sobre tabelas analíticas.&lt;/li&gt;
&lt;li&gt;Histórico e versionamento.&lt;/li&gt;
&lt;li&gt;Pipelines de transformação e atualização.&lt;/li&gt;
&lt;li&gt;Ecossistema de ferramentas compatíveis.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Quando avaliar Hudi
&lt;/h3&gt;

&lt;p&gt;Hudi merece atenção quando buscamos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ingestão incremental frequente.&lt;/li&gt;
&lt;li&gt;Upserts.&lt;/li&gt;
&lt;li&gt;Processamento de CDC.&lt;/li&gt;
&lt;li&gt;Estratégias de atualização.&lt;/li&gt;
&lt;li&gt;Workloads com diferentes requisitos de leitura e escrita.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Esses cenários não são exclusivos.&lt;/p&gt;

&lt;p&gt;Iceberg e Delta também podem atender ingestão incremental, e Hudi também pode atender analytics.&lt;/p&gt;

&lt;p&gt;A decisão correta depende dos requisitos e de benchmarks representativos.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. O papel dos catálogos
&lt;/h2&gt;

&lt;p&gt;Uma tabela Lakehouse precisa ser descoberta e identificada.&lt;/p&gt;

&lt;p&gt;É aqui que entram os catálogos.&lt;/p&gt;

&lt;p&gt;Um catálogo pode ajudar a gerenciar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Namespaces.&lt;/li&gt;
&lt;li&gt;Identificadores de tabelas.&lt;/li&gt;
&lt;li&gt;Localização de metadados.&lt;/li&gt;
&lt;li&gt;Integração com engines.&lt;/li&gt;
&lt;li&gt;Controles de acesso, conforme a implementação.&lt;/li&gt;
&lt;li&gt;Descoberta e organização.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma arquitetura conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  DATA CONSUMERS
                  /      |      \
                 v       v       v
               Spark   Trino   Analytics
                  \      |      /
                   v     v     v
                    CATALOG
                       |
                       v
                 TABLE METADATA
                       |
                       v
                 OBJECT STORAGE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em ambientes Iceberg, por exemplo, diferentes implementações de catálogo podem ser utilizadas.&lt;/p&gt;

&lt;p&gt;A escolha do catálogo também influencia concorrência, operação, governança e integração.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Formato de tabela e catálogo são decisões relacionadas, mas distintas.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  22. Multi-engine: uma promessa que exige validação
&lt;/h2&gt;

&lt;p&gt;Uma vantagem importante das arquiteturas abertas é permitir que diferentes engines trabalhem com os mesmos datasets.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                SHARED TABLE
                     |
          +----------+----------+
          |          |          |
          v          v          v
        Spark      Trino      Flink
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas precisamos distinguir:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Conseguir ler uma tabela&lt;/strong&gt; não significa necessariamente &lt;strong&gt;conseguir executar todas as operações de escrita e manutenção com as mesmas garantias&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Compatibilidade pode variar para:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;MERGE.&lt;/li&gt;
&lt;li&gt;DELETE.&lt;/li&gt;
&lt;li&gt;UPDATE.&lt;/li&gt;
&lt;li&gt;Time Travel.&lt;/li&gt;
&lt;li&gt;Evolução de schema.&lt;/li&gt;
&lt;li&gt;Branches.&lt;/li&gt;
&lt;li&gt;Deletion vectors.&lt;/li&gt;
&lt;li&gt;Recursos específicos do formato.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Em arquiteturas multi-engine, a matriz de compatibilidade deve fazer parte da documentação técnica.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. Schema Evolution vs. Schema Enforcement
&lt;/h2&gt;

&lt;p&gt;Esses dois conceitos são frequentemente confundidos.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Schema Enforcement&lt;/strong&gt; significa validar operações de escrita contra as regras de schema suportadas pela tabela.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Schema Evolution&lt;/strong&gt; significa permitir alterações controladas na estrutura da tabela.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Schema V1
---------
customer_id
customer_name
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Schema V2
---------
customer_id
customer_name
customer_segment
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A evolução pode ser legítima.&lt;/p&gt;

&lt;p&gt;Mas isso não significa que qualquer mudança de tipo ou remoção de coluna deva ser aceita automaticamente.&lt;/p&gt;

&lt;p&gt;Em produção, precisamos de contratos de dados, compatibilidade e governança.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. ACID não substitui Data Quality
&lt;/h2&gt;

&lt;p&gt;Uma tabela pode ser transacionalmente consistente e ainda conter dados incorretos.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pedido_id = 100
valor = -999999
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A operação pode ser tecnicamente válida para o mecanismo de armazenamento, mas inválida para o negócio.&lt;/p&gt;

&lt;p&gt;Por isso, precisamos separar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TRANSACTIONAL CONSISTENCY
           |
           v
      TABLE INTEGRITY
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;BUSINESS DATA QUALITY
           |
           v
       VALIDATION RULES
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em pipelines corporativos, podemos utilizar ferramentas e estratégias de qualidade de dados para validar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Unicidade.&lt;/li&gt;
&lt;li&gt;Completude.&lt;/li&gt;
&lt;li&gt;Domínios válidos.&lt;/li&gt;
&lt;li&gt;Integridade referencial.&lt;/li&gt;
&lt;li&gt;Consistência temporal.&lt;/li&gt;
&lt;li&gt;Regras de negócio.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Lakehouse não elimina essa responsabilidade.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. Compactação e manutenção
&lt;/h2&gt;

&lt;p&gt;Na Parte 12, discutimos o Small Files Problem.&lt;/p&gt;

&lt;p&gt;Formatos de tabela modernos oferecem mecanismos de manutenção, embora as operações e os comandos variem.&lt;/p&gt;

&lt;p&gt;Podemos precisar de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Compactação de arquivos.&lt;/li&gt;
&lt;li&gt;Reorganização de dados.&lt;/li&gt;
&lt;li&gt;Expiração de snapshots.&lt;/li&gt;
&lt;li&gt;Limpeza de arquivos órfãos.&lt;/li&gt;
&lt;li&gt;Manutenção de metadados.&lt;/li&gt;
&lt;li&gt;Otimização de layout.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma arquitetura típica:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Incremental Writes
        |
        v
Many Small Files
        |
        v
Compaction
        |
        v
Larger Optimized Files
        |
        v
Improved Read Efficiency
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas manutenção também consome recursos.&lt;/p&gt;

&lt;p&gt;Compactar dados frequentemente demais pode aumentar custos sem ganhos proporcionais.&lt;/p&gt;

&lt;p&gt;A política deve considerar o padrão de escrita e leitura.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. Retenção e Time Travel
&lt;/h2&gt;

&lt;p&gt;Imagine uma tabela que recebe alterações diariamente.&lt;/p&gt;

&lt;p&gt;Ao longo do tempo, podem existir múltiplos snapshots e arquivos antigos.&lt;/p&gt;

&lt;p&gt;Manter todas as versões indefinidamente pode aumentar custos.&lt;/p&gt;

&lt;p&gt;Por outro lado, remover versões cedo demais pode comprometer auditoria e recuperação.&lt;/p&gt;

&lt;p&gt;Precisamos equilibrar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Storage Cost
     |
     v
Retention Policy
     |
     v
Time Travel Requirements
     |
     v
Audit &amp;amp; Recovery
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Uma política de retenção deve considerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Requisitos regulatórios.&lt;/li&gt;
&lt;li&gt;Necessidades de auditoria.&lt;/li&gt;
&lt;li&gt;Janelas de reprocessamento.&lt;/li&gt;
&lt;li&gt;Frequência de atualização.&lt;/li&gt;
&lt;li&gt;Custo de armazenamento.&lt;/li&gt;
&lt;li&gt;Estratégia de backup.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Time Travel e backup são mecanismos complementares, não equivalentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Lakehouse e CDC
&lt;/h2&gt;

&lt;p&gt;Um dos cenários mais relevantes é integrar dados operacionais a uma plataforma analítica.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;             POSTGRESQL
                 |
                 v
              CDC EVENTS
                 |
                 v
                KAFKA
                 |
                 v
          SPARK STRUCTURED
              STREAMING
                 |
                 v
             LAKEHOUSE
                 |
                 v
          ANALYTICAL TABLES
                 |
                 v
              BI / ML / AI
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse tipo de arquitetura pode reduzir a dependência de cargas completas.&lt;/p&gt;

&lt;p&gt;Mas exige atenção a:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Chaves primárias.&lt;/li&gt;
&lt;li&gt;Semântica de exclusão.&lt;/li&gt;
&lt;li&gt;Ordem de eventos.&lt;/li&gt;
&lt;li&gt;Reprocessamento.&lt;/li&gt;
&lt;li&gt;Idempotência.&lt;/li&gt;
&lt;li&gt;Checkpoints.&lt;/li&gt;
&lt;li&gt;Consistência entre microbatches.&lt;/li&gt;
&lt;li&gt;Evolução de schema.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A tecnologia de tabela é apenas uma parte da solução.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Um exemplo de arquitetura corporativa
&lt;/h2&gt;

&lt;p&gt;Considere uma plataforma que recebe dados de CRM, ERP, APIs e eventos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                     DATA SOURCES
                 /       |       \
                v        v        v
               ERP      CRM      APIs
                \        |       /
                 v       v      v
                  INGESTION LAYER
                         |
                         v
                    RAW STORAGE
                         |
                         v
                 SPARK PROCESSING
                         |
                         v
                  LAKEHOUSE TABLES
                         |
              +----------+----------+
              |          |          |
              v          v          v
           Analytics     BI       ML / AI
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nessa arquitetura, podemos utilizar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Spark para processamento distribuído.&lt;/li&gt;
&lt;li&gt;Parquet para armazenamento colunar.&lt;/li&gt;
&lt;li&gt;Um formato de tabela para gerenciamento transacional.&lt;/li&gt;
&lt;li&gt;Um catálogo para descoberta e integração.&lt;/li&gt;
&lt;li&gt;Orquestração para execução.&lt;/li&gt;
&lt;li&gt;Observabilidade para operação.&lt;/li&gt;
&lt;li&gt;Data Quality para validação.&lt;/li&gt;
&lt;li&gt;Governança para controle de acesso e rastreabilidade.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O Lakehouse não é um produto isolado.&lt;/p&gt;

&lt;p&gt;É uma arquitetura composta por diferentes responsabilidades.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. O impacto em performance e custos
&lt;/h2&gt;

&lt;p&gt;Formatos de tabela podem melhorar a eficiência operacional, mas também introduzem custos de gerenciamento.&lt;/p&gt;

&lt;p&gt;Precisamos considerar:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Benefícios potenciais&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Leitura consistente.&lt;/li&gt;
&lt;li&gt;Melhor gerenciamento de arquivos.&lt;/li&gt;
&lt;li&gt;Pruning baseado em metadados.&lt;/li&gt;
&lt;li&gt;Atualizações incrementais.&lt;/li&gt;
&lt;li&gt;Redução de reprocessamentos completos.&lt;/li&gt;
&lt;li&gt;Histórico de versões.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Custos e desafios&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Metadados adicionais.&lt;/li&gt;
&lt;li&gt;Operações de manutenção.&lt;/li&gt;
&lt;li&gt;Compactação.&lt;/li&gt;
&lt;li&gt;Controle de concorrência.&lt;/li&gt;
&lt;li&gt;Compatibilidade entre engines.&lt;/li&gt;
&lt;li&gt;Evolução operacional.&lt;/li&gt;
&lt;li&gt;Retenção de versões.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma arquitetura Lakehouse bem projetada pode melhorar a relação entre performance, governança e custo.&lt;/p&gt;

&lt;p&gt;Mas não devemos assumir que será automaticamente mais barata em todos os cenários.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. O que eu avaliaria em uma prova de conceito
&lt;/h2&gt;

&lt;p&gt;Antes de escolher Iceberg, Delta ou Hudi para uma plataforma corporativa, eu construiria uma PoC com workloads reais.&lt;/p&gt;

&lt;p&gt;A avaliação incluiria:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Dimensão&lt;/th&gt;
&lt;th&gt;O que medir&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ingestão&lt;/td&gt;
&lt;td&gt;Throughput e latência&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Upsert&lt;/td&gt;
&lt;td&gt;Tempo e custo de atualização&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consulta&lt;/td&gt;
&lt;td&gt;Latência e volume lido&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Concorrência&lt;/td&gt;
&lt;td&gt;Comportamento de leituras e escritas simultâneas&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Schema&lt;/td&gt;
&lt;td&gt;Evolução e compatibilidade&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Manutenção&lt;/td&gt;
&lt;td&gt;Compactação e limpeza&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Metadados&lt;/td&gt;
&lt;td&gt;Tempo de planejamento&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recuperação&lt;/td&gt;
&lt;td&gt;Reprocessamento e histórico&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Interoperabilidade&lt;/td&gt;
&lt;td&gt;Operações suportadas por engine&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Custo&lt;/td&gt;
&lt;td&gt;Computação, armazenamento e operação&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A escolha deve ser fundamentada em resultados.&lt;/p&gt;

&lt;p&gt;Não apenas em popularidade ou preferência tecnológica.&lt;/p&gt;




&lt;h2&gt;
  
  
  31. Erros comuns em projetos Lakehouse
&lt;/h2&gt;

&lt;p&gt;Alguns erros merecem atenção.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tratar Parquet como uma tabela transacional.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Parquet é um formato físico de arquivo. Não oferece, sozinho, todo o gerenciamento de tabelas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Acreditar que ACID garante qualidade de dados.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Consistência transacional e qualidade de negócio são responsabilidades diferentes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ignorar o catálogo.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A integração entre engines depende de decisões de metadados e catálogo.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Habilitar retenção curta sem avaliar auditoria.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso pode eliminar versões necessárias.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Não planejar compactação.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pipelines incrementais podem acumular muitos arquivos pequenos.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Assumir compatibilidade total entre engines.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Recursos de leitura e escrita podem variar.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Escolher tecnologia sem benchmark.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;O melhor formato depende dos workloads.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Um modelo mental para decisões arquiteturais
&lt;/h2&gt;

&lt;p&gt;Quando preciso definir uma estratégia Lakehouse, começo com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 BUSINESS REQUIREMENTS
                          |
                          v
                     DATA WORKLOAD
                          |
            +-------------+-------------+
            |             |             |
            v             v             v
          Batch        Streaming       CDC
            |             |             |
            +-------------+-------------+
                          |
                          v
                  TABLE REQUIREMENTS
                          |
             +------------+------------+
             |            |            |
             v            v            v
           ACID        Time Travel    Evolution
             |            |            |
             +------------+------------+
                          |
                          v
                   ENGINE SUPPORT
                          |
                          v
                     CATALOG
                          |
                          v
                    TABLE FORMAT
                          |
                          v
                  STORAGE PLATFORM
                          |
                          v
                   OBSERVABILITY
                          |
                          v
                  COST &amp;amp; GOVERNANCE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa abordagem evita começar pela ferramenta antes de compreender o problema.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;Na Parte 12, aprendemos que formatos colunares como Parquet podem melhorar a eficiência de leitura.&lt;/p&gt;

&lt;p&gt;Agora demos um passo além.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Uma plataforma analítica moderna precisa gerenciar não apenas os bytes armazenados, mas também o estado lógico das tabelas.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso envolve:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Quais arquivos pertencem à tabela.&lt;/li&gt;
&lt;li&gt;Qual versão está ativa.&lt;/li&gt;
&lt;li&gt;Como alterações são confirmadas.&lt;/li&gt;
&lt;li&gt;Como operações concorrentes são tratadas.&lt;/li&gt;
&lt;li&gt;Como schemas evoluem.&lt;/li&gt;
&lt;li&gt;Como versões anteriores são consultadas.&lt;/li&gt;
&lt;li&gt;Como arquivos antigos são mantidos ou removidos.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Iceberg, Delta Lake e Hudi oferecem abordagens diferentes para resolver esses desafios.&lt;/p&gt;

&lt;p&gt;O princípio mais importante é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;O formato de arquivo organiza os dados. O formato de tabela organiza a consistência, a evolução e o ciclo de vida desses dados.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  34. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando alguém diz:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Vamos construir um Data Lake com Spark e Parquet.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Minha próxima pergunta é:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como essas tabelas serão gerenciadas?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Quero entender:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Como faremos UPDATE e DELETE?
              |
              v
Como trataremos CDC?
              |
              v
Como garantiremos consistência?
              |
              v
Qual catálogo utilizaremos?
              |
              v
Quais engines terão acesso?
              |
              v
Como evoluiremos o schema?
              |
              v
Como manteremos os arquivos?
              |
              v
Qual será a política de retenção?
              |
              v
Como mediremos custo e performance?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essas perguntas representam a diferença entre armazenar arquivos e construir uma plataforma de dados sustentável.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora entendemos os fundamentos das arquiteturas Lakehouse.&lt;/p&gt;

&lt;p&gt;Mas uma plataforma de dados não está completa apenas porque armazena e processa informações corretamente.&lt;/p&gt;

&lt;p&gt;Precisamos garantir que os dados sejam confiáveis, rastreáveis e governáveis.&lt;/p&gt;

&lt;p&gt;Na próxima parte, vamos estudar:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data Quality, Data Contracts e Governança de Dados com Apache Spark.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Vamos explorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qualidade de dados.&lt;/li&gt;
&lt;li&gt;Regras de validação.&lt;/li&gt;
&lt;li&gt;Schema Enforcement.&lt;/li&gt;
&lt;li&gt;Data Contracts.&lt;/li&gt;
&lt;li&gt;Data Lineage.&lt;/li&gt;
&lt;li&gt;Catálogos.&lt;/li&gt;
&lt;li&gt;Metadados.&lt;/li&gt;
&lt;li&gt;Observabilidade.&lt;/li&gt;
&lt;li&gt;Quarentena de registros.&lt;/li&gt;
&lt;li&gt;Testes automatizados.&lt;/li&gt;
&lt;li&gt;Monitoramento de pipelines.&lt;/li&gt;
&lt;li&gt;Integração com arquiteturas Lakehouse.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central será:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como transformar pipelines Spark em produtos de dados confiáveis e governáveis?&lt;/strong&gt;&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 12:&lt;/strong&gt; CSV, JSON, Parquet e ORC — performance começa no armazenamento&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 14 — Data Quality, Data Contracts e Governança de Dados com Apache Spark&lt;/p&gt;

</description>
      <category>data</category>
      <category>lake</category>
      <category>spark</category>
      <category>python</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 12: CSV, JSON, Parquet e ORC — performance começa no armazenamento</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 19:57:18 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-12-csv-json-parquet-e-orc-performance-comeca-no-armazenamento-5b7l</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-12-csv-json-parquet-e-orc-performance-comeca-no-armazenamento-5b7l</guid>
      <description>&lt;p&gt;&lt;em&gt;Nota histórica: o título desta série faz referência à minha trajetória em Engenharia de Dados iniciada em 2012. As tecnologias e práticas apresentadas evoluíram em momentos diferentes. Este artigo revisita esses fundamentos sob a perspectiva das arquiteturas modernas de dados.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nos capítulos anteriores, exploramos como o Apache Spark distribui o processamento entre Drivers, Executors e Tasks.&lt;/p&gt;

&lt;p&gt;Entendemos partições, shuffle, memória, lineage, tolerância a falhas e dimensionamento de recursos.&lt;/p&gt;

&lt;p&gt;Na Parte 11, discutimos uma questão importante:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como utilizar CPU, memória e paralelismo de maneira eficiente?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Agora vamos examinar um fator que pode mudar significativamente o comportamento de uma aplicação sem alterar sua lógica de negócio:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O formato dos dados.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Imagine dois pipelines executando a mesma consulta sobre o mesmo conjunto de informações.&lt;/p&gt;

&lt;p&gt;O primeiro lê arquivos CSV.&lt;/p&gt;

&lt;p&gt;O segundo lê arquivos Parquet.&lt;/p&gt;

&lt;p&gt;Ambos executam:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;vendas&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado = &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;SP&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;categoria&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A lógica é idêntica.&lt;/p&gt;

&lt;p&gt;Mas a quantidade de dados lidos, o trabalho de interpretação e o plano de execução podem ser diferentes.&lt;/p&gt;

&lt;p&gt;Por quê?&lt;/p&gt;

&lt;p&gt;Porque o Spark não processa dados independentemente de como eles estão armazenados.&lt;/p&gt;

&lt;p&gt;O armazenamento influencia diretamente a leitura.&lt;/p&gt;

&lt;p&gt;E a leitura influencia o custo do processamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. O formato dos dados faz parte da arquitetura
&lt;/h2&gt;

&lt;p&gt;Quando pensamos em performance, normalmente lembramos de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Quantidade de Executors.&lt;/li&gt;
&lt;li&gt;Memória disponível.&lt;/li&gt;
&lt;li&gt;Número de partições.&lt;/li&gt;
&lt;li&gt;Estratégias de join.&lt;/li&gt;
&lt;li&gt;Shuffle.&lt;/li&gt;
&lt;li&gt;CPU.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas existe uma camada anterior:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como os dados foram gravados?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                   DATA SOURCE
                       |
                       v
                 FILE FORMAT
                       |
                       v
                  DATA READER
                       |
                       v
                SPARK DATAFRAME
                       |
                       v
                 QUERY PLAN
                       |
                       v
                 EXECUTION
                       |
                       v
                    RESULT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O formato pode determinar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Como os registros são representados.&lt;/li&gt;
&lt;li&gt;Como os tipos são armazenados.&lt;/li&gt;
&lt;li&gt;Quais metadados estão disponíveis.&lt;/li&gt;
&lt;li&gt;Quais colunas precisam ser lidas.&lt;/li&gt;
&lt;li&gt;Quais filtros podem ser aplicados antecipadamente.&lt;/li&gt;
&lt;li&gt;Quanto espaço os dados ocupam.&lt;/li&gt;
&lt;li&gt;Quanto trabalho de parsing será necessário.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Portanto:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Otimização de Spark não começa necessariamente no Executor. Muitas vezes começa na organização dos dados.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  2. Row-based vs. Columnar
&lt;/h2&gt;

&lt;p&gt;Uma das distinções mais importantes é entre armazenamento orientado a linhas e armazenamento orientado a colunas.&lt;/p&gt;

&lt;p&gt;Imagine uma tabela:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;id&lt;/th&gt;
&lt;th&gt;cliente&lt;/th&gt;
&lt;th&gt;estado&lt;/th&gt;
&lt;th&gt;valor&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Ana&lt;/td&gt;
&lt;td&gt;SP&lt;/td&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Bruno&lt;/td&gt;
&lt;td&gt;RJ&lt;/td&gt;
&lt;td&gt;200&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Carla&lt;/td&gt;
&lt;td&gt;SP&lt;/td&gt;
&lt;td&gt;300&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Em uma representação orientada a linhas, os valores de cada registro são armazenados próximos uns dos outros.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ROW-ORIENTED

[1, Ana, SP, 100]
[2, Bruno, RJ, 200]
[3, Carla, SP, 300]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em uma representação orientada a colunas, valores da mesma coluna são agrupados.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;COLUMNAR

id:
[1, 2, 3]

cliente:
[Ana, Bruno, Carla]

estado:
[SP, RJ, SP]

valor:
[100, 200, 300]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa diferença influencia o tipo de operação que cada formato favorece.&lt;/p&gt;

&lt;p&gt;Se queremos recuperar registros completos, uma organização por linhas pode ser conveniente.&lt;/p&gt;

&lt;p&gt;Se queremos calcular:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;vendas&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;uma organização colunar pode permitir ler principalmente a coluna &lt;code&gt;valor&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;É por isso que formatos colunares são tão importantes em workloads analíticos.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. CSV: simples, interoperável e caro de interpretar
&lt;/h2&gt;

&lt;p&gt;CSV significa &lt;em&gt;Comma-Separated Values&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;É um formato textual amplamente utilizado para intercâmbio de dados.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;id,cliente,estado,valor
1,Ana,SP,100
2,Bruno,RJ,200
3,Carla,SP,300
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sua principal vantagem é a simplicidade.&lt;/p&gt;

&lt;p&gt;Arquivos CSV podem ser produzidos e consumidos por uma enorme variedade de ferramentas.&lt;/p&gt;

&lt;p&gt;Mas existem limitações.&lt;/p&gt;

&lt;p&gt;CSV não preserva nativamente um schema tipado como o de um formato colunar.&lt;/p&gt;

&lt;p&gt;O valor:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;pode representar um inteiro, decimal ou texto, dependendo da interpretação.&lt;/p&gt;

&lt;p&gt;Além disso, delimitadores, aspas, escapes, valores nulos e codificação precisam ser tratados corretamente.&lt;/p&gt;

&lt;p&gt;Em grandes volumes, esse parsing pode representar um custo relevante.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Lendo CSV com PySpark
&lt;/h2&gt;

&lt;p&gt;Um exemplo básico:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FormatosDeDados&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df_csv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;header&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df_csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;printSchema&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sem schema explícito ou inferência habilitada, as colunas normalmente são interpretadas como strings.&lt;/p&gt;

&lt;p&gt;Podemos utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_csv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;header&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;inferSchema&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas a inferência exige examinar os dados para determinar tipos.&lt;/p&gt;

&lt;p&gt;Em pipelines de produção, quando conhecemos o contrato dos dados, geralmente é melhor fornecer o schema.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Schema explícito: menos ambiguidade
&lt;/h2&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.types&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;IntegerType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;DecimalType&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;IntegerType&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;DecimalType&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;df_csv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;header&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora declaramos a estrutura esperada.&lt;/p&gt;

&lt;p&gt;Isso traz benefícios:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Contrato de tipos mais claro.&lt;/li&gt;
&lt;li&gt;Menor dependência de inferência.&lt;/li&gt;
&lt;li&gt;Comportamento mais previsível.&lt;/li&gt;
&lt;li&gt;Melhor rastreabilidade.&lt;/li&gt;
&lt;li&gt;Validação mais objetiva.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Entretanto, declarar um campo como não anulável não substitui uma rotina completa de qualidade de dados. A validação de registros malformados e de restrições de negócio deve ser tratada explicitamente.&lt;/p&gt;

&lt;p&gt;Essa preocupação é particularmente importante quando recebemos arquivos de sistemas externos.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. JSON: flexibilidade com custo de parsing
&lt;/h2&gt;

&lt;p&gt;JSON é outro formato muito utilizado.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"cliente"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Ana"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"estado"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"SP"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"valor"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ele permite representar estruturas mais complexas, como arrays e objetos aninhados.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pedido_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"cliente"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"nome"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Ana"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"itens"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"produto"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"A"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"quantidade"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"produto"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"B"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"quantidade"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa flexibilidade é útil em APIs, eventos e integrações.&lt;/p&gt;

&lt;p&gt;Mas existe um custo.&lt;/p&gt;

&lt;p&gt;O Spark precisa interpretar a estrutura textual e convertê-la para os tipos internos utilizados no processamento.&lt;/p&gt;

&lt;p&gt;Além disso, schemas variáveis podem dificultar governança e evolução de pipelines.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Lendo JSON no Spark
&lt;/h2&gt;

&lt;p&gt;Para arquivos JSON Lines, com um objeto JSON por linha:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_json&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/eventos.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos fornecer um schema explícito:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.types&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;StringType&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;schema_eventos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;StructType&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LongType&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nc"&gt;StructField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;StringType&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;df_json&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;schema_eventos&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/eventos.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;É importante distinguir JSON Lines de um documento JSON convencional contendo um array ou objeto multilinha.&lt;/p&gt;

&lt;p&gt;Para determinados arquivos JSON multilinha, podemos precisar de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_json&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;multiLine&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/eventos_multilinha.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O modo multilinha pode afetar a estratégia de leitura e o paralelismo. Por isso, não devemos habilitá-lo indiscriminadamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Parquet: armazenamento colunar para analytics
&lt;/h2&gt;

&lt;p&gt;Agora chegamos a um dos formatos mais importantes no ecossistema moderno de Engenharia de Dados.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Apache Parquet&lt;/strong&gt; é um formato colunar desenvolvido para armazenamento e processamento analítico.&lt;/p&gt;

&lt;p&gt;Ele suporta:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Schema tipado.&lt;/li&gt;
&lt;li&gt;Compressão.&lt;/li&gt;
&lt;li&gt;Codificações eficientes.&lt;/li&gt;
&lt;li&gt;Metadados.&lt;/li&gt;
&lt;li&gt;Organização em grupos de linhas.&lt;/li&gt;
&lt;li&gt;Leitura seletiva de colunas.&lt;/li&gt;
&lt;li&gt;Estatísticas que podem auxiliar a eliminação de dados desnecessários.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Considere uma consulta:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;estado&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;vendas&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;estado&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se o dataset possui 80 colunas, mas a consulta utiliza apenas duas, o Parquet pode permitir uma leitura muito mais seletiva do que um arquivo textual tradicional.&lt;/p&gt;

&lt;p&gt;Essa é uma de suas principais vantagens.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Lendo e escrevendo Parquet
&lt;/h2&gt;

&lt;p&gt;Leitura:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_parquet&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas_parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Escrita:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas_parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora temos um dataset armazenado em formato colunar.&lt;/p&gt;

&lt;p&gt;O schema é preservado nos metadados do formato.&lt;/p&gt;

&lt;p&gt;Podemos inspecioná-lo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_parquet&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;printSchema&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E executar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;spark_sum&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_parquet&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SP&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;agg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nf"&gt;spark_sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;total&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa consulta pode se beneficiar de diferentes otimizações de leitura e execução.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Column Pruning
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Column Pruning&lt;/strong&gt; significa evitar a leitura de colunas que não são necessárias para a consulta.&lt;/p&gt;

&lt;p&gt;Imagine um dataset com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;customer_id
customer_name
email
phone
address
city
state
country
product_id
product_name
quantity
price
discount
tax
shipping
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas a consulta utiliza apenas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df_parquet&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Com um formato colunar e um leitor compatível, Spark pode solicitar apenas as colunas necessárias.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PARQUET DATASET
      |
      +-- customer_id  [READ]
      +-- customer_name
      +-- email
      +-- phone
      +-- address
      +-- state
      +-- price        [READ]
      +-- tax
      +-- shipping
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode reduzir leitura, transferência e trabalho de decodificação.&lt;/p&gt;

&lt;p&gt;A economia real depende do layout físico, da compressão, do sistema de armazenamento e do plano de execução.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. Predicate Pushdown
&lt;/h2&gt;

&lt;p&gt;Outro conceito importante é &lt;strong&gt;Predicate Pushdown&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;filtrado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df_parquet&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SP&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Spark pode encaminhar filtros compatíveis ao leitor da fonte.&lt;/p&gt;

&lt;p&gt;Com Parquet, estatísticas e metadados podem ajudar a evitar a leitura de determinados grupos de dados que não satisfazem o filtro.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PARQUET FILE
    |
    +-- Row Group A: estado = RJ
    |
    +-- Row Group B: estado = SP
    |
    +-- Row Group C: estado = MG
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Para uma consulta restrita a &lt;code&gt;SP&lt;/code&gt;, metadados adequados podem permitir ignorar alguns grupos.&lt;/p&gt;

&lt;p&gt;Mas há uma ressalva importante:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Predicate Pushdown não significa que todo filtro será resolvido integralmente pelo armazenamento.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;O Spark ainda pode precisar avaliar linhas após a leitura.&lt;/p&gt;

&lt;p&gt;O resultado depende do predicado, do formato, das estatísticas disponíveis e do leitor utilizado.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. Como verificar o plano de leitura
&lt;/h2&gt;

&lt;p&gt;Podemos utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;filtrado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;formatted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No plano físico, vale procurar informações como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ReadSchema
PushedFilters
PartitionFilters
DataFilters
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esses campos ajudam a entender quais colunas estão sendo solicitadas e como os filtros estão sendo tratados.&lt;/p&gt;

&lt;p&gt;Por exemplo, um &lt;code&gt;ReadSchema&lt;/code&gt; contendo apenas duas colunas indica que a leitura foi planejada para solicitar somente essas colunas.&lt;/p&gt;

&lt;p&gt;Já &lt;code&gt;PushedFilters&lt;/code&gt; mostra filtros encaminhados à fonte.&lt;/p&gt;

&lt;p&gt;Entretanto, a presença de um filtro no plano não comprova, sozinha, quantos bytes foram efetivamente economizados.&lt;/p&gt;

&lt;p&gt;Para isso, precisamos combinar análise do plano com métricas de execução e medições do armazenamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. ORC: outro formato colunar importante
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;ORC — Optimized Row Columnar&lt;/strong&gt; — também é um formato colunar voltado a workloads analíticos.&lt;/p&gt;

&lt;p&gt;Assim como Parquet, oferece recursos como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Armazenamento tipado.&lt;/li&gt;
&lt;li&gt;Compressão.&lt;/li&gt;
&lt;li&gt;Estatísticas.&lt;/li&gt;
&lt;li&gt;Leitura seletiva.&lt;/li&gt;
&lt;li&gt;Organização eficiente para consultas analíticas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_orc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;orc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas_orc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Escrita:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;orc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas_orc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ORC possui forte presença histórica no ecossistema Hive e em determinadas arquiteturas analíticas.&lt;/p&gt;

&lt;p&gt;Parquet e ORC são ambos formatos colunares capazes de oferecer boa performance.&lt;/p&gt;

&lt;p&gt;A escolha deve considerar o ecossistema, a compatibilidade e os workloads reais.&lt;/p&gt;

&lt;p&gt;Não existe um vencedor universal.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. Comparando os quatro formatos
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Característica&lt;/th&gt;
&lt;th&gt;CSV&lt;/th&gt;
&lt;th&gt;JSON&lt;/th&gt;
&lt;th&gt;Parquet&lt;/th&gt;
&lt;th&gt;ORC&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Representação&lt;/td&gt;
&lt;td&gt;Texto tabular&lt;/td&gt;
&lt;td&gt;Texto estruturado&lt;/td&gt;
&lt;td&gt;Binário colunar&lt;/td&gt;
&lt;td&gt;Binário colunar&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Schema tipado embutido&lt;/td&gt;
&lt;td&gt;Não&lt;/td&gt;
&lt;td&gt;Não como contrato rígido&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Estruturas aninhadas&lt;/td&gt;
&lt;td&gt;Limitadas&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Leitura seletiva de colunas&lt;/td&gt;
&lt;td&gt;Limitada&lt;/td&gt;
&lt;td&gt;Limitada&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compressão e codificação colunar&lt;/td&gt;
&lt;td&gt;Não&lt;/td&gt;
&lt;td&gt;Não&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Uso frequente&lt;/td&gt;
&lt;td&gt;Intercâmbio&lt;/td&gt;
&lt;td&gt;APIs e eventos&lt;/td&gt;
&lt;td&gt;Data Lakes analíticos&lt;/td&gt;
&lt;td&gt;Analytics e ecossistema Hive&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Facilidade de leitura humana&lt;/td&gt;
&lt;td&gt;Alta&lt;/td&gt;
&lt;td&gt;Alta&lt;/td&gt;
&lt;td&gt;Baixa&lt;/td&gt;
&lt;td&gt;Baixa&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A escolha depende da finalidade.&lt;/p&gt;

&lt;p&gt;Para integração simples entre sistemas, CSV pode ser suficiente.&lt;/p&gt;

&lt;p&gt;Para eventos e APIs, JSON pode ser conveniente.&lt;/p&gt;

&lt;p&gt;Para processamento analítico recorrente, Parquet e ORC frequentemente oferecem vantagens importantes.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Compressão: menos bytes não significa sempre menos tempo
&lt;/h2&gt;

&lt;p&gt;Formatos colunares podem utilizar diferentes codecs de compressão.&lt;/p&gt;

&lt;p&gt;No Parquet, por exemplo, podemos configurar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;compression&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;snappy&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas_snappy&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Também existem outros codecs, conforme o ambiente e a versão utilizados.&lt;/p&gt;

&lt;p&gt;Compressão envolve um trade-off:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Menos dados armazenados
          |
          v
Menos bytes para transferir
          |
          v
Mais trabalho de compressão/descompressão
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em determinadas situações, reduzir I/O compensa o custo de CPU.&lt;/p&gt;

&lt;p&gt;Em outras, codecs mais pesados podem aumentar a latência.&lt;/p&gt;

&lt;p&gt;Por isso, a escolha deve considerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Frequência de leitura.&lt;/li&gt;
&lt;li&gt;Frequência de escrita.&lt;/li&gt;
&lt;li&gt;Custo de armazenamento.&lt;/li&gt;
&lt;li&gt;CPU disponível.&lt;/li&gt;
&lt;li&gt;Latência.&lt;/li&gt;
&lt;li&gt;Taxa de compressão.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  16. Partition Pruning
&lt;/h2&gt;

&lt;p&gt;Agora precisamos distinguir dois conceitos que possuem nomes semelhantes.&lt;/p&gt;

&lt;p&gt;Na Parte 3, estudamos as partições utilizadas pelo Spark para distribuir o processamento.&lt;/p&gt;

&lt;p&gt;Aqui vamos falar também de &lt;strong&gt;particionamento físico de datasets por diretórios&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Imagine um Data Lake organizado assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;vendas/
  ano=2025/
    mes=01/
    mes=02/
  ano=2026/
    mes=01/
    mes=02/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos ler:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E filtrar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ano&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;2026&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Quando o particionamento é reconhecido corretamente, o Spark pode evitar a leitura de diretórios que não correspondem aos filtros.&lt;/p&gt;

&lt;p&gt;Esse mecanismo é conhecido como &lt;strong&gt;Partition Pruning&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Predicate Pushdown vs. Partition Pruning
&lt;/h2&gt;

&lt;p&gt;Esses mecanismos são diferentes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Partition Pruning&lt;/strong&gt; utiliza informações de particionamento do dataset para eliminar arquivos ou diretórios que não precisam ser considerados.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Predicate Pushdown&lt;/strong&gt; encaminha predicados ao leitor da fonte para permitir otimizações adicionais.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;QUERY
  |
  v
Partition Pruning
  |
  v
Relevant Files
  |
  v
Predicate Pushdown
  |
  v
Relevant Data Groups / Records
  |
  v
Spark Processing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os dois podem trabalhar juntos.&lt;/p&gt;

&lt;p&gt;E podem reduzir significativamente o volume de dados efetivamente processado.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. Escrevendo dados particionados
&lt;/h2&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;partitionBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ano&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas_particionadas&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode gerar uma estrutura semelhante a:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;vendas_particionadas/
  ano=2025/
    mes=1/
      part-00000-...
  ano=2026/
    mes=1/
      part-00000-...
    mes=2/
      part-00000-...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A quantidade de arquivos depende do particionamento de execução, da distribuição dos dados e da escrita.&lt;/p&gt;

&lt;p&gt;O &lt;code&gt;partitionBy()&lt;/code&gt; não garante um único arquivo por diretório.&lt;/p&gt;

&lt;p&gt;Essa distinção é importante.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. O perigo do particionamento excessivo
&lt;/h2&gt;

&lt;p&gt;Imagine particionar dados por:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;customer_id
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;quando existem milhões de clientes.&lt;/p&gt;

&lt;p&gt;Podemos acabar criando um número enorme de diretórios e arquivos pequenos.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;customer_id=1/
customer_id=2/
customer_id=3/
...
customer_id=1000000/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode gerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Sobrecarga de metadados.&lt;/li&gt;
&lt;li&gt;Listagem de muitos objetos.&lt;/li&gt;
&lt;li&gt;Arquivos pequenos.&lt;/li&gt;
&lt;li&gt;Maior custo operacional.&lt;/li&gt;
&lt;li&gt;Planejamento de leitura mais caro.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Portanto:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Particionar fisicamente um Data Lake não significa simplesmente escolher todas as colunas utilizadas em filtros.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Precisamos avaliar cardinalidade, distribuição, volume por partição e padrões reais de consulta.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Small Files Problem
&lt;/h2&gt;

&lt;p&gt;Um dos problemas mais comuns em Data Lakes é o excesso de arquivos pequenos.&lt;/p&gt;

&lt;p&gt;Imagine dois datasets com o mesmo volume total.&lt;/p&gt;

&lt;h3&gt;
  
  
  Dataset A
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100 arquivos de 100 MB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Total aproximado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Dataset B
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100.000 arquivos de 100 KB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Total aproximado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Apesar do volume semelhante, o comportamento pode ser muito diferente.&lt;/p&gt;

&lt;p&gt;No Dataset B, Spark e o sistema de armazenamento precisam lidar com um número muito maior de objetos.&lt;/p&gt;

&lt;p&gt;Isso pode aumentar custos de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Listagem.&lt;/li&gt;
&lt;li&gt;Metadados.&lt;/li&gt;
&lt;li&gt;Abertura de arquivos.&lt;/li&gt;
&lt;li&gt;Planejamento.&lt;/li&gt;
&lt;li&gt;Scheduling.&lt;/li&gt;
&lt;li&gt;Operações remotas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Em armazenamento de objetos, esse overhead pode ser especialmente relevante.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. Por que arquivos pequenos aparecem?
&lt;/h2&gt;

&lt;p&gt;Existem várias causas possíveis:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Microbatches frequentes.&lt;/li&gt;
&lt;li&gt;Escritas muito particionadas.&lt;/li&gt;
&lt;li&gt;Muitos Executors escrevendo simultaneamente.&lt;/li&gt;
&lt;li&gt;Particionamento por colunas de alta cardinalidade.&lt;/li&gt;
&lt;li&gt;Pipelines incrementais sem compactação.&lt;/li&gt;
&lt;li&gt;Estratégias inadequadas de saída.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Imagine um pipeline que escreve pequenos arquivos a cada minuto.&lt;/p&gt;

&lt;p&gt;Mesmo que cada execução seja rápida, o acúmulo ao longo de meses pode produzir uma estrutura difícil de consultar.&lt;/p&gt;

&lt;p&gt;Por isso, otimização de armazenamento não é apenas uma atividade pontual.&lt;/p&gt;

&lt;p&gt;Também é manutenção operacional.&lt;/p&gt;




&lt;h2&gt;
  
  
  22. &lt;code&gt;repartition()&lt;/code&gt; e &lt;code&gt;coalesce()&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Spark oferece mecanismos para modificar o número de partições de um DataFrame.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_reparticionado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;repartition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa operação normalmente envolve redistribuição de dados, podendo introduzir shuffle.&lt;/p&gt;

&lt;p&gt;Já:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_reduzido&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;coalesce&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;é frequentemente utilizada para reduzir o número de partições sem um shuffle completo.&lt;/p&gt;

&lt;p&gt;Mas existe uma ressalva:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reduzir partições não significa necessariamente equilibrar os dados.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;coalesce()&lt;/code&gt; pode preservar desequilíbrios e reduzir o paralelismo.&lt;/p&gt;

&lt;p&gt;Portanto, a escolha depende do objetivo.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. Controlando arquivos de saída
&lt;/h2&gt;

&lt;p&gt;Imagine que queremos escrever um dataset com um número mais controlado de partições de execução.&lt;/p&gt;

&lt;p&gt;Podemos utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;repartition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/saida&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em uma escrita simples, isso pode produzir aproximadamente um arquivo de dados por partição de escrita, mas o resultado não deve ser tratado como garantia universal.&lt;/p&gt;

&lt;p&gt;A quantidade real de arquivos pode variar por causa de particionamento físico, limites de registros por arquivo e comportamento da plataforma.&lt;/p&gt;

&lt;p&gt;Também existe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;maxRecordsPerFile&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;500_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/saida&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse parâmetro limita a quantidade de registros por arquivo, mas não garante um tamanho específico em MB.&lt;/p&gt;

&lt;p&gt;O tamanho depende da estrutura e da compressibilidade dos dados.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Por que &lt;code&gt;coalesce(1)&lt;/code&gt; pode ser perigoso?
&lt;/h2&gt;

&lt;p&gt;É comum encontrar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;coalesce&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/saida&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A intenção geralmente é produzir um único arquivo.&lt;/p&gt;

&lt;p&gt;Para datasets pequenos, isso pode ser aceitável.&lt;/p&gt;

&lt;p&gt;Mas para grandes volumes, concentrar a escrita em uma única partição pode criar um gargalo.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100 Partitions
      |
      v
coalesce(1)
      |
      v
1 Output Partition
      |
      v
Limited Parallelism
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode aumentar o tempo de escrita e a pressão de recursos.&lt;/p&gt;

&lt;p&gt;Além disso, a API de escrita Spark normalmente produz um diretório de saída contendo arquivos de dados e metadados, não necessariamente um arquivo com o nome exato informado.&lt;/p&gt;

&lt;p&gt;Em produção, geralmente prefiro manter a escrita distribuída e utilizar estratégias apropriadas de compactação quando necessário.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. Um experimento reproduzível
&lt;/h2&gt;

&lt;p&gt;Vamos construir um pequeno laboratório para comparar CSV, JSON, Parquet e ORC.&lt;/p&gt;

&lt;p&gt;O objetivo não é provar que um formato será sempre mais rápido.&lt;/p&gt;

&lt;p&gt;É aprender a medir.&lt;/p&gt;

&lt;p&gt;Primeiro, criamos um dataset sintético:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;concat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;pmod&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;master&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;local[*]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;BenchmarkFormatos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1_000_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;dados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;concat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente_&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;cast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;string&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cliente&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;pmod&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;int&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado_codigo&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;pmod&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;lit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10_000&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;100.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora escrevemos nos quatro formatos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/tmp/spark-format-benchmark&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dados&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;header&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dados&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dados&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dados&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;overwrite&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;orc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/orc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse diretório é apropriado para um experimento local. Em um cluster, os caminhos precisam ser acessíveis conforme o ambiente de execução.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. Medindo o tempo de leitura e consulta
&lt;/h2&gt;

&lt;p&gt;Vamos utilizar uma função simples de benchmark.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;perf_counter&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;medir_consulta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nome&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;carregar&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;inicio&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;carregar&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;df&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado_codigo&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;agg&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sum&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;duracao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;inicio&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;nome&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;duracao&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;s | &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resultado=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;duracao&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Para evitar que o CSV dependa de inferência de schema, utilizamos o schema conhecido:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dados&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora executamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;tempos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;

&lt;span class="n"&gt;tempos&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CSV&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;medir_consulta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CSV&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;header&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;tempos&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;JSON&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;medir_consulta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;JSON&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;tempos&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;medir_consulta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;tempos&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ORC&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;medir_consulta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ORC&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;orc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/orc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os resultados dependem da máquina, do sistema de arquivos, da versão do Spark e das condições de execução.&lt;/p&gt;

&lt;p&gt;Não devemos apresentar tempos hipotéticos como resultados medidos.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Como tornar o benchmark mais confiável
&lt;/h2&gt;

&lt;p&gt;Uma única execução não é suficiente para conclusões definitivas.&lt;/p&gt;

&lt;p&gt;Existem fatores como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Inicialização da JVM.&lt;/li&gt;
&lt;li&gt;Aquecimento de código.&lt;/li&gt;
&lt;li&gt;Cache do sistema operacional.&lt;/li&gt;
&lt;li&gt;Cache de metadados.&lt;/li&gt;
&lt;li&gt;Concorrência.&lt;/li&gt;
&lt;li&gt;Variações de disco.&lt;/li&gt;
&lt;li&gt;Quantidade de arquivos.&lt;/li&gt;
&lt;li&gt;Compressão.&lt;/li&gt;
&lt;li&gt;Planejamento da consulta.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Para um benchmark mais rigoroso, devemos:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Executar uma rodada de aquecimento.&lt;/li&gt;
&lt;li&gt;Repetir cada cenário várias vezes.&lt;/li&gt;
&lt;li&gt;Alternar a ordem dos formatos.&lt;/li&gt;
&lt;li&gt;Registrar mediana e dispersão.&lt;/li&gt;
&lt;li&gt;Comparar volumes físicos.&lt;/li&gt;
&lt;li&gt;Inspecionar os planos de execução.&lt;/li&gt;
&lt;li&gt;Monitorar CPU, I/O e memória.&lt;/li&gt;
&lt;li&gt;Utilizar datasets representativos.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Também é importante separar dois experimentos diferentes:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Benchmark de escrita:&lt;/strong&gt; mede o custo de produzir cada formato.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Benchmark de leitura:&lt;/strong&gt; mede o custo de consultar dados já armazenados.&lt;/p&gt;

&lt;p&gt;Misturar os dois pode esconder trade-offs importantes.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Como inspecionar as otimizações
&lt;/h2&gt;

&lt;p&gt;Vamos comparar os planos físicos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_csv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;header&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;df_parquet&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;diretorio&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/parquet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Definimos consultas equivalentes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;consulta_csv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_csv&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado_codigo&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;consulta_parquet&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df_parquet&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado_codigo&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;consulta_csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;formatted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;consulta_parquet&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;formatted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Compare:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ReadSchema
PushedFilters
DataFilters
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A análise mostra como o Spark planeja a leitura de cada formato.&lt;/p&gt;

&lt;p&gt;O objetivo não é apenas medir segundos.&lt;/p&gt;

&lt;p&gt;É compreender &lt;strong&gt;por que&lt;/strong&gt; os resultados são diferentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. O formato influencia o custo do Data Lake
&lt;/h2&gt;

&lt;p&gt;Imagine um Data Lake que armazena centenas de terabytes.&lt;/p&gt;

&lt;p&gt;O formato influencia:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Storage Size
     |
     v
Data Scanned
     |
     v
I/O Operations
     |
     v
Compute Time
     |
     v
Infrastructure Cost
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Um formato colunar bem organizado pode reduzir a quantidade de dados lidos em determinadas consultas.&lt;/p&gt;

&lt;p&gt;Isso pode diminuir:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tempo de processamento.&lt;/li&gt;
&lt;li&gt;Tráfego de dados.&lt;/li&gt;
&lt;li&gt;Utilização de CPU.&lt;/li&gt;
&lt;li&gt;Custo de armazenamento.&lt;/li&gt;
&lt;li&gt;Custo computacional.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas a economia depende do workload.&lt;/p&gt;

&lt;p&gt;Uma transformação que precisa ler todas as colunas e todos os registros pode apresentar ganhos diferentes de uma consulta altamente seletiva.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. Parquet não resolve tudo
&lt;/h2&gt;

&lt;p&gt;É importante evitar a conclusão:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Basta converter tudo para Parquet e a performance estará resolvida.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Não é assim.&lt;/p&gt;

&lt;p&gt;Podemos ter um dataset Parquet com:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Milhões de arquivos pequenos.&lt;/li&gt;
&lt;li&gt;Particionamento físico inadequado.&lt;/li&gt;
&lt;li&gt;Data skew.&lt;/li&gt;
&lt;li&gt;Compressão mal escolhida.&lt;/li&gt;
&lt;li&gt;Schemas inconsistentes.&lt;/li&gt;
&lt;li&gt;Estatísticas pouco úteis.&lt;/li&gt;
&lt;li&gt;Leituras repetidas desnecessárias.&lt;/li&gt;
&lt;li&gt;Joins caros.&lt;/li&gt;
&lt;li&gt;Layout incompatível com os filtros.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O formato é apenas uma dimensão da arquitetura.&lt;/p&gt;

&lt;p&gt;Uma organização ruim pode comprometer parte das vantagens do armazenamento colunar.&lt;/p&gt;




&lt;h2&gt;
  
  
  31. Formato de arquivo vs. formato de tabela
&lt;/h2&gt;

&lt;p&gt;Outra distinção importante nas arquiteturas modernas é separar o formato físico dos arquivos do formato de tabela.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Apache Parquet
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;é um formato de arquivo.&lt;/p&gt;

&lt;p&gt;Já tecnologias como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Apache Iceberg
Delta Lake
Apache Hudi
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;adicionam mecanismos de gerenciamento de tabelas e metadados, com capacidades que variam entre elas.&lt;/p&gt;

&lt;p&gt;Esses mecanismos podem envolver:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Controle de snapshots.&lt;/li&gt;
&lt;li&gt;Evolução de schema.&lt;/li&gt;
&lt;li&gt;Gerenciamento de arquivos.&lt;/li&gt;
&lt;li&gt;Operações transacionais.&lt;/li&gt;
&lt;li&gt;Evolução de partições.&lt;/li&gt;
&lt;li&gt;Manutenção e compactação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Muitas dessas tecnologias utilizam Parquet como formato físico de dados.&lt;/p&gt;

&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TABLE FORMAT
     |
     v
TABLE METADATA
     |
     v
DATA FILES
     |
     v
PARQUET
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa separação é fundamental para compreender arquiteturas Lakehouse.&lt;/p&gt;

&lt;p&gt;Mas formatos de tabela merecem um capítulo próprio.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Como escolher um formato?
&lt;/h2&gt;

&lt;p&gt;Minha decisão começa pelo objetivo.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Intercâmbio simples entre sistemas&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;CSV pode ser suficiente quando o contrato é claro e o volume é administrável.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Eventos, APIs e estruturas semiestruturadas&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;JSON pode ser conveniente, especialmente em camadas de ingestão.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Consultas analíticas recorrentes&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Parquet costuma ser uma excelente opção inicial.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ambientes com integração forte ao ecossistema ORC/Hive&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;ORC pode ser uma escolha natural.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tabelas analíticas com requisitos transacionais e evolução operacional&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Vale avaliar um formato de tabela apropriado, além do formato físico.&lt;/p&gt;

&lt;p&gt;Não se trata de escolher a tecnologia mais sofisticada.&lt;/p&gt;

&lt;p&gt;Trata-se de escolher a representação adequada ao uso.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. Uma arquitetura por camadas
&lt;/h2&gt;

&lt;p&gt;Em um Data Lake moderno, podemos encontrar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                   DATA SOURCES
                        |
          +-------------+-------------+
          |             |             |
          v             v             v
         APIs         Files         Events
          |             |             |
          +-------------+-------------+
                        |
                        v
                    RAW LAYER
                 JSON / CSV / etc.
                        |
                        v
                 VALIDATION &amp;amp; ETL
                        |
                        v
                 CURATED LAYER
                    PARQUET
                        |
                        v
                 ANALYTICAL TABLES
                        |
                        v
                 SQL / BI / ML / AI
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa é uma arquitetura conceitual, não uma regra universal.&lt;/p&gt;

&lt;p&gt;Dependendo do sistema, podemos preservar dados brutos em formatos originais e utilizar formatos colunares em camadas de consumo.&lt;/p&gt;

&lt;p&gt;Também podemos utilizar formatos de tabela para gerenciar datasets analíticos.&lt;/p&gt;

&lt;p&gt;O importante é que cada camada tenha uma responsabilidade clara.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Checklist de produção
&lt;/h2&gt;

&lt;p&gt;Antes de definir o armazenamento de um pipeline Spark, gosto de responder:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qual é a origem dos dados?&lt;/li&gt;
&lt;li&gt;O schema é estável?&lt;/li&gt;
&lt;li&gt;Existem estruturas aninhadas?&lt;/li&gt;
&lt;li&gt;Quais colunas são consultadas?&lt;/li&gt;
&lt;li&gt;Quais filtros são mais frequentes?&lt;/li&gt;
&lt;li&gt;Qual é a cardinalidade das colunas candidatas a particionamento?&lt;/li&gt;
&lt;li&gt;Quantos arquivos serão produzidos?&lt;/li&gt;
&lt;li&gt;Qual é o tamanho médio dos arquivos?&lt;/li&gt;
&lt;li&gt;Existe necessidade de compactação?&lt;/li&gt;
&lt;li&gt;Quais codecs são suportados?&lt;/li&gt;
&lt;li&gt;O formato permite column pruning?&lt;/li&gt;
&lt;li&gt;Existe predicate pushdown?&lt;/li&gt;
&lt;li&gt;O particionamento permite pruning?&lt;/li&gt;
&lt;li&gt;Como será tratada a evolução de schema?&lt;/li&gt;
&lt;li&gt;Existe necessidade de transações ou snapshots?&lt;/li&gt;
&lt;li&gt;Qual é o custo de armazenamento e leitura?&lt;/li&gt;
&lt;li&gt;Como a performance será medida?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Essas perguntas ajudam a evitar decisões baseadas apenas em preferência tecnológica.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;O Spark pode executar exatamente a mesma lógica sobre formatos diferentes.&lt;/p&gt;

&lt;p&gt;Mas o trabalho necessário para obter os dados pode variar bastante.&lt;/p&gt;

&lt;p&gt;Uma consulta analítica não depende apenas de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CPU
Memory
Executors
Partitions
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ela também depende de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;File Format
Compression
Metadata
Column Selection
Filter Pushdown
Physical Partitioning
File Size
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Por isso:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Performance em Spark começa antes da execução: começa na forma como os dados são armazenados, organizados e disponibilizados para leitura.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  36. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando recebo um pipeline Spark lento, não começo necessariamente aumentando recursos.&lt;/p&gt;

&lt;p&gt;Primeiro investigo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual formato está sendo lido?
            |
            v
Quantos arquivos existem?
            |
            v
Qual é o tamanho médio?
            |
            v
O schema está correto?
            |
            v
Todas as colunas são necessárias?
            |
            v
Os filtros estão sendo aproveitados?
            |
            v
Existe partition pruning?
            |
            v
O plano físico está adequado?
            |
            v
Existe shuffle desnecessário?
            |
            v
Qual é o gargalo real?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Às vezes, melhorar o layout dos dados traz mais benefício do que adicionar Executors.&lt;/p&gt;

&lt;p&gt;E essa é uma das lições mais importantes da Engenharia de Dados moderna.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora entendemos como CSV, JSON, Parquet e ORC influenciam o processamento.&lt;/p&gt;

&lt;p&gt;Mas existe uma evolução importante que precisamos explorar:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como transformar arquivos em tabelas analíticas confiáveis?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;No próximo capítulo, vamos entrar no universo das arquiteturas Lakehouse e dos formatos de tabela.&lt;/p&gt;

&lt;p&gt;Vamos estudar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Data Lake vs. Data Warehouse vs. Lakehouse.&lt;/li&gt;
&lt;li&gt;Apache Iceberg.&lt;/li&gt;
&lt;li&gt;Delta Lake.&lt;/li&gt;
&lt;li&gt;Apache Hudi.&lt;/li&gt;
&lt;li&gt;Formatos de arquivo vs. formatos de tabela.&lt;/li&gt;
&lt;li&gt;Snapshots.&lt;/li&gt;
&lt;li&gt;Evolução de schema.&lt;/li&gt;
&lt;li&gt;ACID.&lt;/li&gt;
&lt;li&gt;Time Travel.&lt;/li&gt;
&lt;li&gt;Compactação.&lt;/li&gt;
&lt;li&gt;Gerenciamento de metadados.&lt;/li&gt;
&lt;li&gt;Estratégias de atualização incremental.&lt;/li&gt;
&lt;li&gt;Integração com Spark SQL.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central será:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Por que armazenar arquivos Parquet não é suficiente para construir uma plataforma analítica transacional e governável?&lt;/strong&gt;&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 11:&lt;/strong&gt; Dimensionamento de recursos, paralelismo e custos&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 13 — Apache Iceberg, Delta Lake e Hudi: a evolução do Data Lake para Lakehouse&lt;/p&gt;

</description>
      <category>python</category>
      <category>apache</category>
      <category>spark</category>
      <category>bigdata</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 11: Dimensionamento de recursos, paralelismo e custos</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 19:53:11 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-11-dimensionamento-de-recursos-paralelismo-e-custos-5477</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-11-dimensionamento-de-recursos-paralelismo-e-custos-5477</guid>
      <description>&lt;p&gt;&lt;em&gt;Nota de atualização (2026): este artigo conecta os fundamentos da minha jornada de aprendizado do Apache Spark aos mecanismos modernos de dimensionamento e gerenciamento de recursos. Configurações, valores padrão e recursos como Dynamic Allocation e Adaptive Query Execution evoluíram ao longo das versões. Os exemplos são ilustrativos e devem ser validados na versão e plataforma utilizadas.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nos artigos anteriores, construímos uma visão da arquitetura do Apache Spark.&lt;/p&gt;

&lt;p&gt;Entendemos o papel do Driver, dos Executors, das partições, das Tasks, do shuffle e do gerenciamento de memória.&lt;/p&gt;

&lt;p&gt;Na Parte 10, vimos como aplicações Spark são submetidas e executadas em diferentes ambientes, incluindo Standalone, YARN e Kubernetes.&lt;/p&gt;

&lt;p&gt;Agora surge uma pergunta prática:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quantos Executors uma aplicação Spark realmente precisa?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;E outras igualmente importantes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Quantos cores devemos atribuir a cada Executor?&lt;/li&gt;
&lt;li&gt;Quanta memória é necessária?&lt;/li&gt;
&lt;li&gt;Quantas partições devemos utilizar?&lt;/li&gt;
&lt;li&gt;Quando aumentar recursos melhora a performance?&lt;/li&gt;
&lt;li&gt;Quando estamos apenas aumentando custos?&lt;/li&gt;
&lt;li&gt;Como funciona Dynamic Allocation?&lt;/li&gt;
&lt;li&gt;Como identificar um cluster subutilizado?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Essas perguntas fazem parte de uma disciplina maior: &lt;strong&gt;engenharia de performance e capacidade&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;O objetivo não é encontrar uma configuração mágica.&lt;/p&gt;

&lt;p&gt;É construir um processo de dimensionamento baseado em comportamento observado, métricas e custo.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. O problema de dimensionar recursos sem medir
&lt;/h2&gt;

&lt;p&gt;Imagine um pipeline Spark que processa 500 GB diariamente.&lt;/p&gt;

&lt;p&gt;A primeira configuração utiliza:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;4 Executors
4 cores por Executor
8 GB de heap por Executor
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O processamento demora 40 minutos.&lt;/p&gt;

&lt;p&gt;Alguém decide dobrar os recursos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 Executors
4 cores por Executor
8 GB de heap por Executor
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora o processamento demora 32 minutos.&lt;/p&gt;

&lt;p&gt;Houve uma melhora.&lt;/p&gt;

&lt;p&gt;Mas também aumentamos a quantidade de recursos alocados.&lt;/p&gt;

&lt;p&gt;A pergunta correta não é apenas:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;O Job ficou mais rápido?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Precisamos perguntar:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Quanto custou cada execução antes e depois?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;E:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;O ganho de desempenho justifica o consumo adicional?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;É aqui que performance e FinOps começam a se encontrar.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. O modelo básico de recursos do Spark
&lt;/h2&gt;

&lt;p&gt;Uma aplicação Spark utiliza diferentes tipos de recursos.&lt;/p&gt;

&lt;p&gt;Podemos representar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 SPARK APPLICATION
                         |
                         v
                       DRIVER
                         |
                         v
                  EXECUTOR POOL
                         |
             +-----------+-----------+
             |           |           |
             v           v           v
         Executor 1  Executor 2  Executor 3
             |           |           |
             v           v           v
          CPU/RAM     CPU/RAM     CPU/RAM
             |           |           |
             v           v           v
            Tasks       Tasks       Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Driver coordena a aplicação.&lt;/p&gt;

&lt;p&gt;Os Executors disponibilizam recursos para executar Tasks.&lt;/p&gt;

&lt;p&gt;O paralelismo efetivo depende tanto dos recursos quanto do trabalho disponível.&lt;/p&gt;

&lt;p&gt;Isso nos leva a uma distinção essencial:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Recursos disponíveis não são necessariamente recursos utilizados.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Executor Instances
&lt;/h2&gt;

&lt;p&gt;Uma configuração importante é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.executor.instances
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ela define a quantidade inicial ou desejada de Executors em cenários de alocação estática, conforme o Cluster Manager e as demais configurações.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nt"&gt;--conf&lt;/span&gt; spark.executor.instances&lt;span class="o"&gt;=&lt;/span&gt;4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos representar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor 1
Executor 2
Executor 3
Executor 4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas quatro Executors não significam automaticamente quatro Tasks simultâneas.&lt;/p&gt;

&lt;p&gt;Precisamos considerar a quantidade de cores atribuída a cada Executor e os recursos solicitados por Task.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Executor Cores
&lt;/h2&gt;

&lt;p&gt;Outra configuração:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.executor.cores
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nt"&gt;--executor-cores&lt;/span&gt; 4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em uma configuração comum, na qual cada Task utiliza um core, um Executor com quatro cores pode executar até quatro Tasks simultaneamente.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor
   |
   +-- Core 1 → Task A
   +-- Core 2 → Task B
   +-- Core 3 → Task C
   +-- Core 4 → Task D
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;4 Executors
×
4 cores
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;obtemos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;16 cores alocados aos Executors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em condições apropriadas, isso permite até aproximadamente 16 Tasks simultâneas.&lt;/p&gt;

&lt;p&gt;Entretanto, a concorrência real pode ser limitada por outros fatores, como recursos por Task, disponibilidade de partições, scheduling e características do ambiente.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. O que acontece quando temos poucas partições?
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;16 cores disponíveis
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;mas apenas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;4 partições
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;em determinada Stage.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Available slots: 16

Task 1  ████████
Task 2  ████████
Task 3  ████████
Task 4  ████████

Idle slots: 12
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Não importa que tenhamos 16 cores disponíveis.&lt;/p&gt;

&lt;p&gt;Se aquela Stage possui apenas quatro Tasks executáveis, não conseguiremos ocupar todos os slots com esse trabalho.&lt;/p&gt;

&lt;p&gt;Esse é um ponto central:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;O paralelismo de uma Stage é limitado, entre outros fatores, pela quantidade de Tasks disponíveis.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Portanto, aumentar Executors sem considerar partições pode produzir desperdício.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. E quando temos partições demais?
&lt;/h2&gt;

&lt;p&gt;Agora imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;16 cores disponíveis
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100.000 partições
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ter muitas partições não significa necessariamente melhor performance.&lt;/p&gt;

&lt;p&gt;Cada Task possui custos associados a:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Scheduling.&lt;/li&gt;
&lt;li&gt;Inicialização.&lt;/li&gt;
&lt;li&gt;Serialização.&lt;/li&gt;
&lt;li&gt;Comunicação.&lt;/li&gt;
&lt;li&gt;Gerenciamento.&lt;/li&gt;
&lt;li&gt;Leitura de dados.&lt;/li&gt;
&lt;li&gt;Processamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Se as Tasks forem muito pequenas, o overhead pode se tornar significativo.&lt;/p&gt;

&lt;p&gt;Podemos encontrar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task execution: 20 ms
Scheduling overhead: relevante
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nesse cenário, estamos criando um volume enorme de unidades de trabalho para processar quantidades pequenas de dados.&lt;/p&gt;

&lt;p&gt;O objetivo é encontrar uma granularidade apropriada.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Partições pequenas vs. partições grandes
&lt;/h2&gt;

&lt;p&gt;Considere um dataset de 100 GB.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cenário A
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10 partições
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Média aproximada:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10 GB por partição
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Cenário B
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1.000 partições
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Média aproximada:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100 MB por partição
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Cenário C
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100.000 partições
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Média aproximada:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 MB por partição
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esses valores representam apenas divisões matemáticas simplificadas.&lt;/p&gt;

&lt;p&gt;Na prática, o tamanho de uma partição depende da distribuição dos dados, do formato, da operação e da estratégia de particionamento.&lt;/p&gt;

&lt;p&gt;Mesmo assim, o exemplo demonstra o trade-off:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Poucas partições
       |
       v
Menos paralelismo potencial
       +
Tasks possivelmente grandes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Muitas partições
       |
       v
Mais paralelismo potencial
       +
Maior overhead de Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A configuração ideal depende do workload.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Paralelismo não é a mesma coisa que concorrência
&lt;/h2&gt;

&lt;p&gt;Esses conceitos são frequentemente confundidos.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Paralelismo disponível&lt;/strong&gt; representa a capacidade de executar trabalho simultaneamente.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Concorrência efetiva&lt;/strong&gt; representa quantas Tasks estão realmente executando naquele momento.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;32 slots disponíveis
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;mas uma Stage com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A concorrência máxima daquela Stage será limitada pelas oito Tasks.&lt;/p&gt;

&lt;p&gt;Agora imagine uma Stage com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1.000 Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os 32 slots podem executar ondas sucessivas de Tasks.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Wave 1  → 32 Tasks
Wave 2  → 32 Tasks
Wave 3  → 32 Tasks
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A duração total depende da quantidade de ondas, da duração das Tasks e de possíveis desequilíbrios.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. O conceito de Task Waves
&lt;/h2&gt;

&lt;p&gt;Uma forma útil de raciocinar é por ondas de execução.&lt;/p&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;128 Tasks
32 slots
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em um cenário idealizado, com Tasks de duração semelhante:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;128 / 32 = 4 ondas
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;128 Tasks
16 slots
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Teríamos aproximadamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 ondas
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso sugere que aumentar slots pode reduzir o tempo de execução.&lt;/p&gt;

&lt;p&gt;Mas somente enquanto existir trabalho paralelizável suficiente e os demais recursos não se tornarem gargalos.&lt;/p&gt;

&lt;p&gt;Se a aplicação estiver limitada por rede, armazenamento ou skew, aumentar cores pode produzir ganhos muito menores.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Amdahl e os limites do paralelismo
&lt;/h2&gt;

&lt;p&gt;Existe um princípio clássico da computação paralela: nem todo trabalho pode ser acelerado indefinidamente.&lt;/p&gt;

&lt;p&gt;Uma aplicação pode possuir partes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Parallelizable Work
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e partes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Serial / Coordination Work
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A Lei de Amdahl descreve o limite teórico de aceleração quando apenas uma fração do trabalho pode ser paralelizada:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
S(N)=\frac{1}{(1-P)+\frac{P}{N}}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Onde:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;(S(N)) é o speedup.&lt;/li&gt;
&lt;li&gt;(P) é a fração paralelizável.&lt;/li&gt;
&lt;li&gt;(N) é a quantidade de unidades de processamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por exemplo, se 90% do trabalho for paralelizável, o limite teórico de speedup, mesmo com recursos infinitos, será 10 vezes.&lt;/p&gt;

&lt;p&gt;Em Spark, o comportamento real também envolve rede, shuffle, armazenamento, overhead e distribuição de dados.&lt;/p&gt;

&lt;p&gt;Portanto, aumentar Executors não garante aceleração linear.&lt;/p&gt;


&lt;h2&gt;
  
  
  11. Executor Memory
&lt;/h2&gt;

&lt;p&gt;Outra configuração fundamental é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.executor.memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ela define a heap JVM do Executor.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nt"&gt;--executor-memory&lt;/span&gt; 8g
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas, como vimos na Parte 7:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Heap
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;não representa necessariamente toda a memória utilizada pelo processo ou container.&lt;/p&gt;

&lt;p&gt;Também precisamos considerar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Memory Overhead
Off-Heap
Python Workers
Native Memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A quantidade de memória apropriada depende das operações executadas.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Filter
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;pode apresentar necessidades diferentes de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Large Aggregation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Sort-Merge Join
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Não existe uma proporção universal entre memória e cores que funcione para todos os Jobs.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. A relação entre cores e memória
&lt;/h2&gt;

&lt;p&gt;Imagine um Executor com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 cores
8 GB de heap
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em um cenário comum de uma Task por core, várias Tasks podem disputar os recursos disponíveis.&lt;/p&gt;

&lt;p&gt;Agora compare:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;4 cores
8 GB de heap
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A quantidade de Tasks concorrentes potencialmente diminui.&lt;/p&gt;

&lt;p&gt;Isso pode reduzir a pressão de memória por Executor em determinados workloads.&lt;/p&gt;

&lt;p&gt;Mas também pode reduzir a capacidade de processamento paralelo.&lt;/p&gt;

&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Cores
       +
Executor Memory
       +
Task Memory Demand
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;precisam ser analisados conjuntamente.&lt;/p&gt;

&lt;p&gt;Não devemos ajustar esses parâmetros isoladamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. Executors maiores ou mais Executors?
&lt;/h2&gt;

&lt;p&gt;Considere duas configurações ilustrativas.&lt;/p&gt;

&lt;h3&gt;
  
  
  Configuração A
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;4 Executors
8 cores cada
16 GB heap cada
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Total:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;32 cores
64 GB heap
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Configuração B
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 Executors
4 cores cada
8 GB heap cada
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Total:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;32 cores
64 GB heap
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os recursos agregados são semelhantes.&lt;/p&gt;

&lt;p&gt;Mas o comportamento pode ser diferente.&lt;/p&gt;

&lt;p&gt;Entre os fatores envolvidos estão:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Concorrência dentro de cada Executor.&lt;/li&gt;
&lt;li&gt;Pressão de memória.&lt;/li&gt;
&lt;li&gt;Garbage Collection.&lt;/li&gt;
&lt;li&gt;Distribuição de Tasks.&lt;/li&gt;
&lt;li&gt;Custos de processos.&lt;/li&gt;
&lt;li&gt;Overhead por Executor.&lt;/li&gt;
&lt;li&gt;Falhas e recuperação.&lt;/li&gt;
&lt;li&gt;Restrições dos nós.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Nenhuma das configurações é automaticamente superior.&lt;/p&gt;

&lt;p&gt;O resultado precisa ser medido.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. &lt;code&gt;spark.default.parallelism&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Uma configuração conhecida é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.default.parallelism
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ela influencia o paralelismo padrão de determinadas operações RDD quando o número de partições não é informado explicitamente.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallelize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1_000_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O particionamento depende do contexto e das configurações aplicáveis.&lt;/p&gt;

&lt;p&gt;Podemos especificar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallelize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1_000_000&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="mi"&gt;100&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora solicitamos 100 partições para esse RDD.&lt;/p&gt;

&lt;p&gt;Mas é importante não confundir:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.default.parallelism
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.sql.shuffle.partitions
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;São configurações utilizadas em contextos diferentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. &lt;code&gt;spark.sql.shuffle.partitions&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;No Spark SQL, uma configuração relevante é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.sql.shuffle.partitions
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ela define o número inicial de partições utilizado em determinadas operações de shuffle, como joins e agregações.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nt"&gt;--conf&lt;/span&gt; spark.sql.shuffle.partitions&lt;span class="o"&gt;=&lt;/span&gt;200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse valor não significa que todos os DataFrames terão 200 partições.&lt;/p&gt;

&lt;p&gt;Ele se aplica a situações específicas do planejamento de shuffle.&lt;/p&gt;

&lt;p&gt;Além disso, mecanismos modernos como Adaptive Query Execution podem modificar o particionamento efetivo durante a execução.&lt;/p&gt;

&lt;p&gt;Portanto, o valor configurado não deve ser interpretado como uma garantia de quantidade final de Tasks em todas as Stages.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. O impacto do Shuffle no dimensionamento
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;vendas&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos ter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Input Partitions
       |
       v
Partial Aggregation
       |
       v
Shuffle
       |
       v
Reduce Partitions
       |
       v
Final Aggregation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se as partições de shuffle forem grandes demais, podemos observar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Spill.&lt;/li&gt;
&lt;li&gt;Pressão de memória.&lt;/li&gt;
&lt;li&gt;Tasks demoradas.&lt;/li&gt;
&lt;li&gt;GC.&lt;/li&gt;
&lt;li&gt;OOM.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Se forem pequenas demais, podemos observar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Muitas Tasks.&lt;/li&gt;
&lt;li&gt;Overhead.&lt;/li&gt;
&lt;li&gt;Fragmentação do trabalho.&lt;/li&gt;
&lt;li&gt;Baixa eficiência.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Dimensionar recursos exige compreender o plano de execução.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Adaptive Query Execution
&lt;/h2&gt;

&lt;p&gt;O Adaptive Query Execution, conhecido como &lt;strong&gt;AQE&lt;/strong&gt;, permite que Spark SQL adapte determinados aspectos do plano físico utilizando estatísticas obtidas durante a execução.&lt;/p&gt;

&lt;p&gt;Entre os recursos possíveis estão:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Coalescing de partições após shuffle.&lt;/li&gt;
&lt;li&gt;Tratamento de determinados casos de skew.&lt;/li&gt;
&lt;li&gt;Alteração de estratégias de join quando aplicável.&lt;/li&gt;
&lt;li&gt;Ajustes baseados em estatísticas de execução.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Initial Physical Plan
         |
         v
Execution Statistics
         |
         v
Adaptive Optimization
         |
         v
Updated Physical Plan
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso reduz a necessidade de acertar manualmente todos os detalhes antecipadamente.&lt;/p&gt;

&lt;p&gt;Mas AQE não elimina a necessidade de engenharia de performance.&lt;/p&gt;

&lt;p&gt;Ele não corrige automaticamente qualquer problema de distribuição ou dimensionamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. Dynamic Allocation
&lt;/h2&gt;

&lt;p&gt;Agora chegamos a um recurso importante para eficiência operacional.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dynamic Allocation&lt;/strong&gt; permite que Spark ajuste a quantidade de Executors durante a execução, conforme a demanda e as condições do ambiente.&lt;/p&gt;

&lt;p&gt;Sem Dynamic Allocation, podemos imaginar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executors = 10
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;durante toda a aplicação.&lt;/p&gt;

&lt;p&gt;Mesmo quando existe pouco trabalho.&lt;/p&gt;

&lt;p&gt;Com Dynamic Allocation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Low Demand
    |
    v
Fewer Executors

High Demand
    |
    v
More Executors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O objetivo é aproximar a quantidade de recursos alocados da necessidade real.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. Como Dynamic Allocation funciona conceitualmente?
&lt;/h2&gt;

&lt;p&gt;Imagine um pipeline com três fases.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fase 1 — Leitura e preparação
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Moderate Parallelism
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Fase 2 — Shuffle e agregação
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;High Parallelism
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Fase 3 — Escrita final
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Lower Parallelism
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos representar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executors
   ^
12 |          ███████████
10 |          ███████████
 8 |          ███████████
 6 |    █████████████████
 4 |    █████████████████████
 2 |█████████████████████████
   +----------------------------&amp;gt; Tempo
       Fase 1  Fase 2  Fase 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse é apenas um desenho conceitual.&lt;/p&gt;

&lt;p&gt;Na prática, o comportamento depende de políticas de alocação, backlog de Tasks, timeouts, limites e suporte da infraestrutura.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Configurando Dynamic Allocation
&lt;/h2&gt;

&lt;p&gt;Um exemplo ilustrativo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;spark-submit &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--master&lt;/span&gt; yarn &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--deploy-mode&lt;/span&gt; cluster &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--conf&lt;/span&gt; spark.dynamicAllocation.enabled&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--conf&lt;/span&gt; spark.dynamicAllocation.minExecutors&lt;span class="o"&gt;=&lt;/span&gt;2 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--conf&lt;/span&gt; spark.dynamicAllocation.initialExecutors&lt;span class="o"&gt;=&lt;/span&gt;4 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--conf&lt;/span&gt; spark.dynamicAllocation.maxExecutors&lt;span class="o"&gt;=&lt;/span&gt;12 &lt;span class="se"&gt;\&lt;/span&gt;
  app.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nesse exemplo, estabelecemos limites de alocação.&lt;/p&gt;

&lt;p&gt;Mas habilitar Dynamic Allocation não significa que qualquer cluster esteja automaticamente preparado.&lt;/p&gt;

&lt;p&gt;O ambiente precisa suportar os mecanismos necessários para preservar ou recuperar dados de shuffle quando Executors são removidos, conforme a configuração e a versão utilizadas.&lt;/p&gt;

&lt;p&gt;As estratégias disponíveis podem envolver shuffle tracking, external shuffle service ou outros mecanismos suportados.&lt;/p&gt;

&lt;p&gt;É importante validar a documentação da versão e da plataforma.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. Dynamic Allocation não significa custo mínimo automático
&lt;/h2&gt;

&lt;p&gt;Imagine que a aplicação solicita mais Executors.&lt;/p&gt;

&lt;p&gt;O Cluster Manager pode ou não conseguir disponibilizá-los imediatamente.&lt;/p&gt;

&lt;p&gt;Além disso, a infraestrutura pode cobrar por recursos provisionados, não apenas por Executors efetivamente ocupados.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark Executors
       |
       v
Kubernetes Pods
       |
       v
Cluster Nodes
       |
       v
Cloud Billing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reduzir Executors não significa necessariamente reduzir instantaneamente a quantidade de máquinas cobradas.&lt;/p&gt;

&lt;p&gt;Isso depende da arquitetura de autoscaling e do modelo de cobrança.&lt;/p&gt;

&lt;p&gt;Portanto:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Dynamic Allocation e autoscaling da infraestrutura são mecanismos relacionados, mas diferentes.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  22. Dynamic Allocation vs. Cluster Autoscaling
&lt;/h2&gt;

&lt;p&gt;Essa distinção merece atenção.&lt;/p&gt;

&lt;h3&gt;
  
  
  Dynamic Allocation
&lt;/h3&gt;

&lt;p&gt;Ajusta a quantidade de Executors solicitados pela aplicação Spark.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cluster Autoscaling
&lt;/h3&gt;

&lt;p&gt;Ajusta a capacidade computacional disponível no cluster, quando suportado e configurado.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark Workload
      |
      v
Dynamic Allocation
      |
      v
Executor Demand
      |
      v
Cluster Autoscaler
      |
      v
Compute Nodes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os dois mecanismos podem trabalhar juntos.&lt;/p&gt;

&lt;p&gt;Mas também podem operar de maneira independente.&lt;/p&gt;

&lt;p&gt;Uma aplicação pode solicitar Executors adicionais e permanecer aguardando porque não existem recursos disponíveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. O papel do Data Skew
&lt;/h2&gt;

&lt;p&gt;Agora imagine que temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100 Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Distribuídas assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;99 Tasks → 30 segundos
1 Task   → 20 minutos
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A Stage continuará aguardando a Task mais lenta.&lt;/p&gt;

&lt;p&gt;Adicionar dezenas de Executors pode não resolver.&lt;/p&gt;

&lt;p&gt;Porque o problema não é falta de slots.&lt;/p&gt;

&lt;p&gt;É uma distribuição desigual do trabalho.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task 1  ███
Task 2  ███
Task 3  ███
...
Task 99 ███

Task 100 █████████████████████████████
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso é um exemplo clássico de &lt;strong&gt;straggler&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Data skew pode transformar um cluster grande em uma aplicação lenta.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Quando aumentar Executors ajuda?
&lt;/h2&gt;

&lt;p&gt;Aumentar Executors pode ajudar quando:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Existem muitas Tasks aguardando recursos.&lt;/li&gt;
&lt;li&gt;O processamento é suficientemente paralelizável.&lt;/li&gt;
&lt;li&gt;O cluster possui capacidade disponível.&lt;/li&gt;
&lt;li&gt;A aplicação não está dominada por gargalos externos.&lt;/li&gt;
&lt;li&gt;As Tasks estão razoavelmente equilibradas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas pode trazer pouco benefício quando:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Existem poucas partições.&lt;/li&gt;
&lt;li&gt;O workload é dominado por uma única Task.&lt;/li&gt;
&lt;li&gt;O armazenamento é o gargalo.&lt;/li&gt;
&lt;li&gt;O shuffle está saturando a rede.&lt;/li&gt;
&lt;li&gt;Existe uma etapa serial relevante.&lt;/li&gt;
&lt;li&gt;O Driver está sobrecarregado.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por isso, a decisão precisa ser baseada em métricas.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. CPU-bound, Memory-bound e I/O-bound
&lt;/h2&gt;

&lt;p&gt;Uma classificação útil para diagnóstico é identificar o principal recurso limitante.&lt;/p&gt;

&lt;h3&gt;
  
  
  CPU-bound
&lt;/h3&gt;

&lt;p&gt;A aplicação passa grande parte do tempo realizando computação.&lt;/p&gt;

&lt;p&gt;Exemplos possíveis:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Complex Transformations
Heavy Computation
Expensive Expressions
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Memory-bound
&lt;/h3&gt;

&lt;p&gt;A aplicação sofre pressão relacionada à memória.&lt;/p&gt;

&lt;p&gt;Sinais possíveis:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;High GC
Spill
OOM
Large Aggregations
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  I/O-bound
&lt;/h3&gt;

&lt;p&gt;A aplicação espera por leitura, escrita ou movimentação de dados.&lt;/p&gt;

&lt;p&gt;Exemplos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Slow Storage
Heavy Shuffle
Network Bottlenecks
Remote Reads
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Na prática, um Job pode apresentar diferentes gargalos em Stages diferentes.&lt;/p&gt;

&lt;p&gt;Por isso, classificar a aplicação inteira com um único rótulo pode ser insuficiente.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. O que medir na Spark UI?
&lt;/h2&gt;

&lt;p&gt;Quando estou dimensionando recursos, quero observar:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Jobs e Stages&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Duração.&lt;/li&gt;
&lt;li&gt;Quantidade de Tasks.&lt;/li&gt;
&lt;li&gt;Stages críticas.&lt;/li&gt;
&lt;li&gt;Tempo de execução.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Tasks&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Distribuição de duração.&lt;/li&gt;
&lt;li&gt;Shuffle Read.&lt;/li&gt;
&lt;li&gt;Shuffle Write.&lt;/li&gt;
&lt;li&gt;Spill.&lt;/li&gt;
&lt;li&gt;GC Time.&lt;/li&gt;
&lt;li&gt;Input Size.&lt;/li&gt;
&lt;li&gt;Output Size.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Executors&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tasks executadas.&lt;/li&gt;
&lt;li&gt;Falhas.&lt;/li&gt;
&lt;li&gt;Memória.&lt;/li&gt;
&lt;li&gt;GC.&lt;/li&gt;
&lt;li&gt;Dados persistidos.&lt;/li&gt;
&lt;li&gt;Comportamento agregado.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;SQL&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Plano físico.&lt;/li&gt;
&lt;li&gt;Exchanges.&lt;/li&gt;
&lt;li&gt;Joins.&lt;/li&gt;
&lt;li&gt;Agregações.&lt;/li&gt;
&lt;li&gt;Métricas dos operadores.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Onde o tempo está sendo gasto?&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  27. Um exemplo de diagnóstico
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Job Duration: 30 minutos
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Observamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Stage 1: 2 minutos
Stage 2: 3 minutos
Stage 3: 24 minutos
Stage 4: 1 minuto
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A Stage 3 domina o tempo total.&lt;/p&gt;

&lt;p&gt;Agora investigamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Stage 3
    |
    v
Shuffle-heavy Join
    |
    v
Large Spill
    |
    v
Uneven Task Duration
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Talvez o problema seja:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Data Skew
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nesse caso, simplesmente dobrar o cluster pode não ser a melhor intervenção.&lt;/p&gt;

&lt;p&gt;Precisamos avaliar distribuição de chaves, estratégia de join e particionamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Benchmarking de configurações
&lt;/h2&gt;

&lt;p&gt;Em vez de escolher recursos por intuição, podemos realizar experimentos controlados.&lt;/p&gt;

&lt;p&gt;Imagine três configurações.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Configuração&lt;/th&gt;
&lt;th&gt;Executors&lt;/th&gt;
&lt;th&gt;Cores/Executor&lt;/th&gt;
&lt;th&gt;Heap/Executor&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Executamos o mesmo workload com dados comparáveis.&lt;/p&gt;

&lt;p&gt;Registramos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tempo total.&lt;/li&gt;
&lt;li&gt;Duração por Stage.&lt;/li&gt;
&lt;li&gt;CPU.&lt;/li&gt;
&lt;li&gt;GC.&lt;/li&gt;
&lt;li&gt;Spill.&lt;/li&gt;
&lt;li&gt;Shuffle.&lt;/li&gt;
&lt;li&gt;Recursos alocados.&lt;/li&gt;
&lt;li&gt;Custo estimado.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O objetivo é descobrir qual configuração entrega o melhor equilíbrio para o requisito operacional.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. Exemplo de análise de custo
&lt;/h2&gt;

&lt;p&gt;Considere um exemplo hipotético.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Configuração&lt;/th&gt;
&lt;th&gt;Tempo&lt;/th&gt;
&lt;th&gt;Custo/hora do conjunto de recursos&lt;/th&gt;
&lt;th&gt;Custo por execução&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;40 min&lt;/td&gt;
&lt;td&gt;R$ 12&lt;/td&gt;
&lt;td&gt;R$ 8,00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B&lt;/td&gt;
&lt;td&gt;30 min&lt;/td&gt;
&lt;td&gt;R$ 24&lt;/td&gt;
&lt;td&gt;R$ 12,00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C&lt;/td&gt;
&lt;td&gt;25 min&lt;/td&gt;
&lt;td&gt;R$ 36&lt;/td&gt;
&lt;td&gt;R$ 15,00&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Os valores são ilustrativos.&lt;/p&gt;

&lt;p&gt;A configuração C é mais rápida.&lt;/p&gt;

&lt;p&gt;Mas a configuração A possui menor custo direto por execução.&lt;/p&gt;

&lt;p&gt;A decisão depende do SLA.&lt;/p&gt;

&lt;p&gt;Se o pipeline precisa terminar em até 60 minutos, A pode ser suficiente.&lt;/p&gt;

&lt;p&gt;Se precisa terminar em até 30 minutos, B ou C podem ser consideradas.&lt;/p&gt;

&lt;p&gt;A escolha correta depende do objetivo de negócio.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. Custo por execução vs. tempo de execução
&lt;/h2&gt;

&lt;p&gt;Uma métrica útil é:&lt;/p&gt;

&lt;p&gt;[&lt;/p&gt;

&lt;h1&gt;
  
  
  \text{Custo por execução}
&lt;/h1&gt;

&lt;p&gt;\text{Custo por hora}&lt;br&gt;
\times&lt;br&gt;
\text{Tempo em horas}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Custo/hora = R$ 24
Tempo = 30 minutos
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Então:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
24 \times 0{,}5 = R\$ 12&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Esse cálculo simplificado não inclui necessariamente armazenamento, transferência, serviços auxiliares ou regras específicas de cobrança.&lt;/p&gt;

&lt;p&gt;Mas ajuda a comparar configurações.&lt;/p&gt;

&lt;p&gt;Em produção, devemos considerar o custo total relevante.&lt;/p&gt;


&lt;h2&gt;
  
  
  31. Custo por volume processado
&lt;/h2&gt;

&lt;p&gt;Outra métrica interessante:&lt;/p&gt;

&lt;p&gt;[&lt;/p&gt;
&lt;h1&gt;
  
  
  \text{Custo por TB}
&lt;/h1&gt;

&lt;p&gt;\frac{\text{Custo da execução}}{\text{TB processados}}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Dados processados = 2 TB
Custo = R$ 20
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Então:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
\text{Custo por TB} = R\$ 10&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Essa métrica ajuda a comparar eficiência ao longo do tempo.&lt;/p&gt;

&lt;p&gt;Mas precisamos manter comparáveis:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tipo de processamento.&lt;/li&gt;
&lt;li&gt;Complexidade.&lt;/li&gt;
&lt;li&gt;Formato dos dados.&lt;/li&gt;
&lt;li&gt;Volume.&lt;/li&gt;
&lt;li&gt;SLA.&lt;/li&gt;
&lt;li&gt;Infraestrutura.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Processar 1 TB com um filtro simples não equivale a processar 1 TB com múltiplos joins e agregações complexas.&lt;/p&gt;


&lt;h2&gt;
  
  
  32. O problema da superprovisão
&lt;/h2&gt;

&lt;p&gt;Superprovisão acontece quando alocamos recursos significativamente acima da necessidade real.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;64 cores disponíveis
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;mas a aplicação utiliza efetivamente apenas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 cores
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;durante grande parte do tempo.&lt;/p&gt;

&lt;p&gt;Podemos estar pagando por capacidade ociosa.&lt;/p&gt;

&lt;p&gt;Entretanto, baixa utilização não deve ser analisada isoladamente.&lt;/p&gt;

&lt;p&gt;Pode existir uma exigência de baixa latência ou necessidade de absorver picos.&lt;/p&gt;

&lt;p&gt;Por isso, precisamos relacionar utilização com SLA e características do workload.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. O problema do subdimensionamento
&lt;/h2&gt;

&lt;p&gt;O extremo oposto também é ruim.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Dataset: 2 TB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2 Executors
2 cores cada
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Talvez o Job funcione.&lt;/p&gt;

&lt;p&gt;Mas pode apresentar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Muitas ondas de Tasks.&lt;/li&gt;
&lt;li&gt;Tempo excessivo.&lt;/li&gt;
&lt;li&gt;Pressão de memória.&lt;/li&gt;
&lt;li&gt;Spill.&lt;/li&gt;
&lt;li&gt;Baixa capacidade de absorver variações.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Se o SLA exigir processamento rápido, os recursos podem ser insuficientes.&lt;/p&gt;

&lt;p&gt;Portanto, o objetivo não é utilizar o menor cluster possível em qualquer circunstância.&lt;/p&gt;

&lt;p&gt;É utilizar recursos suficientes para atender aos requisitos com eficiência.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Dimensionamento também depende do SLA
&lt;/h2&gt;

&lt;p&gt;Imagine dois pipelines.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pipeline A
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Processamento diário
SLA: 4 horas
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Pipeline B
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Processamento a cada 15 minutos
SLA: 5 minutos
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mesmo que processem volumes semelhantes, os requisitos operacionais são diferentes.&lt;/p&gt;

&lt;p&gt;O Pipeline B pode precisar de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Mais paralelismo.&lt;/li&gt;
&lt;li&gt;Inicialização mais rápida.&lt;/li&gt;
&lt;li&gt;Menor latência.&lt;/li&gt;
&lt;li&gt;Capacidade disponível.&lt;/li&gt;
&lt;li&gt;Estratégias diferentes de provisionamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Resource Sizing
      |
      v
Business SLA
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;é uma relação fundamental.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. Um processo de tuning orientado por evidências
&lt;/h2&gt;

&lt;p&gt;Minha abordagem preferida é iterativa.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. Establish Baseline
         |
         v
2. Measure
         |
         v
3. Identify Bottleneck
         |
         v
4. Form Hypothesis
         |
         v
5. Change One Variable
         |
         v
6. Run Benchmark
         |
         v
7. Compare Results
         |
         v
8. Keep or Revert
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse processo evita alterar vários parâmetros ao mesmo tempo e depois não saber qual mudança produziu o resultado.&lt;/p&gt;

&lt;p&gt;Também facilita documentar decisões técnicas.&lt;/p&gt;




&lt;h2&gt;
  
  
  36. Exemplo de tuning controlado
&lt;/h2&gt;

&lt;p&gt;Imagine que identificamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Shuffle partitions muito grandes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hipótese:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Aumentar o número inicial de partições de shuffle pode reduzir o tamanho do trabalho por Task.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Testamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.sql.shuffle.partitions=200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.sql.shuffle.partitions=400
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mantendo os demais fatores comparáveis.&lt;/p&gt;

&lt;p&gt;Observamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task Duration
Spill
GC
Total Stage Duration
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se a Stage melhorar sem criar overhead excessivo, a mudança pode ser válida.&lt;/p&gt;

&lt;p&gt;Se piorar, revertimos.&lt;/p&gt;

&lt;p&gt;A ideia não é que 400 seja melhor que 200.&lt;/p&gt;

&lt;p&gt;A ideia é testar uma hipótese com métricas.&lt;/p&gt;




&lt;h2&gt;
  
  
  37. Evite copiar configurações de outros clusters
&lt;/h2&gt;

&lt;p&gt;Uma configuração pode funcionar muito bem em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cluster A
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e mal em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cluster B
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mesmo com o mesmo código.&lt;/p&gt;

&lt;p&gt;As diferenças podem envolver:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU.&lt;/li&gt;
&lt;li&gt;Memória.&lt;/li&gt;
&lt;li&gt;Rede.&lt;/li&gt;
&lt;li&gt;Disco.&lt;/li&gt;
&lt;li&gt;Armazenamento remoto.&lt;/li&gt;
&lt;li&gt;Versão do Spark.&lt;/li&gt;
&lt;li&gt;JVM.&lt;/li&gt;
&lt;li&gt;Cluster Manager.&lt;/li&gt;
&lt;li&gt;Dados.&lt;/li&gt;
&lt;li&gt;Concorrência com outras aplicações.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por isso, parâmetros de tuning não devem ser tratados como receitas universais.&lt;/p&gt;

&lt;p&gt;O que funciona em uma plataforma gerenciada pode não funcionar da mesma maneira em um cluster local ou em Kubernetes.&lt;/p&gt;




&lt;h2&gt;
  
  
  38. Um exemplo de configuração inicial
&lt;/h2&gt;

&lt;p&gt;Considere um ambiente YARN compatível com alocação estática.&lt;/p&gt;

&lt;p&gt;Um exemplo de submissão seria:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;spark-submit &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--master&lt;/span&gt; yarn &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--deploy-mode&lt;/span&gt; cluster &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--driver-memory&lt;/span&gt; 4g &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--executor-memory&lt;/span&gt; 8g &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--executor-cores&lt;/span&gt; 4 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--num-executors&lt;/span&gt; 4 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--conf&lt;/span&gt; spark.dynamicAllocation.enabled&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--conf&lt;/span&gt; spark.sql.shuffle.partitions&lt;span class="o"&gt;=&lt;/span&gt;200 &lt;span class="se"&gt;\&lt;/span&gt;
  app.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa configuração representa apenas um ponto de partida ilustrativo.&lt;/p&gt;

&lt;p&gt;Não significa que seja adequada para um dataset específico.&lt;/p&gt;

&lt;p&gt;Em particular, precisamos validar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Memória total incluindo overhead.&lt;/li&gt;
&lt;li&gt;Capacidade do cluster.&lt;/li&gt;
&lt;li&gt;Quantidade de partições.&lt;/li&gt;
&lt;li&gt;Carga concorrente.&lt;/li&gt;
&lt;li&gt;Comportamento de shuffle.&lt;/li&gt;
&lt;li&gt;SLA.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  39. O que eu não faria em produção
&lt;/h2&gt;

&lt;p&gt;Eu evitaria começar com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Maximum Executors
Maximum Memory
Maximum Cores
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;sem conhecer o workload.&lt;/p&gt;

&lt;p&gt;Também evitaria:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Minimum Resources
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;sem verificar se o SLA será atendido.&lt;/p&gt;

&lt;p&gt;E evitaria alterar simultaneamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Count
Executor Cores
Executor Memory
Shuffle Partitions
Dynamic Allocation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;porque isso dificulta identificar a causa dos ganhos ou regressões.&lt;/p&gt;

&lt;p&gt;Ajustes devem ser acompanhados de observabilidade e experimentação.&lt;/p&gt;




&lt;h2&gt;
  
  
  40. O papel do FinOps em Engenharia de Dados
&lt;/h2&gt;

&lt;p&gt;FinOps não é apenas reduzir a conta da cloud.&lt;/p&gt;

&lt;p&gt;É relacionar consumo tecnológico com valor entregue.&lt;/p&gt;

&lt;p&gt;Em Spark, isso significa entender:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Data Volume
     |
     v
Compute Resources
     |
     v
Execution Time
     |
     v
Infrastructure Cost
     |
     v
Business Outcome
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Um pipeline que termina cinco minutos mais rápido pode justificar maior custo se isso evitar uma penalidade de SLA.&lt;/p&gt;

&lt;p&gt;Outro pipeline pode economizar muito dinheiro simplesmente reduzindo recursos ociosos.&lt;/p&gt;

&lt;p&gt;A decisão precisa considerar contexto.&lt;/p&gt;




&lt;h2&gt;
  
  
  41. Métricas que eu acompanharia
&lt;/h2&gt;

&lt;p&gt;Para uma aplicação Spark em produção, considero útil acompanhar:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Métrica&lt;/th&gt;
&lt;th&gt;O que ajuda a avaliar&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Duração total&lt;/td&gt;
&lt;td&gt;Desempenho geral&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Duração por Stage&lt;/td&gt;
&lt;td&gt;Gargalos específicos&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quantidade de Tasks&lt;/td&gt;
&lt;td&gt;Granularidade e paralelismo&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Task Duration&lt;/td&gt;
&lt;td&gt;Distribuição de trabalho&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Shuffle Read/Write&lt;/td&gt;
&lt;td&gt;Movimentação de dados&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spill&lt;/td&gt;
&lt;td&gt;Pressão sobre memória e disco&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GC Time&lt;/td&gt;
&lt;td&gt;Pressão de memória JVM&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Executor CPU&lt;/td&gt;
&lt;td&gt;Utilização de processamento&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Executor Memory&lt;/td&gt;
&lt;td&gt;Consumo de recursos&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failed Tasks&lt;/td&gt;
&lt;td&gt;Estabilidade&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Custo por execução&lt;/td&gt;
&lt;td&gt;Eficiência financeira&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SLA Compliance&lt;/td&gt;
&lt;td&gt;Resultado operacional&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;O valor real aparece quando relacionamos essas métricas.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;High Cost
    +
Low CPU Utilization
    +
Long Runtime
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;pode indicar um gargalo que não será resolvido simplesmente aumentando CPU.&lt;/p&gt;




&lt;h2&gt;
  
  
  42. O modelo mental completo
&lt;/h2&gt;

&lt;p&gt;Depois de estudar arquitetura, memória, shuffle e implantação, podemos construir uma visão integrada:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                    WORKLOAD
                       |
                       v
                   DATA VOLUME
                       |
                       v
                   PARTITIONS
                       |
                       v
                     TASKS
                       |
                       v
                 PARALLELISM
                       |
                       v
               EXECUTOR RESOURCES
                       |
          +------------+------------+
          |            |            |
          v            v            v
         CPU         MEMORY        NETWORK
          |            |            |
          +------------+------------+
                       |
                       v
                 EXECUTION TIME
                       |
                       v
                RESOURCE COST
                       |
                       v
                  BUSINESS SLA
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse modelo ajuda a compreender por que dimensionamento não pode ser reduzido a uma única configuração.&lt;/p&gt;




&lt;h2&gt;
  
  
  43. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;Se eu pudesse resumir dimensionamento de recursos no Spark em uma frase:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;O melhor cluster não é necessariamente o maior ou o menor, mas aquele que atende ao SLA com o menor custo total aceitável e estabilidade adequada.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Para chegar lá, precisamos compreender:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;O workload.&lt;/li&gt;
&lt;li&gt;A distribuição dos dados.&lt;/li&gt;
&lt;li&gt;O paralelismo.&lt;/li&gt;
&lt;li&gt;O comportamento das Tasks.&lt;/li&gt;
&lt;li&gt;O consumo de memória.&lt;/li&gt;
&lt;li&gt;O shuffle.&lt;/li&gt;
&lt;li&gt;Os gargalos.&lt;/li&gt;
&lt;li&gt;A infraestrutura.&lt;/li&gt;
&lt;li&gt;O custo.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Dimensionamento é um processo contínuo de engenharia.&lt;/p&gt;




&lt;h2&gt;
  
  
  44. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando alguém pergunta:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Quantos Executors devo utilizar?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Minha resposta começa com outras perguntas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual volume de dados?
        |
        v
Quais transformações?
        |
        v
Quantas partições?
        |
        v
Qual é o plano físico?
        |
        v
Quanto shuffle?
        |
        v
Qual é o consumo de memória?
        |
        v
Qual é o SLA?
        |
        v
Qual é o custo atual?
        |
        v
Qual gargalo foi observado?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Só depois considero valores específicos de configuração.&lt;/p&gt;

&lt;p&gt;Porque o objetivo não é decorar parâmetros.&lt;/p&gt;

&lt;p&gt;É entender como o Spark transforma recursos em processamento.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Até aqui, estudamos como o Spark executa aplicações e utiliza recursos.&lt;/p&gt;

&lt;p&gt;Agora vamos avançar para um tema que influencia diretamente a performance e a arquitetura de Data Lakes:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;formatos de armazenamento e leitura de dados.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;No próximo capítulo, vamos explorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CSV.&lt;/li&gt;
&lt;li&gt;JSON.&lt;/li&gt;
&lt;li&gt;Parquet.&lt;/li&gt;
&lt;li&gt;ORC.&lt;/li&gt;
&lt;li&gt;Formatos row-based e columnar.&lt;/li&gt;
&lt;li&gt;Compressão.&lt;/li&gt;
&lt;li&gt;Schema.&lt;/li&gt;
&lt;li&gt;Predicate Pushdown.&lt;/li&gt;
&lt;li&gt;Column Pruning.&lt;/li&gt;
&lt;li&gt;Partition Pruning.&lt;/li&gt;
&lt;li&gt;Small Files Problem.&lt;/li&gt;
&lt;li&gt;Estratégias de leitura e escrita.&lt;/li&gt;
&lt;li&gt;Impacto no Spark SQL.&lt;/li&gt;
&lt;li&gt;Relação entre formatos, performance e custo.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central será:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Por que o mesmo dataset pode ter comportamentos tão diferentes no Spark dependendo de como está armazenado?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 10:&lt;/strong&gt; Client Mode vs. Cluster Mode — onde o Spark realmente executa&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 12 — CSV, JSON, Parquet e ORC: como o formato dos dados afeta a performance no Spark&lt;/p&gt;

</description>
      <category>bigdata</category>
      <category>apache</category>
      <category>spark</category>
      <category>python</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 10: Client Mode vs. Cluster Mode — onde o Spark realmente executa</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 19:50:43 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-10-client-mode-vs-cluster-mode-onde-o-spark-realmente-executa-5b1e</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-10-client-mode-vs-cluster-mode-onde-o-spark-realmente-executa-5b1e</guid>
      <description>&lt;p&gt;&lt;em&gt;Nota de atualização (2026): este artigo revisita conceitos da minha jornada de aprendizado de Apache Spark e os conecta aos ambientes modernos de implantação. Algumas opções de execução, integrações e recomendações surgiram ou evoluíram depois dos primeiros anos do projeto.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nos nove artigos anteriores, exploramos a arquitetura do Apache Spark por diferentes perspectivas.&lt;/p&gt;

&lt;p&gt;Aprendemos que uma aplicação Spark possui um &lt;strong&gt;Driver&lt;/strong&gt;, que coordena o processamento, e &lt;strong&gt;Executors&lt;/strong&gt;, responsáveis por executar Tasks sobre partições de dados.&lt;/p&gt;

&lt;p&gt;Também vimos que o Spark organiza o trabalho em Jobs, Stages e Tasks, utiliza shuffle para redistribuir dados, gerencia memória, mantém lineage para recuperação de falhas e oferece mecanismos como Broadcast Variables e Accumulators.&lt;/p&gt;

&lt;p&gt;Mas existe uma pergunta que ainda não respondemos completamente:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Onde o Driver realmente executa quando enviamos uma aplicação Spark para um cluster?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;E uma segunda pergunta igualmente importante:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quem inicia, gerencia e encerra os Executors?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Essas questões nos levam a três conceitos fundamentais:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Deployment Mode.&lt;/li&gt;
&lt;li&gt;Cluster Manager.&lt;/li&gt;
&lt;li&gt;Application Submission.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Vamos entender como eles se relacionam e por que essa distinção é essencial para desenvolver, implantar e operar aplicações Spark em produção.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. O que acontece quando executamos uma aplicação Spark?
&lt;/h2&gt;

&lt;p&gt;Considere uma aplicação PySpark:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;VendasAnalytics&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;vendas&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/vendas&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;vendas&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ano&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;2026&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O código parece simples.&lt;/p&gt;

&lt;p&gt;Mas, dependendo de onde executamos a aplicação, existe uma infraestrutura por trás:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  APPLICATION
                       |
                       v
                     DRIVER
                       |
                       v
                 CLUSTER MANAGER
                       |
             +---------+---------+
             |         |         |
             v         v         v
         EXECUTOR 1 EXECUTOR 2 EXECUTOR 3
             |         |         |
           TASKS     TASKS     TASKS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Driver coordena o processamento.&lt;/p&gt;

&lt;p&gt;O Cluster Manager participa da alocação e do gerenciamento dos recursos necessários.&lt;/p&gt;

&lt;p&gt;Os Executors realizam o trabalho distribuído.&lt;/p&gt;

&lt;p&gt;A localização e o ciclo de vida desses componentes dependem do ambiente de implantação.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. Driver, Executor e Cluster Manager não são a mesma coisa
&lt;/h2&gt;

&lt;p&gt;Antes de falar sobre Client Mode e Cluster Mode, precisamos separar três responsabilidades.&lt;/p&gt;

&lt;h3&gt;
  
  
  Driver
&lt;/h3&gt;

&lt;p&gt;O Driver é o processo que coordena a aplicação Spark.&lt;/p&gt;

&lt;p&gt;Entre suas responsabilidades estão:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Executar o programa principal.&lt;/li&gt;
&lt;li&gt;Construir e coordenar operações Spark.&lt;/li&gt;
&lt;li&gt;Planejar Jobs e Stages.&lt;/li&gt;
&lt;li&gt;Agendar Tasks.&lt;/li&gt;
&lt;li&gt;Acompanhar resultados e falhas.&lt;/li&gt;
&lt;li&gt;Interagir com os Executors.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Executor
&lt;/h3&gt;

&lt;p&gt;Os Executors são processos associados à aplicação Spark que executam Tasks e armazenam dados intermediários ou persistidos conforme necessário.&lt;/p&gt;

&lt;p&gt;Eles utilizam recursos como CPU, memória, disco e rede.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cluster Manager
&lt;/h3&gt;

&lt;p&gt;O Cluster Manager gerencia recursos computacionais do ambiente.&lt;/p&gt;

&lt;p&gt;Dependendo da plataforma, ele pode ser responsável por alocar containers, Pods ou processos para os componentes Spark.&lt;/p&gt;

&lt;p&gt;Uma visão simplificada:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver
  |
  | Solicita recursos e coordena execução
  |
  v
Cluster Manager
  |
  | Disponibiliza recursos
  |
  v
Executors
  |
  | Executam Tasks
  |
  v
Distributed Processing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;É importante lembrar que o Cluster Manager não substitui o scheduler interno do Spark.&lt;/p&gt;

&lt;p&gt;O Spark continua responsável por organizar e distribuir suas Tasks entre os Executors disponíveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. O que é Deployment Mode?
&lt;/h2&gt;

&lt;p&gt;Deployment Mode define, principalmente, &lt;strong&gt;onde o processo Driver será executado em relação ao ambiente que submete a aplicação&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Os dois modos mais conhecidos são:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Client Mode
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cluster Mode
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa distinção parece pequena, mas afeta diretamente:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Conectividade.&lt;/li&gt;
&lt;li&gt;Logs.&lt;/li&gt;
&lt;li&gt;Disponibilidade do Driver.&lt;/li&gt;
&lt;li&gt;Empacotamento da aplicação.&lt;/li&gt;
&lt;li&gt;Operação em produção.&lt;/li&gt;
&lt;li&gt;Diagnóstico de falhas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Vamos começar pelo Client Mode.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Client Mode
&lt;/h2&gt;

&lt;p&gt;No Client Mode, o Driver executa no ambiente de onde a aplicação é submetida.&lt;/p&gt;

&lt;p&gt;Imagine que estamos utilizando uma máquina de desenvolvimento ou um servidor de borda para executar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;spark-submit &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--deploy-mode&lt;/span&gt; client &lt;span class="se"&gt;\&lt;/span&gt;
  app.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em um ambiente de cluster compatível, a arquitetura conceitual é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;       MÁQUINA DE SUBMISSÃO
       +-------------------+
       |                   |
       |      DRIVER       |
       |                   |
       +---------+---------+
                 |
                 | Comunicação
                 |
                 v
          CLUSTER MANAGER
                 |
       +---------+---------+
       |         |         |
       v         v         v
   EXECUTOR 1 EXECUTOR 2 EXECUTOR 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Driver permanece no processo de submissão.&lt;/p&gt;

&lt;p&gt;Os Executors podem executar em outros nós.&lt;/p&gt;

&lt;p&gt;Isso significa que o Driver precisa conseguir se comunicar adequadamente com os Executors e com os serviços relevantes do cluster.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Por que Client Mode é útil?
&lt;/h2&gt;

&lt;p&gt;Client Mode costuma ser conveniente em cenários como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Desenvolvimento interativo.&lt;/li&gt;
&lt;li&gt;Depuração.&lt;/li&gt;
&lt;li&gt;Sessões de notebooks.&lt;/li&gt;
&lt;li&gt;Execução a partir de servidores de borda.&lt;/li&gt;
&lt;li&gt;Aplicações nas quais desejamos acompanhar diretamente a saída do Driver.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Imagine que estamos investigando um pipeline.&lt;/p&gt;

&lt;p&gt;Queremos observar mensagens como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Starting Spark application...
Loading dataset...
Executing aggregation...
Writing output...
Application finished.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No Client Mode, a saída do Driver normalmente fica associada ao ambiente de submissão.&lt;/p&gt;

&lt;p&gt;Isso facilita determinados fluxos de desenvolvimento e troubleshooting.&lt;/p&gt;

&lt;p&gt;Mas também traz responsabilidades.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. A máquina cliente faz parte da execução
&lt;/h2&gt;

&lt;p&gt;Um erro comum é imaginar:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Enviei a aplicação para o cluster. Agora minha máquina não importa mais.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Em Client Mode, isso não é verdade.&lt;/p&gt;

&lt;p&gt;O Driver continua executando no ambiente cliente.&lt;/p&gt;

&lt;p&gt;Se esse processo for encerrado, a aplicação poderá ser interrompida.&lt;/p&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Developer Laptop
       |
       v
     Driver
       |
       v
    Cluster
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se o laptop for desligado ou o processo Driver terminar, não devemos assumir que a aplicação continuará normalmente.&lt;/p&gt;

&lt;p&gt;O comportamento exato depende do ambiente e de seus mecanismos de supervisão, mas o princípio permanece:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O Driver é parte ativa da aplicação.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Client Mode e conectividade
&lt;/h2&gt;

&lt;p&gt;Outra consequência importante é a rede.&lt;/p&gt;

&lt;p&gt;Os Executors precisam se comunicar com o Driver.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Developer Laptop
       |
       | VPN / Firewall / NAT
       |
       v
   Spark Cluster
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se os Executors não conseguem alcançar o endereço anunciado pelo Driver, a aplicação pode falhar mesmo que exista CPU e memória disponíveis.&lt;/p&gt;

&lt;p&gt;Problemas comuns incluem:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Endereço incorreto do Driver.&lt;/li&gt;
&lt;li&gt;DNS.&lt;/li&gt;
&lt;li&gt;Firewall.&lt;/li&gt;
&lt;li&gt;Portas bloqueadas.&lt;/li&gt;
&lt;li&gt;NAT.&lt;/li&gt;
&lt;li&gt;Instabilidade de VPN.&lt;/li&gt;
&lt;li&gt;Interfaces de rede configuradas incorretamente.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Em ambientes corporativos, essa é uma das razões para utilizar servidores de borda ou plataformas de notebooks integradas ao cluster.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Cluster Mode
&lt;/h2&gt;

&lt;p&gt;Agora vamos mudar a localização do Driver.&lt;/p&gt;

&lt;p&gt;No Cluster Mode, o Driver é executado dentro do ambiente gerenciado pelo cluster, conforme os mecanismos da plataforma utilizada.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;       MÁQUINA DE SUBMISSÃO
       +-------------------+
       |                   |
       |   spark-submit    |
       |                   |
       +---------+---------+
                 |
                 | Submete aplicação
                 |
                 v
          CLUSTER MANAGER
                 |
       +---------+---------+
       |                   |
       v                   v
     DRIVER             EXECUTORS
       |                   |
       +---------+---------+
                 |
                 v
          DISTRIBUTED JOB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A diferença central é:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O Driver não precisa permanecer no processo cliente que iniciou a submissão.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Isso torna o Cluster Mode adequado a muitos cenários de execução automatizada.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Cluster Mode é melhor para produção?
&lt;/h2&gt;

&lt;p&gt;Frequentemente, Cluster Mode é uma escolha conveniente para Jobs batch de produção.&lt;/p&gt;

&lt;p&gt;Mas não devemos transformar isso em uma regra absoluta.&lt;/p&gt;

&lt;p&gt;Existem plataformas gerenciadas e arquiteturas específicas em que o ciclo de vida da aplicação é controlado de outras maneiras.&lt;/p&gt;

&lt;p&gt;O importante é compreender os requisitos.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Pipeline noturno
       |
       v
Scheduler
       |
       v
Application Submission
       |
       v
Cluster-managed Driver
       |
       v
Executors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nesse cenário, não queremos depender de uma sessão interativa de desenvolvimento para manter a aplicação ativa.&lt;/p&gt;

&lt;p&gt;Cluster Mode ajuda a separar o ambiente de submissão do ambiente de execução.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Client Mode vs. Cluster Mode
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Característica&lt;/th&gt;
&lt;th&gt;Client Mode&lt;/th&gt;
&lt;th&gt;Cluster Mode&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Localização do Driver&lt;/td&gt;
&lt;td&gt;Ambiente de submissão&lt;/td&gt;
&lt;td&gt;Ambiente gerenciado pelo cluster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Uso frequente&lt;/td&gt;
&lt;td&gt;Desenvolvimento e sessões interativas&lt;/td&gt;
&lt;td&gt;Jobs automatizados e batch&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Logs do Driver&lt;/td&gt;
&lt;td&gt;Próximos ao processo cliente&lt;/td&gt;
&lt;td&gt;Normalmente obtidos pela plataforma&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dependência do cliente&lt;/td&gt;
&lt;td&gt;Driver permanece no cliente&lt;/td&gt;
&lt;td&gt;Submissão pode terminar sem encerrar necessariamente a aplicação&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Conectividade&lt;/td&gt;
&lt;td&gt;Executors precisam alcançar o Driver cliente&lt;/td&gt;
&lt;td&gt;Driver executa na rede do cluster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Disponibilidade&lt;/td&gt;
&lt;td&gt;Depende do processo Driver cliente&lt;/td&gt;
&lt;td&gt;Depende do cluster e da supervisão&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Melhor escolha&lt;/td&gt;
&lt;td&gt;Depende do cenário&lt;/td&gt;
&lt;td&gt;Depende do cenário&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;O ponto principal é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Client Mode e Cluster Mode descrevem a localização do Driver, não a quantidade de dados processados nem o grau de paralelismo da aplicação.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  11. Deployment Mode não é Cluster Manager
&lt;/h2&gt;

&lt;p&gt;Agora precisamos evitar outra confusão.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Deployment Mode&lt;/strong&gt; responde:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Onde o Driver executa?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;strong&gt;Cluster Manager&lt;/strong&gt; responde:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Qual sistema gerencia os recursos do ambiente?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;São dimensões diferentes.&lt;/p&gt;

&lt;p&gt;Por exemplo, dependendo do suporte da plataforma, podemos encontrar combinações como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;YARN + Client Mode
YARN + Cluster Mode

Kubernetes + Client Mode
Kubernetes + Cluster Mode

Standalone + Client Mode
Standalone + Cluster Mode
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Entretanto, a disponibilidade de cada combinação depende do Cluster Manager, da linguagem e da versão do Spark.&lt;/p&gt;

&lt;p&gt;Em particular, o suporte a aplicações Python em Standalone Cluster Mode possui restrições documentadas em versões do Spark. Portanto, não devemos presumir que todas as combinações funcionam igualmente.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. Spark Standalone
&lt;/h2&gt;

&lt;p&gt;Spark Standalone é o gerenciador de cluster próprio do Apache Spark.&lt;/p&gt;

&lt;p&gt;Sua arquitetura clássica envolve:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;             STANDALONE MASTER
                     |
           +---------+---------+
           |         |         |
           v         v         v
        WORKER 1  WORKER 2  WORKER 3
           |         |         |
           v         v         v
        EXECUTOR  EXECUTOR  EXECUTOR
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Master coordena a disponibilidade dos recursos do cluster Standalone.&lt;/p&gt;

&lt;p&gt;Os Workers disponibilizam recursos computacionais.&lt;/p&gt;

&lt;p&gt;O Driver e os Executors são iniciados conforme o modo de implantação e os recursos disponíveis.&lt;/p&gt;

&lt;p&gt;Esse modelo é útil para compreender a arquitetura do Spark sem adicionar toda a complexidade de um gerenciador externo.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. Spark em YARN
&lt;/h2&gt;

&lt;p&gt;YARN significa:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Yet Another Resource Negotiator.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Ele faz parte do ecossistema Hadoop e gerencia recursos para aplicações distribuídas.&lt;/p&gt;

&lt;p&gt;Uma visão simplificada:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  YARN
                   |
             ResourceManager
                   |
        +----------+----------+
        |          |          |
        v          v          v
     NodeManager NodeManager NodeManager
        |          |          |
        v          v          v
     Containers Containers Containers
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Spark pode utilizar esses recursos para executar seus componentes.&lt;/p&gt;

&lt;p&gt;No YARN, existe também o conceito de &lt;strong&gt;ApplicationMaster&lt;/strong&gt;, cujo papel e relacionamento com o Driver variam conforme o deployment mode.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. YARN Client Mode
&lt;/h2&gt;

&lt;p&gt;No YARN Client Mode, o Driver executa no ambiente cliente.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;        CLIENT MACHINE
        +------------+
        |   DRIVER   |
        +-----+------+
              |
              v
        YARN CLUSTER
              |
        ApplicationMaster
              |
       +------+------+
       |             |
       v             v
   Executor 1    Executor 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O ApplicationMaster participa do gerenciamento dos recursos da aplicação no YARN.&lt;/p&gt;

&lt;p&gt;O Driver permanece no cliente.&lt;/p&gt;

&lt;p&gt;Esse detalhe é importante ao interpretar logs e falhas.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. YARN Cluster Mode
&lt;/h2&gt;

&lt;p&gt;No YARN Cluster Mode, o Driver executa no cluster, dentro do contexto da aplicação gerenciada pelo YARN.&lt;/p&gt;

&lt;p&gt;Uma visão conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;       SUBMISSION CLIENT
               |
               v
         YARN CLUSTER
               |
               v
        ApplicationMaster
        +--------------+
        |    DRIVER    |
        +------+-------+
               |
        +------+------+
        |             |
        v             v
    Executor 1    Executor 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nesse modo, o Driver não depende de continuar executando na máquina que fez a submissão.&lt;/p&gt;

&lt;p&gt;Para workloads batch, isso frequentemente simplifica a operação.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. Spark em Kubernetes
&lt;/h2&gt;

&lt;p&gt;Kubernetes utiliza uma arquitetura baseada em recursos como Pods, Deployments, Services e outros objetos.&lt;/p&gt;

&lt;p&gt;No Spark, podemos executar o Driver e os Executors em Pods.&lt;/p&gt;

&lt;p&gt;Uma visão típica de Cluster Mode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;             spark-submit
                  |
                  v
          KUBERNETES API
                  |
                  v
              DRIVER POD
                  |
                  v
          EXECUTOR PODS
          /     |     \
         v      v      v
       Exec 1 Exec 2 Exec 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Driver solicita os recursos necessários por meio da integração Spark com Kubernetes.&lt;/p&gt;

&lt;p&gt;Os Executors executam em Pods associados à aplicação.&lt;/p&gt;

&lt;p&gt;Isso facilita a integração com práticas modernas de infraestrutura containerizada.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Kubernetes não elimina a arquitetura do Spark
&lt;/h2&gt;

&lt;p&gt;É importante compreender:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Kubernetes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;não substitui:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark DAG Scheduler
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;nem:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark Task Scheduler
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Kubernetes gerencia recursos e o ciclo de vida dos Pods.&lt;/p&gt;

&lt;p&gt;O Spark continua responsável por:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Construir planos de execução.&lt;/li&gt;
&lt;li&gt;Organizar Jobs e Stages.&lt;/li&gt;
&lt;li&gt;Distribuir Tasks.&lt;/li&gt;
&lt;li&gt;Coordenar Executors.&lt;/li&gt;
&lt;li&gt;Gerenciar shuffle e recomputação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Podemos pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Kubernetes
    |
    v
Resource Orchestration
    |
    v
Spark Application
    |
    v
Distributed Data Processing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;São responsabilidades complementares.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. E o modo local?
&lt;/h2&gt;

&lt;p&gt;Antes de utilizar um cluster real, podemos executar Spark localmente.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;master&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;local[*]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SparkLocal&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O parâmetro:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;local[*]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;indica que o Spark pode utilizar os núcleos lógicos disponíveis na máquina local.&lt;/p&gt;

&lt;p&gt;Também podemos utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;local[2]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;para configurar dois threads de execução local.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;master&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;local[2]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SparkLocal&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso é útil para desenvolvimento e experimentação.&lt;/p&gt;

&lt;p&gt;Mas precisamos lembrar que uma execução local não reproduz integralmente:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Latência de rede entre nós.&lt;/li&gt;
&lt;li&gt;Comportamento de um cluster real.&lt;/li&gt;
&lt;li&gt;Falhas distribuídas.&lt;/li&gt;
&lt;li&gt;Restrições de containers.&lt;/li&gt;
&lt;li&gt;Escalabilidade de armazenamento.&lt;/li&gt;
&lt;li&gt;Dinâmica de alocação de recursos.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Portanto, testes locais são importantes, mas não substituem validação no ambiente de destino.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. O que é &lt;code&gt;spark-submit&lt;/code&gt;?
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;spark-submit&lt;/code&gt; é a ferramenta de submissão de aplicações Spark.&lt;/p&gt;

&lt;p&gt;Ela permite informar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Aplicação.&lt;/li&gt;
&lt;li&gt;Cluster Manager.&lt;/li&gt;
&lt;li&gt;Deployment Mode.&lt;/li&gt;
&lt;li&gt;Recursos.&lt;/li&gt;
&lt;li&gt;Configurações.&lt;/li&gt;
&lt;li&gt;Dependências.&lt;/li&gt;
&lt;li&gt;Argumentos da aplicação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A estrutura geral é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;spark-submit &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--master&lt;/span&gt; &amp;lt;MASTER&amp;gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--deploy-mode&lt;/span&gt; &amp;lt;MODE&amp;gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--conf&lt;/span&gt; &amp;lt;&lt;span class="nv"&gt;KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;VALUE&amp;gt; &lt;span class="se"&gt;\&lt;/span&gt;
  app.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os valores exatos dependem do ambiente.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Exemplo local com &lt;code&gt;spark-submit&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Vamos criar uma aplicação simples.&lt;/p&gt;

&lt;p&gt;Arquivo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;app.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Conteúdo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ExemploSparkSubmit&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;dados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1_000_001&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dados&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Quantidade de números pares:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Executamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;spark-submit &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--master&lt;/span&gt; &lt;span class="s2"&gt;"local[*]"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  app.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Resultado esperado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Quantidade de números pares: 500000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse exemplo é suficiente para compreender a estrutura básica de uma submissão.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. Exemplo de submissão para YARN
&lt;/h2&gt;

&lt;p&gt;Em um ambiente YARN configurado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;spark-submit &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--master&lt;/span&gt; yarn &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--deploy-mode&lt;/span&gt; cluster &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--driver-memory&lt;/span&gt; 4g &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--executor-memory&lt;/span&gt; 8g &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--executor-cores&lt;/span&gt; 2 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--num-executors&lt;/span&gt; 4 &lt;span class="se"&gt;\&lt;/span&gt;
  app.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Aqui estamos solicitando uma aplicação com determinados recursos.&lt;/p&gt;

&lt;p&gt;Mas atenção:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Esses valores são ilustrativos, não uma recomendação universal de dimensionamento.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;O tamanho adequado depende de:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Volume de dados.&lt;/li&gt;
&lt;li&gt;Tipo de operação.&lt;/li&gt;
&lt;li&gt;Quantidade de partições.&lt;/li&gt;
&lt;li&gt;Shuffle.&lt;/li&gt;
&lt;li&gt;Memória.&lt;/li&gt;
&lt;li&gt;Concorrência.&lt;/li&gt;
&lt;li&gt;Limites do cluster.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Esse raciocínio conecta diretamente a Parte 7.&lt;/p&gt;




&lt;h2&gt;
  
  
  22. Exemplo de submissão para Kubernetes
&lt;/h2&gt;

&lt;p&gt;Em um ambiente Kubernetes com Spark configurado, uma submissão pode seguir este formato:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;spark-submit &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--master&lt;/span&gt; k8s://https://K8S_API_SERVER:6443 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--deploy-mode&lt;/span&gt; cluster &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--name&lt;/span&gt; spark-example &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--conf&lt;/span&gt; spark.kubernetes.container.image&lt;span class="o"&gt;=&lt;/span&gt;REGISTRY/SPARK_IMAGE:TAG &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--conf&lt;/span&gt; spark.executor.instances&lt;span class="o"&gt;=&lt;/span&gt;3 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nb"&gt;local&lt;/span&gt;:///opt/spark/app.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os valores são placeholders e precisam ser substituídos pelos recursos reais do ambiente.&lt;/p&gt;

&lt;p&gt;Observe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;k8s://
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;que identifica a integração com Kubernetes.&lt;/p&gt;

&lt;p&gt;E:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;local:///opt/spark/app.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;que, nesse exemplo, indica um arquivo disponível dentro da imagem/container utilizado na execução.&lt;/p&gt;

&lt;p&gt;A imagem precisa conter a aplicação nesse caminho, além de estar preparada para executar Spark corretamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. &lt;code&gt;--master&lt;/code&gt; e &lt;code&gt;--deploy-mode&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Esses dois parâmetros merecem atenção especial.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;--master&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Indica o ambiente de execução ou o gerenciador de cluster.&lt;/p&gt;

&lt;p&gt;Exemplos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;local[*]
spark://HOST:7077
yarn
k8s://https://HOST:6443
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  &lt;code&gt;--deploy-mode&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Indica a localização do Driver.&lt;/p&gt;

&lt;p&gt;Exemplos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;client
cluster
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nt"&gt;--master&lt;/span&gt; yarn &lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="nt"&gt;--deploy-mode&lt;/span&gt; cluster
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;significa:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Utilizar YARN como Cluster Manager e executar o Driver no ambiente do cluster.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;São decisões relacionadas, mas distintas.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. O que acontece depois do &lt;code&gt;spark-submit&lt;/code&gt;?
&lt;/h2&gt;

&lt;p&gt;Uma sequência conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. spark-submit
       |
       v
2. Application Submission
       |
       v
3. Driver Initialization
       |
       v
4. SparkSession / SparkContext
       |
       v
5. Resource Allocation
       |
       v
6. Executors Available
       |
       v
7. Action Triggered
       |
       v
8. Job Created
       |
       v
9. Stages Planned
       |
       v
10. Tasks Scheduled
       |
       v
11. Distributed Execution
       |
       v
12. Results / Writes
       |
       v
13. Application Completion
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A ordem exata de alguns eventos depende do Cluster Manager e do modo de implantação.&lt;/p&gt;

&lt;p&gt;Mas essa sequência ajuda a conectar submissão e execução distribuída.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. O Driver precisa acessar os dados?
&lt;/h2&gt;

&lt;p&gt;Essa pergunta merece cuidado.&lt;/p&gt;

&lt;p&gt;O Driver normalmente coordena o processamento, mas a leitura distribuída de grandes datasets é realizada pelos Executors.&lt;/p&gt;

&lt;p&gt;Ainda assim, dependendo do código e das fontes utilizadas, o Driver pode precisar acessar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Metadados.&lt;/li&gt;
&lt;li&gt;Catálogos.&lt;/li&gt;
&lt;li&gt;Configurações.&lt;/li&gt;
&lt;li&gt;Endpoints de serviços.&lt;/li&gt;
&lt;li&gt;Informações de schema.&lt;/li&gt;
&lt;li&gt;Caminhos e credenciais.&lt;/li&gt;
&lt;li&gt;Recursos necessários para planejamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Não devemos assumir que apenas os Executors precisam de conectividade com os sistemas externos.&lt;/p&gt;

&lt;p&gt;A arquitetura precisa considerar o acesso de cada componente.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. O erro clássico de caminhos locais
&lt;/h2&gt;

&lt;p&gt;Imagine uma aplicação:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/home/usuario/dados.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ela funciona no notebook local.&lt;/p&gt;

&lt;p&gt;Depois enviamos para um cluster.&lt;/p&gt;

&lt;p&gt;Agora os Executors estão em outras máquinas.&lt;/p&gt;

&lt;p&gt;O caminho:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/home/usuario/dados.csv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;pode não existir nesses nós.&lt;/p&gt;

&lt;p&gt;O problema não está necessariamente no código de leitura.&lt;/p&gt;

&lt;p&gt;Está na arquitetura de armazenamento.&lt;/p&gt;

&lt;p&gt;Em produção, geralmente utilizamos fontes acessíveis aos componentes relevantes do cluster, como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HDFS
S3
ADLS
GCS
Distributed Storage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou volumes compartilhados adequadamente configurados.&lt;/p&gt;

&lt;p&gt;Essa é uma das diferenças mais importantes entre desenvolvimento local e execução distribuída.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Código local não é necessariamente código distribuído
&lt;/h2&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/home/usuario/config.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;arquivo&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;configuracao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;arquivo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se isso acontecer no Driver, o arquivo precisa estar acessível ao Driver.&lt;/p&gt;

&lt;p&gt;Agora imagine que uma função executada pelos workers tente abrir esse mesmo caminho.&lt;/p&gt;

&lt;p&gt;Os Executors podem estar em máquinas ou containers diferentes.&lt;/p&gt;

&lt;p&gt;Então precisamos perguntar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Onde o arquivo existe?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual processo precisa acessá-lo?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse raciocínio se conecta à Parte 9 sobre closures, serialização e variáveis distribuídas.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Empacotamento da aplicação
&lt;/h2&gt;

&lt;p&gt;Em projetos reais, uma aplicação Spark raramente é apenas um arquivo.&lt;/p&gt;

&lt;p&gt;Podemos ter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark-project/
|
+-- app.py
+-- src/
|   +-- transformations/
|   +-- validations/
|   +-- connectors/
|
+-- config/
+-- tests/
+-- requirements.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em produção, precisamos garantir que o código e suas dependências estejam disponíveis onde serão executados.&lt;/p&gt;

&lt;p&gt;Isso pode envolver:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Pacotes Python.&lt;/li&gt;
&lt;li&gt;Arquivos distribuídos.&lt;/li&gt;
&lt;li&gt;Imagens de container.&lt;/li&gt;
&lt;li&gt;Bibliotecas JVM.&lt;/li&gt;
&lt;li&gt;Dependências nativas.&lt;/li&gt;
&lt;li&gt;Configuração de ambiente.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma aplicação que funciona localmente pode falhar em produção simplesmente porque uma dependência não foi disponibilizada aos Executors.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. Python e ambiente de execução
&lt;/h2&gt;

&lt;p&gt;Em PySpark, precisamos lembrar que existe execução de código Python associada aos workers.&lt;/p&gt;

&lt;p&gt;Isso significa que versões e dependências Python importam.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver
  |
  +-- Python Environment
  |
Executors
  |
  +-- Python Workers
      |
      +-- Required Libraries
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se uma biblioteca existe apenas na máquina cliente, mas não está disponível no ambiente dos workers, determinadas operações podem falhar.&lt;/p&gt;

&lt;p&gt;Em ambientes containerizados, uma estratégia comum é construir imagens que contenham as dependências necessárias.&lt;/p&gt;

&lt;p&gt;Em outras plataformas, podemos distribuir ambientes ou pacotes conforme os mecanismos suportados.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. Logs: onde procurar?
&lt;/h2&gt;

&lt;p&gt;Agora chegamos a uma questão prática.&lt;/p&gt;

&lt;p&gt;Quando uma aplicação falha, onde estão os logs?&lt;/p&gt;

&lt;p&gt;A resposta depende do ambiente.&lt;/p&gt;

&lt;h3&gt;
  
  
  Client Mode
&lt;/h3&gt;

&lt;p&gt;Os logs do Driver normalmente estão associados ao processo de submissão.&lt;/p&gt;

&lt;p&gt;Os logs dos Executors ficam no ambiente onde eles executam.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cluster Mode
&lt;/h3&gt;

&lt;p&gt;Os logs do Driver normalmente precisam ser consultados por meio dos mecanismos da plataforma.&lt;/p&gt;

&lt;p&gt;Em Kubernetes, por exemplo, podemos investigar os logs do Driver Pod.&lt;/p&gt;

&lt;p&gt;Em YARN, podemos utilizar os mecanismos de logs da aplicação.&lt;/p&gt;

&lt;p&gt;O ponto principal é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Saber onde o Driver executa ajuda a saber onde procurar o erro.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  31. Spark UI
&lt;/h2&gt;

&lt;p&gt;A Spark UI continua sendo uma das ferramentas mais importantes de observabilidade.&lt;/p&gt;

&lt;p&gt;Ela permite investigar informações como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Jobs
Stages
Tasks
Executors
Storage
SQL Queries
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Dependendo do ambiente e da configuração, podemos acessar a interface diretamente ou por meio de proxies, túneis e interfaces da plataforma.&lt;/p&gt;

&lt;p&gt;A Spark UI normalmente utiliza a porta &lt;code&gt;4040&lt;/code&gt; como ponto inicial para aplicações em execução, mas a porta pode variar quando já está ocupada ou quando a plataforma utiliza mecanismos próprios.&lt;/p&gt;

&lt;p&gt;Não devemos assumir que a interface estará diretamente exposta na rede.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Spark History Server
&lt;/h2&gt;

&lt;p&gt;Existe uma diferença importante entre observar uma aplicação em execução e investigar uma aplicação que já terminou.&lt;/p&gt;

&lt;p&gt;Durante a execução, a Spark UI oferece informações do estado atual.&lt;/p&gt;

&lt;p&gt;Para análise posterior, podemos utilizar o &lt;strong&gt;Spark History Server&lt;/strong&gt;, quando o registro de eventos está habilitado e configurado.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark Application
       |
       v
Event Logs
       |
       v
Persistent Storage
       |
       v
Spark History Server
       |
       v
Post-execution Analysis
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso é especialmente útil em ambientes de produção.&lt;/p&gt;

&lt;p&gt;Imagine um pipeline que executa às 3h da manhã.&lt;/p&gt;

&lt;p&gt;Se ele falhar, precisamos conseguir investigar o que aconteceu depois.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. Falha de Executor vs. falha de Driver
&lt;/h2&gt;

&lt;p&gt;Vamos conectar com a Parte 8.&lt;/p&gt;

&lt;h3&gt;
  
  
  Executor Failure
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor 2
    |
    v
Failure
    |
    v
Lost Tasks / Data
    |
    v
Retry / Recomputation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em determinados cenários, Spark consegue recuperar o trabalho.&lt;/p&gt;

&lt;h3&gt;
  
  
  Driver Failure
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver
   |
   v
Failure
   |
   v
Application Coordination Lost
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A falha do Driver possui consequências diferentes.&lt;/p&gt;

&lt;p&gt;O reinício ou a recuperação da aplicação depende dos mecanismos de implantação, supervisão e tolerância a falhas disponíveis.&lt;/p&gt;

&lt;p&gt;Por isso, escolher Client Mode ou Cluster Mode também influencia a estratégia operacional.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Cluster Mode não significa alta disponibilidade automática
&lt;/h2&gt;

&lt;p&gt;Esse é um ponto importante.&lt;/p&gt;

&lt;p&gt;Executar o Driver dentro do cluster não garante, por si só, que ele seja automaticamente recuperado de qualquer falha.&lt;/p&gt;

&lt;p&gt;Podemos ter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cluster Mode
     |
     v
Driver Pod
     |
     v
Failure
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A recuperação depende de mecanismos como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Supervisão da aplicação.&lt;/li&gt;
&lt;li&gt;Políticas de reinício.&lt;/li&gt;
&lt;li&gt;Recursos do Cluster Manager.&lt;/li&gt;
&lt;li&gt;Orquestração externa.&lt;/li&gt;
&lt;li&gt;Persistência de estado.&lt;/li&gt;
&lt;li&gt;Reexecução do Job.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cluster Mode
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;não é sinônimo de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Exactly-once execution
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;nem de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Automatic complete recovery
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essas garantias precisam ser analisadas separadamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. Orquestração de Jobs Spark
&lt;/h2&gt;

&lt;p&gt;Em ambientes reais, aplicações Spark frequentemente fazem parte de pipelines maiores.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Airflow / Orchestrator
        |
        v
spark-submit
        |
        v
Spark Application
        |
        v
Data Lake
        |
        v
Validation
        |
        v
Analytics
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O orquestrador controla aspectos como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Agendamento.&lt;/li&gt;
&lt;li&gt;Dependências entre Jobs.&lt;/li&gt;
&lt;li&gt;Retentativas.&lt;/li&gt;
&lt;li&gt;Alertas.&lt;/li&gt;
&lt;li&gt;Parâmetros.&lt;/li&gt;
&lt;li&gt;Monitoramento do workflow.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O Spark executa o processamento distribuído.&lt;/p&gt;

&lt;p&gt;São responsabilidades complementares.&lt;/p&gt;




&lt;h2&gt;
  
  
  36. Uma arquitetura de produção
&lt;/h2&gt;

&lt;p&gt;Imagine uma plataforma de Engenharia de Dados:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 ORCHESTRATOR
                      |
                      v
                 SPARK JOB
                      |
                      v
                 CLUSTER MANAGER
                      |
             +--------+--------+
             |        |        |
             v        v        v
          DRIVER   EXECUTOR  EXECUTOR
             |        |        |
             +--------+--------+
                      |
                      v
                  DATA LAKE
                      |
                      v
               DATA QUALITY
                      |
                      v
                 ANALYTICS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Na prática, o Driver coordena os Executors, e o Cluster Manager administra os recursos necessários à aplicação.&lt;/p&gt;

&lt;p&gt;Esse tipo de arquitetura exige decisões sobre:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Deployment Mode.&lt;/li&gt;
&lt;li&gt;Recursos.&lt;/li&gt;
&lt;li&gt;Armazenamento.&lt;/li&gt;
&lt;li&gt;Dependências.&lt;/li&gt;
&lt;li&gt;Logs.&lt;/li&gt;
&lt;li&gt;Segurança.&lt;/li&gt;
&lt;li&gt;Observabilidade.&lt;/li&gt;
&lt;li&gt;Recuperação.&lt;/li&gt;
&lt;li&gt;Custos.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Não basta que o código funcione.&lt;/p&gt;

&lt;p&gt;Precisamos que ele seja operável.&lt;/p&gt;




&lt;h2&gt;
  
  
  37. Um checklist antes de implantar
&lt;/h2&gt;

&lt;p&gt;Antes de enviar uma aplicação Spark para produção, gosto de responder:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Qual Cluster Manager será utilizado?&lt;/li&gt;
&lt;li&gt;O Driver executará em Client ou Cluster Mode?&lt;/li&gt;
&lt;li&gt;Onde os Executors serão iniciados?&lt;/li&gt;
&lt;li&gt;Driver e Executors conseguem se comunicar?&lt;/li&gt;
&lt;li&gt;Os dados estão acessíveis ao cluster?&lt;/li&gt;
&lt;li&gt;As dependências Python estão disponíveis?&lt;/li&gt;
&lt;li&gt;As credenciais estão configuradas com segurança?&lt;/li&gt;
&lt;li&gt;Qual é a política de recursos?&lt;/li&gt;
&lt;li&gt;Onde ficam os logs do Driver?&lt;/li&gt;
&lt;li&gt;Onde ficam os logs dos Executors?&lt;/li&gt;
&lt;li&gt;Os event logs estão habilitados?&lt;/li&gt;
&lt;li&gt;Como a aplicação será monitorada?&lt;/li&gt;
&lt;li&gt;Como falhas serão tratadas?&lt;/li&gt;
&lt;li&gt;O Job pode ser reexecutado com segurança?&lt;/li&gt;
&lt;li&gt;Como serão controlados custos e consumo de recursos?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Essas perguntas transformam uma aplicação experimental em uma aplicação operacionalmente preparada.&lt;/p&gt;




&lt;h2&gt;
  
  
  38. Problemas comuns de implantação
&lt;/h2&gt;

&lt;p&gt;Alguns erros aparecem repetidamente.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A aplicação funciona localmente, mas falha no cluster.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Possíveis causas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Dependências ausentes.&lt;/li&gt;
&lt;li&gt;Arquivos locais indisponíveis.&lt;/li&gt;
&lt;li&gt;Credenciais incorretas.&lt;/li&gt;
&lt;li&gt;Diferenças de versão.&lt;/li&gt;
&lt;li&gt;Configuração de rede.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Executors não conseguem se conectar ao Driver.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Possíveis causas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Firewall.&lt;/li&gt;
&lt;li&gt;DNS.&lt;/li&gt;
&lt;li&gt;Endereço anunciado incorretamente.&lt;/li&gt;
&lt;li&gt;Portas.&lt;/li&gt;
&lt;li&gt;Configuração de rede.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;A aplicação é encerrada após fechar a sessão cliente.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Verifique se o Driver está executando em Client Mode e como o processo é supervisionado.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O Job termina, mas não conseguimos investigar sua execução.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Verifique event logs, retenção e configuração do History Server.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O Driver funciona, mas Tasks falham com erro de biblioteca.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Verifique as dependências nos ambientes dos workers.&lt;/p&gt;

&lt;p&gt;Esses problemas não são necessariamente erros de lógica de transformação.&lt;/p&gt;

&lt;p&gt;Muitas vezes são erros de implantação.&lt;/p&gt;




&lt;h2&gt;
  
  
  39. O que muda em plataformas gerenciadas?
&lt;/h2&gt;

&lt;p&gt;Em plataformas gerenciadas de Spark, muitos detalhes de infraestrutura são abstraídos.&lt;/p&gt;

&lt;p&gt;Podemos encontrar interfaces para:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Criar clusters.&lt;/li&gt;
&lt;li&gt;Definir Jobs.&lt;/li&gt;
&lt;li&gt;Configurar recursos.&lt;/li&gt;
&lt;li&gt;Consultar logs.&lt;/li&gt;
&lt;li&gt;Executar notebooks.&lt;/li&gt;
&lt;li&gt;Agendar pipelines.&lt;/li&gt;
&lt;li&gt;Gerenciar dependências.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Isso simplifica a operação.&lt;/p&gt;

&lt;p&gt;Mas não elimina os fundamentos.&lt;/p&gt;

&lt;p&gt;Continuamos tendo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver
Executors
Tasks
Partitions
Shuffle
Memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A plataforma pode esconder parte da complexidade operacional.&lt;/p&gt;

&lt;p&gt;Ela não elimina a física do processamento distribuído.&lt;/p&gt;




&lt;h2&gt;
  
  
  40. A relação com custos
&lt;/h2&gt;

&lt;p&gt;Agora imagine duas aplicações.&lt;/p&gt;

&lt;h3&gt;
  
  
  Aplicação A
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver
  |
  v
4 Executors
  |
  v
Job completes in 10 minutes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Aplicação B
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver
  |
  v
20 Executors
  |
  v
Job completes in 9 minutes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A aplicação B utiliza muito mais recursos para economizar apenas um minuto.&lt;/p&gt;

&lt;p&gt;Dependendo do modelo de cobrança, isso pode ser economicamente pior.&lt;/p&gt;

&lt;p&gt;Mais Executors não significam automaticamente melhor eficiência.&lt;/p&gt;

&lt;p&gt;Precisamos avaliar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Execution Time
      +
Resource Consumption
      +
Infrastructure Cost
      +
Operational Requirements
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse raciocínio conecta performance tuning e FinOps.&lt;/p&gt;




&lt;h2&gt;
  
  
  41. O modelo mental completo
&lt;/h2&gt;

&lt;p&gt;Agora podemos conectar os dez primeiros capítulos.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  SPARK APPLICATION
                         |
                         v
                     SUBMISSION
                         |
                         v
                   DEPLOYMENT MODE
                         |
              +----------+----------+
              |                     |
              v                     v
           CLIENT                 CLUSTER
              |                     |
              +----------+----------+
                         |
                         v
                       DRIVER
                         |
                         v
                    SPARK PLAN
                         |
                         v
                       JOBS
                         |
                         v
                      STAGES
                         |
                         v
                       TASKS
                         |
                         v
                     EXECUTORS
                         |
              +----------+----------+
              |          |          |
              v          v          v
           MEMORY      SHUFFLE     CPU
              |          |          |
              +----------+----------+
                         |
                         v
                       RESULT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa visão mostra que a arquitetura do Spark possui duas dimensões complementares.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A dimensão de implantação:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Where does it run?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;A dimensão de execução:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;How does it process data?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Para trabalhar com Spark em produção, precisamos compreender as duas.&lt;/p&gt;




&lt;h2&gt;
  
  
  42. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;Se eu tivesse que resumir Client Mode vs. Cluster Mode em uma única ideia, seria:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;O Deployment Mode define onde o Driver executa; o Cluster Manager administra os recursos; e o Spark continua responsável por coordenar o processamento distribuído.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Parece uma distinção simples.&lt;/p&gt;

&lt;p&gt;Mas ela influencia:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Arquitetura.&lt;/li&gt;
&lt;li&gt;Rede.&lt;/li&gt;
&lt;li&gt;Disponibilidade.&lt;/li&gt;
&lt;li&gt;Observabilidade.&lt;/li&gt;
&lt;li&gt;Troubleshooting.&lt;/li&gt;
&lt;li&gt;Segurança.&lt;/li&gt;
&lt;li&gt;Operação.&lt;/li&gt;
&lt;li&gt;Custos.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Quando sabemos onde cada componente executa, muitos erros deixam de parecer misteriosos.&lt;/p&gt;




&lt;h2&gt;
  
  
  43. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando recebo um problema como:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A aplicação Spark funciona no notebook, mas falha em produção.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Não começo necessariamente pelo DataFrame.&lt;/p&gt;

&lt;p&gt;Primeiro quero entender:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual é o ambiente?
        |
        v
Qual é o Cluster Manager?
        |
        v
Client ou Cluster Mode?
        |
        v
Onde está o Driver?
        |
        v
Onde estão os Executors?
        |
        v
Os dados são acessíveis?
        |
        v
As dependências estão disponíveis?
        |
        v
Existe conectividade?
        |
        v
Onde estão os logs?
        |
        v
Qual componente falhou?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Só depois avanço para o plano físico, memória, shuffle e operações distribuídas, quando esses forem os pontos relevantes.&lt;/p&gt;

&lt;p&gt;Essa abordagem reduz o tempo de diagnóstico e evita alterar configurações que não têm relação com a causa real.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora sabemos como uma aplicação Spark é submetida e onde seus componentes executam.&lt;/p&gt;

&lt;p&gt;Mas existe outro desafio importante:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Como configurar corretamente os recursos de uma aplicação Spark?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;No próximo capítulo vamos aprofundar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;spark.executor.instances&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;spark.executor.cores&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;spark.executor.memory&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;spark.driver.memory&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Dynamic Allocation.&lt;/li&gt;
&lt;li&gt;Paralelismo.&lt;/li&gt;
&lt;li&gt;Número de partições.&lt;/li&gt;
&lt;li&gt;Resource sizing.&lt;/li&gt;
&lt;li&gt;Relação entre CPU, memória e shuffle.&lt;/li&gt;
&lt;li&gt;Dimensionamento para workloads batch.&lt;/li&gt;
&lt;li&gt;Trade-offs entre desempenho e custo.&lt;/li&gt;
&lt;li&gt;Estratégias de benchmarking.&lt;/li&gt;
&lt;li&gt;Como evitar configurações copiadas sem diagnóstico.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Vamos conectar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Workload
   |
   v
Data Volume
   |
   v
Partitions
   |
   v
Parallelism
   |
   v
Executor Resources
   |
   v
Execution Time
   |
   v
Infrastructure Cost
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Porque configurar recursos não é apenas escolher números.&lt;/p&gt;

&lt;p&gt;É compreender como a aplicação utiliza o cluster.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 9:&lt;/strong&gt; Broadcast, Accumulators e Closures&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 11 — Dimensionamento e configuração de recursos no Apache Spark&lt;/p&gt;

</description>
      <category>bigdata</category>
      <category>spark</category>
      <category>apache</category>
      <category>python</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 9: Broadcast, Accumulators e Closures</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 19:40:25 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-9-broadcast-accumulators-e-closures-1p3o</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-9-broadcast-accumulators-e-closures-1p3o</guid>
      <description>&lt;p&gt;Nos artigos anteriores, construímos uma visão progressiva da arquitetura do Apache Spark.&lt;/p&gt;

&lt;p&gt;Começamos com RDDs, passamos por partições, Tasks, Executors, Shuffle, Spark SQL, memória e chegamos à tolerância a falhas.&lt;/p&gt;

&lt;p&gt;Na Parte 8, vimos que o Spark consegue reconstruir dados perdidos utilizando &lt;strong&gt;lineage&lt;/strong&gt;, dependências e recomputação.&lt;/p&gt;

&lt;p&gt;Agora surge uma pergunta igualmente importante:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O que acontece com as variáveis do nosso programa quando o processamento sai do Driver e passa a acontecer nos Executors?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Em uma aplicação Python tradicional, podemos escrever:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;contador&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;contador&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;contador&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Resultado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tudo acontece dentro de um único processo.&lt;/p&gt;

&lt;p&gt;Mas o Spark trabalha com computação distribuída.&lt;/p&gt;

&lt;p&gt;Temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                    DRIVER
                       |
             Distribuição de Tasks
                       |
             +---------+---------+
             |         |         |
             v         v         v
         Executor 1 Executor 2 Executor 3
             |         |         |
           Task A    Task B    Task C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E isso muda profundamente o significado de uma variável.&lt;/p&gt;

&lt;p&gt;Neste capítulo, vamos entender:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Closures e serialização.&lt;/li&gt;
&lt;li&gt;Variáveis locais do Driver.&lt;/li&gt;
&lt;li&gt;Broadcast Variables.&lt;/li&gt;
&lt;li&gt;Accumulators.&lt;/li&gt;
&lt;li&gt;Diferenças entre variáveis comuns e mecanismos distribuídos.&lt;/li&gt;
&lt;li&gt;Side effects e retries.&lt;/li&gt;
&lt;li&gt;Problemas de consistência.&lt;/li&gt;
&lt;li&gt;Boas práticas em PySpark.&lt;/li&gt;
&lt;li&gt;Quando usar cada mecanismo.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. O problema das variáveis em sistemas distribuídos
&lt;/h2&gt;

&lt;p&gt;Considere este exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SparkSession&lt;/span&gt;

&lt;span class="n"&gt;spark&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;SparkSession&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;shared-variables&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOrCreate&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;sc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sparkContext&lt;/span&gt;

&lt;span class="n"&gt;numeros&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallelize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="mi"&gt;4&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;contador&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;incrementar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;numero&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;global&lt;/span&gt; &lt;span class="n"&gt;contador&lt;/span&gt;
    &lt;span class="n"&gt;contador&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;numeros&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;incrementar&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;contador&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Alguém acostumado com Python local poderia esperar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas em uma execução distribuída típica, o valor observado no Driver permanece:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Por quê?&lt;/p&gt;

&lt;p&gt;Porque as Tasks não executam diretamente dentro do mesmo espaço de memória do Driver.&lt;/p&gt;

&lt;p&gt;O Spark serializa funções e os dados necessários para executá-las em outros processos.&lt;/p&gt;

&lt;p&gt;O &lt;code&gt;contador&lt;/code&gt; utilizado pelos workers não é um contador global compartilhado com o Driver.&lt;/p&gt;

&lt;p&gt;Essa é nossa primeira lição:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Uma variável Python comum não se transforma automaticamente em estado compartilhado apenas porque está sendo utilizada dentro de uma transformação Spark.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  2. Driver e Executors possuem espaços de memória diferentes
&lt;/h2&gt;

&lt;p&gt;Vamos visualizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DRIVER
|
| contador = 0
|
+-------------------------------+
                                |
                          Serialização
                                |
              +-----------------+-----------------+
              |                 |                 |
              v                 v                 v
          EXECUTOR 1        EXECUTOR 2        EXECUTOR 3
              |                 |                 |
         Python Worker     Python Worker     Python Worker
              |                 |                 |
         Estado local      Estado local      Estado local
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O código enviado para execução remota não passa a compartilhar livremente a memória do processo original.&lt;/p&gt;

&lt;p&gt;Mesmo quando diferentes Tasks são executadas no mesmo Executor, não devemos assumir que variáveis Python comuns constituem um estado global consistente entre elas.&lt;/p&gt;

&lt;p&gt;Isso também significa que:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;minha_lista&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;dentro de uma Task não é uma maneira confiável de preencher uma lista existente no Driver.&lt;/p&gt;

&lt;p&gt;Para produzir resultados distribuídos, normalmente devemos utilizar as próprias transformações e Actions do Spark.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. O que são Closures?
&lt;/h2&gt;

&lt;p&gt;Uma &lt;strong&gt;closure&lt;/strong&gt; é uma função que utiliza valores definidos em seu contexto externo.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;multiplicador&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;multiplicar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;numero&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;multiplicador&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A função utiliza:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;multiplicador&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;mesmo que essa variável não tenha sido definida dentro dela.&lt;/p&gt;

&lt;p&gt;Agora utilizamos a função no Spark:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;multiplicador&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;

&lt;span class="n"&gt;numeros&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallelize&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;numeros&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;multiplicador&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Resultado esperado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[10, 20, 30, 40]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nesse caso, a função utiliza o valor necessário durante sua execução distribuída.&lt;/p&gt;

&lt;p&gt;O Spark precisa preparar e serializar a computação que será enviada aos workers.&lt;/p&gt;

&lt;p&gt;Isso funciona bem para valores pequenos e serializáveis.&lt;/p&gt;

&lt;p&gt;Mas existem consequências importantes.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Serialização de Closures
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;configuracao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;taxa&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.15&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;moeda&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;BRL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;numeros&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;configuracao&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;taxa&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O worker precisa ter acesso aos valores necessários para executar a função.&lt;/p&gt;

&lt;p&gt;Em PySpark, a serialização permite transportar funções e determinados objetos Python entre processos.&lt;/p&gt;

&lt;p&gt;Mas nem todo objeto pode ser serializado corretamente.&lt;/p&gt;

&lt;p&gt;Exemplos problemáticos incluem objetos que dependem de recursos locais, como determinadas conexões abertas, locks ou handles de arquivos.&lt;/p&gt;

&lt;p&gt;Outro problema aparece quando uma closure captura acidentalmente uma estrutura muito grande.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. O problema de capturar objetos gigantes
&lt;/h2&gt;

&lt;p&gt;Imagine uma estrutura de referência:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;referencias&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Categoria A&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Categoria B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Categoria C&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Com poucos registros, não há grande preocupação.&lt;/p&gt;

&lt;p&gt;Agora imagine que &lt;code&gt;referencias&lt;/code&gt; contenha milhões de entradas.&lt;/p&gt;

&lt;p&gt;Se essa estrutura for capturada por uma função enviada para várias Tasks, poderemos gerar custos relevantes de serialização e distribuição.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  DRIVER
                     |
              Large Dictionary
                     |
                 Closure
                     |
             Task Serialization
                     |
          +----------+----------+
          |          |          |
          v          v          v
        Task 1     Task 2     Task 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Precisamos de um mecanismo mais apropriado quando muitos workers devem consultar os mesmos dados de referência.&lt;/p&gt;

&lt;p&gt;É aqui que entram as &lt;strong&gt;Broadcast Variables&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. O que é uma Broadcast Variable?
&lt;/h2&gt;

&lt;p&gt;Uma Broadcast Variable permite disponibilizar um valor somente para leitura aos Executors de maneira eficiente.&lt;/p&gt;

&lt;p&gt;Em vez de incluir repetidamente uma estrutura grande nas closures, podemos criar uma variável broadcast.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;categorias&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Eletrônicos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Livros&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Alimentos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;categorias_broadcast&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;categorias&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora temos um objeto broadcast.&lt;/p&gt;

&lt;p&gt;Para acessar seu conteúdo dentro de uma Task:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;categorias_broadcast&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;produtos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallelize&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;102&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;103&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;produtos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;produto&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;produto&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="n"&gt;categorias_broadcast&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;produto&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Desconhecida&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Resultado esperado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[
    (101, 'Eletrônicos'),
    (102, 'Livros'),
    (103, 'Alimentos')
]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora as Tasks consultam uma referência distribuída pelo mecanismo de broadcast.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Como Broadcast funciona conceitualmente?
&lt;/h2&gt;

&lt;p&gt;Imagine uma tabela de referência que várias Tasks precisam utilizar.&lt;/p&gt;

&lt;p&gt;Sem broadcast explícito:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                   DRIVER
                      |
              Reference Data
                      |
             Task Serialization
                      |
          +-----------+-----------+
          |           |           |
          v           v           v
        Task A      Task B      Task C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Com Broadcast:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                    DRIVER
                       |
                Broadcast Data
                       |
           +-----------+-----------+
           |           |           |
           v           v           v
       Executor 1  Executor 2  Executor 3
           |           |           |
       Shared local  Shared local Shared local
       broadcast     broadcast    broadcast
           |           |           |
         Tasks       Tasks       Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Spark utiliza mecanismos próprios de distribuição e armazenamento para disponibilizar esses dados aos workers.&lt;/p&gt;

&lt;p&gt;O objetivo é reduzir o custo de transportar repetidamente a mesma referência.&lt;/p&gt;

&lt;p&gt;Entretanto, &lt;strong&gt;broadcast não significa memória gratuita&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Os valores precisam caber nos recursos disponíveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Broadcast Variables são somente para leitura
&lt;/h2&gt;

&lt;p&gt;Uma regra importante:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Broadcast Variables devem ser tratadas como valores imutáveis distribuídos.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Não devemos utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;categorias_broadcast&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Nova categoria&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;como mecanismo para atualizar uma referência global.&lt;/p&gt;

&lt;p&gt;Modificar uma cópia local não propaga uma atualização consistente para os demais Executors.&lt;/p&gt;

&lt;p&gt;Se a referência mudar, a aplicação deve distribuir uma nova versão conforme a necessidade.&lt;/p&gt;

&lt;p&gt;Podemos pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Broadcast Version 1
        |
        v
Distributed Read-Only Reference
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Updated Data
     |
     v
New Broadcast Version
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse modelo evita expectativas incorretas sobre sincronização.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Quando usar Broadcast Variables?
&lt;/h2&gt;

&lt;p&gt;Broadcast Variables são úteis quando:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Existe uma referência relativamente pequena.&lt;/li&gt;
&lt;li&gt;Muitas Tasks precisam consultar os mesmos dados.&lt;/li&gt;
&lt;li&gt;O conteúdo não precisa ser modificado durante a execução.&lt;/li&gt;
&lt;li&gt;O custo de distribuir repetidamente a referência seria relevante.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Exemplos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Dicionários de classificação.&lt;/li&gt;
&lt;li&gt;Parâmetros de algoritmos.&lt;/li&gt;
&lt;li&gt;Regras estáticas de processamento.&lt;/li&gt;
&lt;li&gt;Tabelas pequenas de correspondência.&lt;/li&gt;
&lt;li&gt;Configurações utilizadas em transformações distribuídas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mas existe uma distinção importante.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Broadcast Variable não é exatamente Broadcast Join
&lt;/h2&gt;

&lt;p&gt;Na Parte 5, utilizamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;broadcast&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pedidos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clientes&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora utilizamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;referencia&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dicionario&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essas operações compartilham o conceito de distribuir dados para evitar movimentação desnecessária.&lt;/p&gt;

&lt;p&gt;Mas pertencem a mecanismos e abstrações diferentes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Broadcast Variable
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;É um mecanismo para disponibilizar valores somente para leitura ao código executado nos workers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Broadcast Join
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dataframe&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;É uma indicação ao planejador do Spark SQL para considerar uma estratégia de join baseada em broadcast.&lt;/p&gt;

&lt;p&gt;O Spark SQL pode executar um Broadcast Hash Join quando as condições e a semântica permitirem.&lt;/p&gt;

&lt;p&gt;Não devemos tratar os dois recursos como APIs intercambiáveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. Quando preferir um Join em vez de um dicionário Broadcast?
&lt;/h2&gt;

&lt;p&gt;Imagine que temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PEDIDOS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CLIENTES
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ambos como DataFrames.&lt;/p&gt;

&lt;p&gt;Poderíamos transformar &lt;code&gt;CLIENTES&lt;/code&gt; em um dicionário Python e utilizar uma Broadcast Variable.&lt;/p&gt;

&lt;p&gt;Mas isso nem sempre é uma boa escolha.&lt;/p&gt;

&lt;p&gt;Se o problema é um join relacional, geralmente é melhor permitir que o Spark SQL trabalhe com seus mecanismos nativos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pedidos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;clientes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou, quando apropriado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pedidos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clientes&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Assim preservamos oportunidades de otimização pelo Catalyst e pelo planejador físico.&lt;/p&gt;

&lt;p&gt;Uma boa regra é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Para relacionamentos entre DataFrames, comece pela API nativa de join. Para referências Python utilizadas em funções distribuídas, avalie Broadcast Variables.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  12. E quando precisamos acumular resultados?
&lt;/h2&gt;

&lt;p&gt;Agora vamos ao problema oposto.&lt;/p&gt;

&lt;p&gt;Broadcast Variables distribuem valores para leitura.&lt;/p&gt;

&lt;p&gt;Mas imagine que queremos contar eventos observados pelas Tasks.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Quantos registros inválidos foram encontrados?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Talvez desejemos produzir:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor 1 → 12 inválidos
Executor 2 → 8 inválidos
Executor 3 → 15 inválidos
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E obter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Total = 35
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Para determinados casos, Spark disponibiliza outro mecanismo:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Accumulators.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  13. O que é um Accumulator?
&lt;/h2&gt;

&lt;p&gt;Um Accumulator permite que Tasks contribuam para um valor agregado que pode ser consultado pelo Driver.&lt;/p&gt;

&lt;p&gt;Em PySpark:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;invalidos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;accumulator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora podemos incrementá-lo durante uma computação distribuída.&lt;/p&gt;

&lt;p&gt;Exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;dados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallelize&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="mi"&gt;30&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;invalidos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;accumulator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;validar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;numero&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;invalidos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt;

&lt;span class="n"&gt;dados&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;validar&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;invalidos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em uma execução simples, o resultado esperado é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas precisamos entender o comportamento antes de utilizar Accumulators em sistemas reais.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. Como Accumulators funcionam conceitualmente?
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  DRIVER
                     |
              Accumulator
                     |
          +----------+----------+
          |          |          |
          v          v          v
       Executor 1 Executor 2 Executor 3
          |          |          |
       add(10)     add(5)     add(8)
          |          |          |
          +----------+----------+
                     |
                     v
                 DRIVER
                     |
                     v
                    23
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As Tasks contribuem com atualizações.&lt;/p&gt;

&lt;p&gt;O Driver consulta o valor agregado.&lt;/p&gt;

&lt;p&gt;Diferentemente de uma variável Python comum, o Spark possui um mecanismo próprio para essas contribuições.&lt;/p&gt;

&lt;p&gt;Mas isso não transforma o Accumulator em uma variável global de leitura e escrita livre.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Broadcast vs. Accumulator
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Característica&lt;/th&gt;
&lt;th&gt;Broadcast Variable&lt;/th&gt;
&lt;th&gt;Accumulator&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Objetivo&lt;/td&gt;
&lt;td&gt;Distribuir dados de referência&lt;/td&gt;
&lt;td&gt;Agregar contribuições&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fluxo conceitual&lt;/td&gt;
&lt;td&gt;Driver → Executors&lt;/td&gt;
&lt;td&gt;Tasks → Driver&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Leitura pelas Tasks&lt;/td&gt;
&lt;td&gt;Sim&lt;/td&gt;
&lt;td&gt;Não como estado global&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Atualização pelas Tasks&lt;/td&gt;
&lt;td&gt;Não deve ocorrer&lt;/td&gt;
&lt;td&gt;Sim, via &lt;code&gt;add()&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Uso típico&lt;/td&gt;
&lt;td&gt;Lookup e parâmetros&lt;/td&gt;
&lt;td&gt;Contadores e métricas&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Substitui DataFrame?&lt;/td&gt;
&lt;td&gt;Não&lt;/td&gt;
&lt;td&gt;Não&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Uma forma simples de memorizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;BROADCAST

Driver
   |
   v
Executors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ACCUMULATOR

Tasks
   |
   v
Driver
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa distinção é o coração do capítulo.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. Accumulators não devem controlar a lógica do pipeline
&lt;/h2&gt;

&lt;p&gt;Imagine uma regra:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Se contador global ultrapassar 1000,
interrompa o processamento.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Parece simples.&lt;/p&gt;

&lt;p&gt;Mas um Accumulator não fornece às Tasks uma visão global sincronizada e atualizada em tempo real.&lt;/p&gt;

&lt;p&gt;Portanto, não devemos construir lógica distribuída dependente da leitura de um Accumulator como se fosse uma variável compartilhada convencional.&lt;/p&gt;

&lt;p&gt;Seu uso é mais apropriado para observabilidade e agregações auxiliares.&lt;/p&gt;

&lt;p&gt;Exemplos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Registros inválidos
Eventos descartados
Erros de parsing
Ocorrências de determinado padrão
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mesmo nesses casos, precisamos compreender retries e recomputações.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. O problema dos Side Effects
&lt;/h2&gt;

&lt;p&gt;Agora entramos em um assunto extremamente importante.&lt;/p&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;processar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;registro&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;enviar_para_api&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;registro&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;registro&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;processar&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Parece razoável.&lt;/p&gt;

&lt;p&gt;Mas existe um problema:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tasks podem ser executadas novamente.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Lembre-se da Parte 8:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task
 |
 v
Failure
 |
 v
Retry
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se a função executa uma operação externa com efeito permanente, uma nova tentativa pode repetir esse efeito.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task Attempt 1
    |
    v
POST /pagamento
    |
    v
Sucesso externo
    |
    v
Falha antes da conclusão da Task
    |
    v
Retry
    |
    v
POST /pagamento novamente
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora temos uma duplicação potencialmente grave.&lt;/p&gt;

&lt;p&gt;A tolerância a falhas do Spark não garante automaticamente que efeitos externos aconteçam exatamente uma vez.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. Retries e Accumulators
&lt;/h2&gt;

&lt;p&gt;Accumulators também precisam ser interpretados com cuidado.&lt;/p&gt;

&lt;p&gt;A semântica das atualizações depende do contexto de execução.&lt;/p&gt;

&lt;p&gt;Em particular, atualizações realizadas dentro de transformações podem produzir resultados surpreendentes quando existem novas Actions, recomputações ou tentativas de execução.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;contador&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;accumulator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;rdd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallelize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;contar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;numero&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;contador&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;numero&lt;/span&gt;

&lt;span class="n"&gt;transformado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;contar&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;transformado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;transformado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;podem executar a transformação novamente.&lt;/p&gt;

&lt;p&gt;Portanto, o contador pode acumular contribuições de mais de uma Action.&lt;/p&gt;

&lt;p&gt;Isso não significa que existam mais registros únicos na fonte.&lt;/p&gt;

&lt;p&gt;Significa que estamos contando execuções de código.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. Contar registros é diferente de contar execuções
&lt;/h2&gt;

&lt;p&gt;Essa diferença merece destaque.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Dataset:
100 registros
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Executamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Action 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Action 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se uma transformação que atualiza o Accumulator for executada nas duas Actions, as contribuições podem ocorrer novamente.&lt;/p&gt;

&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Número de registros de negócio
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;não é necessariamente igual a:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Número de vezes que uma função foi executada
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa distinção é essencial em sistemas com lazy evaluation e tolerância a falhas.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Para métricas exatas, prefira operações determinísticas
&lt;/h2&gt;

&lt;p&gt;Se precisamos contar registros inválidos com precisão como resultado de negócio, podemos usar as APIs nativas do Spark.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;

&lt;span class="n"&gt;invalidos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora o resultado vem de uma operação distribuída que calcula explicitamente a quantidade de linhas que satisfazem a condição.&lt;/p&gt;

&lt;p&gt;Para múltiplas métricas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;sum&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;spark_sum&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;when&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;metricas&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;agg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;total_registros&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;spark_sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;otherwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registros_invalidos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;metricas&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa abordagem é mais adequada quando o número calculado faz parte da lógica de negócio ou de uma validação de qualidade.&lt;/p&gt;

&lt;p&gt;Accumulators continuam úteis, mas não deveriam substituir automaticamente agregações distribuídas.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. Closures podem carregar estado inesperado
&lt;/h2&gt;

&lt;p&gt;Considere uma classe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Processador&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;configuracao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;taxa&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.1&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;transformar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;valor&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;configuracao&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;taxa&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;processador&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Processador&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;processador&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;transformar&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Dependendo do objeto e de suas referências, a serialização pode incluir mais estado do que imaginamos.&lt;/p&gt;

&lt;p&gt;Em aplicações maiores, isso pode gerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Serialização custosa.&lt;/li&gt;
&lt;li&gt;Objetos não serializáveis.&lt;/li&gt;
&lt;li&gt;Closures grandes.&lt;/li&gt;
&lt;li&gt;Dependências desnecessárias.&lt;/li&gt;
&lt;li&gt;Problemas difíceis de diagnosticar.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma alternativa é capturar apenas o valor necessário:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;taxa&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;processador&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;configuracao&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;taxa&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;valor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;valor&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;taxa&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou, se a referência for grande e reutilizada:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;config_broadcast&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;processador&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;configuracao&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A escolha depende do volume e do padrão de utilização.&lt;/p&gt;




&lt;h2&gt;
  
  
  22. Evite criar conexões externas no Driver para enviá-las às Tasks
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;conexao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;criar_conexao_banco&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;registro&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;conexao&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;executar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;registro&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso é problemático.&lt;/p&gt;

&lt;p&gt;Conexões abertas geralmente estão associadas ao processo, à rede e ao runtime onde foram criadas.&lt;/p&gt;

&lt;p&gt;Elas não deveriam ser tratadas como objetos Python comuns que podemos transportar livremente para outros processos.&lt;/p&gt;

&lt;p&gt;Uma estratégia mais apropriada, quando realmente precisamos de integração externa por Tasks, pode envolver inicialização controlada por partição.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;processar_particao&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;registros&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;conexao&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;criar_conexao&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;registro&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;registros&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="nf"&gt;processar_com_conexao&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="n"&gt;conexao&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;registro&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;conexao&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mapPartitions&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;processar_particao&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas existe uma ressalva fundamental: operações externas precisam considerar idempotência, retries, limites de conexão e tratamento de falhas.&lt;/p&gt;

&lt;p&gt;Em pipelines de produção, conectores nativos e mecanismos de escrita transacional geralmente são preferíveis quando disponíveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. Por que &lt;code&gt;mapPartitions()&lt;/code&gt; pode ajudar?
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Partition 1
    |
    +-- 100.000 registros
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se criarmos uma conexão por registro:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100.000 conexões
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;podemos gerar overhead enorme.&lt;/p&gt;

&lt;p&gt;Com processamento por partição, podemos reutilizar uma conexão dentro da execução daquela partição.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task
 |
 v
Open Connection
 |
 +-- Record 1
 +-- Record 2
 +-- Record 3
 +-- ...
 |
 v
Close Connection
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode reduzir overhead.&lt;/p&gt;

&lt;p&gt;Mas não resolve automaticamente o problema de duplicação causado por retries.&lt;/p&gt;

&lt;p&gt;O sistema externo ainda precisa estar preparado.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Idempotência é essencial
&lt;/h2&gt;

&lt;p&gt;Uma operação é idempotente quando repeti-la não altera o resultado final além da primeira aplicação bem-sucedida.&lt;/p&gt;

&lt;p&gt;Por exemplo, uma gravação baseada em chave única com semântica apropriada de upsert pode ser projetada para tolerar repetição.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task Attempt 1
       |
       v
UPSERT customer_id=123
       |
       v
Retry
       |
       v
UPSERT customer_id=123
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se a operação for corretamente idempotente, a repetição não deve gerar um registro duplicado.&lt;/p&gt;

&lt;p&gt;Agora compare com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;INSERT sem chave de deduplicação
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Repetições podem gerar múltiplos registros.&lt;/p&gt;

&lt;p&gt;Essa preocupação não é específica de Spark.&lt;/p&gt;

&lt;p&gt;É um princípio geral de sistemas distribuídos.&lt;/p&gt;




&lt;h2&gt;
  
  
  25. Broadcast não substitui gerenciamento de estado
&lt;/h2&gt;

&lt;p&gt;Imagine uma referência que muda continuamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Preço atual
Estoque atual
Status do pedido
Saldo financeiro
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se os dados precisam ser atualizados em tempo real e compartilhados consistentemente entre workers, uma Broadcast Variable não é um banco de dados distribuído.&lt;/p&gt;

&lt;p&gt;Ela representa uma referência distribuída somente para leitura.&lt;/p&gt;

&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Broadcast
    ≠
Distributed Mutable Database
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se precisamos de estado mutável com garantias específicas, devemos escolher uma arquitetura apropriada.&lt;/p&gt;

&lt;p&gt;Isso pode envolver:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Sistemas externos de armazenamento.&lt;/li&gt;
&lt;li&gt;Operações transacionais.&lt;/li&gt;
&lt;li&gt;Processamento estruturado de estado.&lt;/li&gt;
&lt;li&gt;Estratégias de consistência.&lt;/li&gt;
&lt;li&gt;Mecanismos específicos de streaming.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O importante é não atribuir a Broadcast Variables capacidades que elas não possuem.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. Accumulator não substitui agregação distribuída
&lt;/h2&gt;

&lt;p&gt;Da mesma forma:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Accumulator
    ≠
Distributed Database
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E também:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Accumulator
    ≠
Reliable Business Counter
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;em todos os contextos.&lt;/p&gt;

&lt;p&gt;Para contar linhas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Para somar valores:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;agg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sum&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Para agrupar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;categoria&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essas operações representam resultados de dados.&lt;/p&gt;

&lt;p&gt;Accumulators representam um mecanismo auxiliar de agregação de contribuições das Tasks.&lt;/p&gt;

&lt;p&gt;A diferença é sutil, mas importante.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Um exemplo prático: validação de dados
&lt;/h2&gt;

&lt;p&gt;Imagine uma ingestão de eventos.&lt;/p&gt;

&lt;p&gt;Cada evento possui:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;event_id
customer_id
event_type
value
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Queremos identificar registros inválidos.&lt;/p&gt;

&lt;p&gt;Podemos utilizar Spark SQL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;when&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;sum&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;spark_sum&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;validado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;withColumn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registro_valido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNotNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isNotNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;metricas&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;validado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;agg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;total&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;spark_sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registro_valido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;otherwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;validos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;spark_sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nf"&gt;when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="nf"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registro_valido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;otherwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;alias&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;invalidos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;metricas&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa solução é declarativa.&lt;/p&gt;

&lt;p&gt;O Spark entende as expressões.&lt;/p&gt;

&lt;p&gt;O resultado pode ser utilizado como métrica de qualidade de dados.&lt;/p&gt;

&lt;p&gt;Não precisamos de um Accumulator para essa contagem.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Um exemplo prático: classificação por referência
&lt;/h2&gt;

&lt;p&gt;Agora imagine que precisamos aplicar uma pequena referência Python a registros de um RDD.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;classificacoes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Alta prioridade&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Média prioridade&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;C&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Baixa prioridade&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Criamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;classificacoes_bc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;classificacoes&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;eventos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallelize&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;EVT001&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;EVT002&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;C&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;EVT003&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;eventos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;evento&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;evento&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="n"&gt;classificacoes_bc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;evento&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Não classificado&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Resultado:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[
    ('EVT001', 'Alta prioridade'),
    ('EVT002', 'Baixa prioridade'),
    ('EVT003', 'Média prioridade')
]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse é um caso simples em que a Broadcast Variable representa bem a intenção.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. Um exemplo prático: observabilidade com Accumulator
&lt;/h2&gt;

&lt;p&gt;Imagine que estamos estudando um parser de linhas.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;linhas&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallelize&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;10&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;20&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;erro&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;30&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;invalido&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Criamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;falhas_parse&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;accumulator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;converter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;linha&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;linha&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;falhas_parse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Executamos uma Action:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;linhas&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;converter&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resultado&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;falhas_parse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em uma execução simples:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[10, 20, None, 30, None]
2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso demonstra o mecanismo.&lt;/p&gt;

&lt;p&gt;Mas, em produção, eu não utilizaria automaticamente o valor do Accumulator como uma métrica de negócio com garantia de contagem exata.&lt;/p&gt;

&lt;p&gt;Para isso, preferiria estruturar os resultados e agregá-los explicitamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. O impacto da Lazy Evaluation
&lt;/h2&gt;

&lt;p&gt;Vamos conectar este capítulo aos anteriores.&lt;/p&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;contador&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;accumulator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;transformado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;registrar_e_transformar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;contador&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Até aqui:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Transformation definida
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas ainda não necessariamente executada.&lt;/p&gt;

&lt;p&gt;Quando fazemos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;transformado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;a computação acontece.&lt;/p&gt;

&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;transformado&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ela pode acontecer novamente.&lt;/p&gt;

&lt;p&gt;Isso significa que side effects dentro de transformações precisam ser tratados com muito cuidado.&lt;/p&gt;

&lt;p&gt;A combinação:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Lazy Evaluation
       +
Retries
       +
Recomputation
       +
Side Effects
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;é uma das fontes clássicas de comportamento inesperado em sistemas distribuídos.&lt;/p&gt;




&lt;h2&gt;
  
  
  31. A relação com Fault Tolerance
&lt;/h2&gt;

&lt;p&gt;Na Parte 8, vimos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task Failed
    |
    v
Retry
    |
    v
Recomputation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora entendemos uma consequência importante.&lt;/p&gt;

&lt;p&gt;Se a Task executa apenas transformações determinísticas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;recomputar é relativamente simples.&lt;/p&gt;

&lt;p&gt;Mas se a Task executa:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nf"&gt;enviar_email&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nf"&gt;cobrar_pagamento&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nf"&gt;inserir_registro_externo&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;recomputar pode repetir efeitos no mundo externo.&lt;/p&gt;

&lt;p&gt;Portanto:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A capacidade de recomputar dados não equivale à garantia de executar efeitos externos exatamente uma vez.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Essa distinção é fundamental para projetar pipelines confiáveis.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Variáveis compartilhadas e performance
&lt;/h2&gt;

&lt;p&gt;Broadcast Variables também estão relacionadas a performance.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1000 Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e uma referência Python relativamente grande.&lt;/p&gt;

&lt;p&gt;Se cada Task precisar receber repetidamente essa estrutura, podemos gerar overhead.&lt;/p&gt;

&lt;p&gt;Com Broadcast, Spark pode distribuir e reutilizar a referência de maneira mais eficiente.&lt;/p&gt;

&lt;p&gt;Mas isso precisa ser equilibrado com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Serialization
Distribution
Executor Memory
Object Size
Reuse
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Broadcast é uma ferramenta de otimização.&lt;/p&gt;

&lt;p&gt;Não uma obrigação para toda variável.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. O que não devemos fazer
&lt;/h2&gt;

&lt;p&gt;Alguns padrões merecem atenção especial.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Modificar variáveis Python locais esperando que o Driver enxergue as alterações:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;contador&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;funcao&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;global&lt;/span&gt; &lt;span class="n"&gt;contador&lt;/span&gt;
    &lt;span class="n"&gt;contador&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Utilizar Broadcast Variable como estado mutável global:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;bc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;contador&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Utilizar Accumulator para controlar decisões distribuídas em tempo real.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Executar operações externas não idempotentes dentro de Tasks sem estratégia de recuperação.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Capturar objetos enormes ou não serializáveis em closures.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Forçar Broadcast Join sem avaliar o tamanho real do dataset.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Todos esses padrões podem produzir problemas de correção, performance ou estabilidade.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Um modelo mental para escolher o mecanismo
&lt;/h2&gt;

&lt;p&gt;Quando preciso compartilhar informações em Spark, começo perguntando:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual é o objetivo?
        |
        +-----------------------------+
        |                             |
        v                             v
Fornecer referência             Produzir resultado
aos workers                     agregado
        |                             |
        v                             v
É somente leitura?           É dado de negócio?
        |                             |
        v                             v
Broadcast Variable?         DataFrame Aggregation?
                                      |
                                      v
                              Métrica auxiliar?
                                      |
                                      v
                                 Accumulator?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E, se a necessidade envolver:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Estado mutável consistente
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Efeitos externos
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;precisamos de uma arquitetura apropriada, não apenas de uma variável compartilhada.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. O que eu procuro em uma revisão de código
&lt;/h2&gt;

&lt;p&gt;Quando reviso um pipeline PySpark, gosto de investigar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A closure captura objetos desnecessários?&lt;/li&gt;
&lt;li&gt;Algum objeto não serializável está sendo enviado aos workers?&lt;/li&gt;
&lt;li&gt;Uma referência grande é distribuída repetidamente?&lt;/li&gt;
&lt;li&gt;Broadcast seria apropriado?&lt;/li&gt;
&lt;li&gt;Existe um join nativo mais adequado?&lt;/li&gt;
&lt;li&gt;Alguma Task modifica estado local esperando atualização no Driver?&lt;/li&gt;
&lt;li&gt;Accumulators estão sendo usados como métricas de negócio?&lt;/li&gt;
&lt;li&gt;Existem side effects?&lt;/li&gt;
&lt;li&gt;A operação externa é idempotente?&lt;/li&gt;
&lt;li&gt;O código suporta retries?&lt;/li&gt;
&lt;li&gt;O comportamento permanece correto após recomputação?&lt;/li&gt;
&lt;li&gt;O resultado é determinístico?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Essas perguntas vão muito além de saber utilizar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Elas fazem parte da engenharia de sistemas distribuídos.&lt;/p&gt;




&lt;h2&gt;
  
  
  36. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;Em uma aplicação local, podemos pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Uma variável
      |
      v
Um processo
      |
      v
Uma memória
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em Spark:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver
  |
  +--------------------------+
  |                          |
  v                          v
Executor 1                Executor 2
  |                          |
Python Worker             Python Worker
  |                          |
Memória local             Memória local
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso exige mecanismos explícitos para determinados padrões de comunicação.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Broadcast Variables&lt;/strong&gt; ajudam a distribuir referências somente para leitura.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Accumulators&lt;/strong&gt; permitem contribuições agregadas das Tasks para o Driver.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Closures&lt;/strong&gt; transportam funções e valores necessários para execução.&lt;/p&gt;

&lt;p&gt;E a &lt;strong&gt;tolerância a falhas&lt;/strong&gt; significa que qualquer efeito colateral precisa ser pensado considerando possíveis reexecuções.&lt;/p&gt;




&lt;h2&gt;
  
  
  37. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando vejo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;minha_funcao&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;não penso apenas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Aplicar função a cada elemento
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Também penso:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Onde a função será executada?
          |
          v
O que será serializado?
          |
          v
Quais variáveis ela captura?
          |
          v
Existe estado compartilhado?
          |
          v
Existe side effect?
          |
          v
O que acontece em um retry?
          |
          v
O resultado continua correto?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa mudança de perspectiva é uma das partes mais importantes de aprender computação distribuída.&lt;/p&gt;

&lt;p&gt;Porque o desafio não é apenas executar código em várias máquinas.&lt;/p&gt;

&lt;p&gt;É garantir que o resultado continue correto quando o sistema se comporta como um sistema distribuído de verdade.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora entendemos como funções, variáveis e estado se comportam quando atravessam processos.&lt;/p&gt;

&lt;p&gt;No próximo artigo vamos entrar em outro tema importante:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Modos de execução e implantação do Apache Spark.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Vamos explorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Client Mode.&lt;/li&gt;
&lt;li&gt;Cluster Mode.&lt;/li&gt;
&lt;li&gt;Onde o Driver executa.&lt;/li&gt;
&lt;li&gt;Onde os Executors executam.&lt;/li&gt;
&lt;li&gt;Cluster Managers.&lt;/li&gt;
&lt;li&gt;Spark Standalone.&lt;/li&gt;
&lt;li&gt;YARN.&lt;/li&gt;
&lt;li&gt;Kubernetes.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;spark-submit&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Diferenças entre desenvolvimento local e produção.&lt;/li&gt;
&lt;li&gt;Logs e observabilidade.&lt;/li&gt;
&lt;li&gt;Falhas de Driver e Executors.&lt;/li&gt;
&lt;li&gt;Boas práticas de implantação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pergunta central será:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Quando submetemos uma aplicação Spark, onde cada componente realmente está executando?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;p&gt;&lt;strong&gt;Série:&lt;/strong&gt; Como aprendi Apache Spark&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;← Parte 8:&lt;/strong&gt; RDD Lineage e tolerância a falhas — como o Spark se recupera&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 10 — Client Mode vs. Cluster Mode: onde o Spark realmente executa&lt;/p&gt;

</description>
      <category>bigdata</category>
      <category>pyspark</category>
      <category>dataengineering</category>
      <category>apache</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 8: RDD Lineage e tolerância a falhas — como o Spark se recupera</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Fri, 06 Nov 2026 19:23:43 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-8-rdd-lineage-e-tolerancia-a-falhas-como-o-spark-se-recupera-4c2i</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-8-rdd-lineage-e-tolerancia-a-falhas-como-o-spark-se-recupera-4c2i</guid>
      <description>&lt;p&gt;Nos primeiros artigos desta série, começamos com uma das abstrações que ajudaram a definir o Apache Spark:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;RDD — Resilient Distributed Dataset.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Falamos bastante sobre:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Distributed Dataset
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Vimos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Partitions
Tasks
Executors
Shuffle
Memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas existe uma palavra no nome que merece um capítulo próprio:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Resilient.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Por que um RDD é resiliente?&lt;/p&gt;

&lt;p&gt;A resposta está em uma das ideias mais elegantes da arquitetura original do Spark:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Lineage.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Em vez de depender apenas da replicação de cada resultado intermediário, Spark mantém informações sobre &lt;strong&gt;como os dados foram produzidos&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Se parte deles for perdida, em muitos casos ela pode ser reconstruída.&lt;/p&gt;

&lt;p&gt;Essa ideia conecta:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Immutability
    ↓
Transformations
    ↓
Lineage
    ↓
DAG
    ↓
Recomputation
    ↓
Fault Tolerance
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E muda a forma como enxergamos falhas em processamento distribuído.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Falhas fazem parte de sistemas distribuídos
&lt;/h2&gt;

&lt;p&gt;Imagine um cluster:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;             DRIVER
                │
      ┌─────────┼─────────┐
      ▼         ▼         ▼
  Executor 1 Executor 2 Executor 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora imagine que estamos processando:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10 TB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;durante horas.&lt;/p&gt;

&lt;p&gt;Nesse período, alguma coisa pode acontecer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor failure
Machine failure
Process crash
Network problem
Disk problem
Container termination
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em um sistema distribuído, a pergunta não deveria ser apenas:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;E se alguma coisa falhar?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Uma pergunta melhor é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Como o sistema continua quando alguma coisa falhar?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Essa diferença é fundamental.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. Um problema simples
&lt;/h2&gt;

&lt;p&gt;Imagine um dataset dividido em quatro partições:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD A

P1
P2
P3
P4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Distribuídas assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor 1
├── P1
└── P2

Executor 2
├── P3
└── P4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora o Executor 2 desaparece.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor 1
├── P1
└── P2

Executor 2
└── X
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Perdemos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P3
P4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O que fazer?&lt;/p&gt;

&lt;p&gt;Uma possibilidade seria manter cópias de tudo.&lt;/p&gt;

&lt;p&gt;Outra possibilidade é saber &lt;strong&gt;como reconstruir o que foi perdido&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Essa segunda ideia é central no modelo de RDDs.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. RDDs são imutáveis
&lt;/h2&gt;

&lt;p&gt;Uma característica importante dos RDDs é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;RDDs são imutáveis.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Quando fazemos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;não estamos modificando &lt;code&gt;rdd1&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Estamos descrevendo outro RDD:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD1
 │
 │ map(x * 2)
 ▼
RDD2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd3&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD1
 │
 │ map
 ▼
RDD2
 │
 │ filter
 ▼
RDD3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cada transformação cria uma nova abstração derivada da anterior.&lt;/p&gt;

&lt;p&gt;Essa propriedade ajuda a tornar a linhagem previsível.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. O que é Lineage?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Lineage&lt;/strong&gt; representa a sequência de dependências e transformações utilizada para produzir um dataset.&lt;/p&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;textFile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/events&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;rdd2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;rdd3&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;purchase&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;rdd4&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd3&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;rdd5&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd4&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reduceByKey&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;textFile
   ↓
 RDD1
   ↓
  map
   ↓
 RDD2
   ↓
 filter
   ↓
 RDD3
   ↓
  map
   ↓
 RDD4
   ↓
reduceByKey
   ↓
 RDD5
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa cadeia descreve:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Como RDD5 foi derivado dos dados originais.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Isso é lineage.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Lineage não significa guardar todas as cópias
&lt;/h2&gt;

&lt;p&gt;Esse ponto é importante.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD1
 ↓
RDD2
 ↓
RDD3
 ↓
RDD4
 ↓
RDD5
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Não precisamos interpretar isso como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Memory

RDD1 complete
RDD2 complete
RDD3 complete
RDD4 complete
RDD5 complete
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Lineage não significa manter automaticamente todas as versões materializadas.&lt;/p&gt;

&lt;p&gt;O que Spark preserva é a informação necessária sobre as dependências entre os datasets.&lt;/p&gt;

&lt;p&gt;Podemos pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Data
+
Transformation Graph
=
Ability to recompute
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa diferença é essencial.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Lazy Evaluation e Lineage
&lt;/h2&gt;

&lt;p&gt;Agora podemos conectar outro conceito da série.&lt;/p&gt;

&lt;p&gt;Quando escrevemos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(...)&lt;/span&gt;
&lt;span class="n"&gt;rdd3&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(...)&lt;/span&gt;
&lt;span class="n"&gt;rdd4&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd3&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(...)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Spark não precisa executar imediatamente cada transformação.&lt;/p&gt;

&lt;p&gt;Ele constrói relações entre os RDDs.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;rdd1
  │
 map
  │
rdd2
  │
filter
  │
rdd3
  │
 map
  │
rdd4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Quando uma Action aparece:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd4&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Lineage
   ↓
Action
   ↓
Job
   ↓
Stages
   ↓
Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Lazy evaluation e lineage estão profundamente conectados.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. O DAG
&lt;/h2&gt;

&lt;p&gt;Essa cadeia de dependências não precisa ser apenas linear.&lt;/p&gt;

&lt;p&gt;Podemos ter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;          RDD A
         /     \
        /       \
       ▼         ▼
    RDD B      RDD C
       \         /
        \       /
         ▼     ▼
          RDD D
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso forma um:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Directed Acyclic Graph — DAG&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Grafo Direcionado Acíclico
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Direcionado porque existem relações de dependência.&lt;/p&gt;

&lt;p&gt;Acíclico porque a computação não forma ciclos arbitrários de dependência entre esses datasets.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source
  ↓
Transform
  ↓
Transform ─────┐
  │            │
  ▼            ▼
Branch A    Branch B
  │            │
  └──────┬─────┘
         ▼
       Result
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O DAG representa a estrutura da computação.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. DAG não é exatamente a mesma coisa que Stage
&lt;/h2&gt;

&lt;p&gt;Essa distinção é importante.&lt;/p&gt;

&lt;p&gt;Podemos ter uma sequência de transformações:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;map
 ↓
filter
 ↓
map
 ↓
reduceByKey
 ↓
map
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas isso não significa necessariamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;5 transformations
=
5 stages
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Como vimos anteriormente, dependências narrow podem ser agrupadas em um mesmo Stage.&lt;/p&gt;

&lt;p&gt;Shuffle frequentemente cria uma fronteira.&lt;/p&gt;

&lt;p&gt;Então podemos ter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;STAGE 1

map
 ↓
filter
 ↓
map
 ↓

====== SHUFFLE ======

STAGE 2

reduce
 ↓
map
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Lineage / DAG
      ↓
Stage boundaries
      ↓
Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;são conceitos relacionados, mas não idênticos.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Narrow Dependencies
&lt;/h2&gt;

&lt;p&gt;Vamos revisitar um conceito importante.&lt;/p&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rdd1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos ter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD1 P1 ───► RDD2 P1
RDD1 P2 ───► RDD2 P2
RDD1 P3 ───► RDD2 P3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cada partição filha depende de uma quantidade limitada de partições anteriores.&lt;/p&gt;

&lt;p&gt;Isso caracteriza uma dependência estreita:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Narrow Dependency.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Agora imagine que perdemos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD2 P2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Spark pode precisar reconstruir apenas o caminho relevante:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD1 P2
   ↓
 map
   ↓
RDD2 P2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa granularidade é extremamente importante para tolerância a falhas.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Recomputation
&lt;/h2&gt;

&lt;p&gt;Aqui chegamos ao coração da resiliência.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source

P1 P2 P3 P4
 │  │  │  │
 ▼  ▼  ▼  ▼

Map

P1' P2' P3' P4'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P3'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;é perdida.&lt;/p&gt;

&lt;p&gt;Spark conhece sua dependência:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source P3
    ↓
   map
    ↓
  P3'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Então pode reconstruí-la.&lt;/p&gt;

&lt;p&gt;Não precisa necessariamente recalcular:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P1'
P2'
P4'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A ideia é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Recompute what was lost from the lineage.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Essa é uma das razões para a palavra:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Resilient.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  11. Resiliência sem replicar tudo
&lt;/h2&gt;

&lt;p&gt;Uma abordagem possível para tolerância a falhas seria:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Resultado intermediário
       ↓
Copy 1
Copy 2
Copy 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas isso possui custos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Storage
Network
Replication
Coordination
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;RDDs introduziram uma ideia poderosa para datasets derivados deterministicamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Don't replicate every intermediate result
              ↓
Keep lineage
              ↓
Recompute lost partitions
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso não elimina a necessidade de persistência ou replicação em todos os contextos.&lt;/p&gt;

&lt;p&gt;Mas cria um mecanismo elegante para recuperar dados derivados.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. Um exemplo mais completo
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;textFile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/logs&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;parsed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parse_line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;errors&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;parsed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ERROR&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;by_service&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;service&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;counts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;by_service&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reduceByKey&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TEXT FILE
    ↓
   raw
    ↓
  parse
    ↓
 parsed
    ↓
 filter
    ↓
 errors
    ↓
   map
    ↓
by_service
    ↓
reduceByKey
    ↓
 counts
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora imagine uma partição de &lt;code&gt;errors&lt;/code&gt; sendo perdida.&lt;/p&gt;

&lt;p&gt;Spark conhece:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;textFile
   ↓
 parse
   ↓
 filter
   ↓
errors partition
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Então pode reconstruir aquela partição a partir da origem e das transformações necessárias.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. E depois de um Shuffle?
&lt;/h2&gt;

&lt;p&gt;Agora a situação fica mais interessante.&lt;/p&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;counts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;by_service&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reduceByKey&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Como vimos na Parte 4:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Map-side work
      ↓
Shuffle Write
      ↓
Network
      ↓
Shuffle Read
      ↓
Reduce-side work
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Temos uma wide dependency.&lt;/p&gt;

&lt;p&gt;Agora uma partição posterior pode depender de dados produzidos por várias partições anteriores.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P1 ───┐
P2 ───┼──► Shuffle ───► R1
P3 ───┤
P4 ───┘
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se determinado resultado for perdido, a recomputação pode ser mais cara porque envolve dependências mais amplas.&lt;/p&gt;

&lt;p&gt;Isso mostra que:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Nem todo lineage possui o mesmo custo de recuperação.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  14. Narrow vs. Wide também importa para Fault Tolerance
&lt;/h2&gt;

&lt;p&gt;Compare:&lt;/p&gt;

&lt;h3&gt;
  
  
  Narrow
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P1 ───► P1'
P2 ───► P2'
P3 ───► P3'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Perdeu &lt;code&gt;P2'&lt;/code&gt;?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Recompute P2
     ↓
     P2'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora:&lt;/p&gt;

&lt;h3&gt;
  
  
  Wide
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P1 ───┐
P2 ───┼──► P'
P3 ───┤
P4 ───┘
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A dependência é mais ampla.&lt;/p&gt;

&lt;p&gt;Isso afeta:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;custo de recomputação;&lt;/li&gt;
&lt;li&gt;organização dos Stages;&lt;/li&gt;
&lt;li&gt;shuffle;&lt;/li&gt;
&lt;li&gt;recuperação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Mais uma vez, shuffle aparece como uma fronteira importante da arquitetura.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Então por que usar Cache?
&lt;/h2&gt;

&lt;p&gt;Se Spark consegue recomputar tudo, poderíamos perguntar:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Por que utilizar cache?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Porque recomputação também custa.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source
 ↓
Expensive Transformation A
 ↓
Expensive Transformation B
 ↓
Dataset X
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora utilizamos &lt;code&gt;Dataset X&lt;/code&gt; cinco vezes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;             Dataset X
            /    |    \
           ▼     ▼     ▼
         Job A Job B Job C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sem persistência, determinadas partes da linhagem podem precisar ser recomputadas.&lt;/p&gt;

&lt;p&gt;Se &lt;code&gt;Dataset X&lt;/code&gt; for caro de produzir, podemos decidir:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;dataset_x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora trocamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Repeated Computation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;por:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Storage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse é um trade-off.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. Cache vs. Recomputation
&lt;/h2&gt;

&lt;p&gt;Podemos pensar assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 Dataset
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
      Recompute             Cache
          │                   │
          ▼                   ▼
     More CPU             More Memory
     More Time            Storage Cost
          │                   │
          └─────────┬─────────┘
                    ▼
                 Trade-off
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A pergunta não é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Cache é bom?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A pergunta é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;O custo de manter esse dataset é menor do que o custo esperado de recomputá-lo?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Essa é uma pergunta muito mais útil.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Persistência e tolerância a falhas
&lt;/h2&gt;

&lt;p&gt;Agora existe uma nuance importante.&lt;/p&gt;

&lt;p&gt;Se persistimos um RDD e uma partição persistida é perdida devido à falha de um Executor, o lineage ainda pode permitir sua recomputação.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD X cached

Executor 1
├── P1
└── P2

Executor 2
├── P3
└── P4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Executor 2 falha:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P3 X
P4 X
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Spark ainda conhece:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Lineage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e pode reconstruir essas partições quando necessário.&lt;/p&gt;

&lt;p&gt;Então:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cache
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Lineage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;não são mecanismos concorrentes.&lt;/p&gt;

&lt;p&gt;Eles se complementam.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. &lt;code&gt;persist()&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Como vimos na Parte 7, podemos usar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;persist&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;para controlar estratégias de persistência.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Persistence
    │
    ├── Memory
    ├── Disk
    ├── Serialized forms
    └── Replication options
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A escolha depende do workload.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Dataset pequeno
+
reutilizado frequentemente
+
caro de recomputar
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;pode justificar uma estratégia diferente de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Dataset enorme
+
utilizado uma vez
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Persistência deve responder a uma necessidade real.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. Mas Lineage pode crescer
&lt;/h2&gt;

&lt;p&gt;Agora imagine um pipeline muito longo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD1
 ↓
RDD2
 ↓
RDD3
 ↓
RDD4
 ↓
RDD5
 ↓
RDD6
 ↓
...
 ↓
RDD100
 ↓
...
 ↓
RDD500
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Temos uma longa cadeia de dependências.&lt;/p&gt;

&lt;p&gt;Em determinados workloads iterativos ou pipelines extensos, essa linhagem pode se tornar indesejavelmente longa ou cara para recuperação.&lt;/p&gt;

&lt;p&gt;Queremos então, em alguns cenários, interromper essa dependência histórica.&lt;/p&gt;

&lt;p&gt;É aqui que aparece outro conceito:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Checkpoint.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  20. O que é Checkpoint?
&lt;/h2&gt;

&lt;p&gt;A ideia do checkpoint é materializar o estado de um dataset em armazenamento confiável e cortar sua dependência da linhagem anterior para fins de recuperação.&lt;/p&gt;

&lt;p&gt;Antes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source
 ↓
A
 ↓
B
 ↓
C
 ↓
D
 ↓
E
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se &lt;code&gt;E&lt;/code&gt; precisar ser reconstruído:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source → A → B → C → D → E
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora imagine um checkpoint em &lt;code&gt;C&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source
 ↓
A
 ↓
B
 ↓
C
 ↓
CHECKPOINT
 ↓
D
 ↓
E
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A recuperação posterior pode partir do checkpoint:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CHECKPOINT
    ↓
    D
    ↓
    E
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso trunca a linhagem anterior.&lt;/p&gt;




&lt;h2&gt;
  
  
  21. Configurando Checkpoint em RDD
&lt;/h2&gt;

&lt;p&gt;Em um contexto de RDD, podemos definir um diretório de checkpoint:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setCheckpointDir&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/path/to/reliable/checkpoints&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;checkpoint&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Quando o RDD for materializado por uma Action, o checkpoint poderá ser efetivamente produzido.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;checkpoint&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;checkpoint()
    ↓
mark RDD
    ↓
Action
    ↓
compute
    ↓
write checkpoint
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Assim como cache, existe interação com lazy evaluation.&lt;/p&gt;




&lt;h2&gt;
  
  
  22. Checkpoint não é Cache
&lt;/h2&gt;

&lt;p&gt;Essa distinção é fundamental.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cache
&lt;/h3&gt;

&lt;p&gt;Objetivo principal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Reuse
 ↓
Performance
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Checkpoint
&lt;/h3&gt;

&lt;p&gt;Objetivo principal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Materialize reliable state
        ↓
Truncate lineage
        ↓
Recovery / lineage management
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Simplificando:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CACHE
"Evite recomputar algo que vou reutilizar."
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;versus:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CHECKPOINT
"Crie um novo ponto materializado e corte a dependência da linhagem anterior."
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os dois podem aparecer juntos.&lt;/p&gt;

&lt;p&gt;Mas resolvem problemas diferentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. Cache não corta Lineage
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A
 ↓
B
 ↓
C
 ↓
D
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Fazemos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ainda existe conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A
 ↓
B
 ↓
C
 ↓
D [cached]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O lineage continua conhecido.&lt;/p&gt;

&lt;p&gt;Se a partição persistida for perdida, Spark pode usar essa linhagem para reconstruí-la.&lt;/p&gt;

&lt;p&gt;Agora com checkpoint:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A
 ↓
B
 ↓
C
 ↓
D
 ↓
CHECKPOINT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;a linhagem anterior pode ser truncada depois que o checkpoint é materializado corretamente.&lt;/p&gt;

&lt;p&gt;Essa é uma diferença arquitetural importante.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Checkpoint também tem custo
&lt;/h2&gt;

&lt;p&gt;Checkpoint não é gratuito.&lt;/p&gt;

&lt;p&gt;Precisamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Compute
   ↓
Write
   ↓
Reliable Storage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso envolve:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;I/O;&lt;/li&gt;
&lt;li&gt;armazenamento;&lt;/li&gt;
&lt;li&gt;tempo;&lt;/li&gt;
&lt;li&gt;gerenciamento.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Portanto, não queremos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;checkpoint everything
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Queremos checkpoint quando existe uma razão arquitetural.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;lineage muito longo;&lt;/li&gt;
&lt;li&gt;processamento iterativo;&lt;/li&gt;
&lt;li&gt;necessidade de ponto confiável;&lt;/li&gt;
&lt;li&gt;custo elevado de reconstrução.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  25. Um pipeline iterativo
&lt;/h2&gt;

&lt;p&gt;Imagine um algoritmo que executa repetidamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Iteration 1
    ↓
Iteration 2
    ↓
Iteration 3
    ↓
Iteration 4
    ↓
...
    ↓
Iteration 100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se cada iteração depende da anterior, podemos formar uma linhagem extensa.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Initial
 ↓
I1
 ↓
I2
 ↓
I3
 ↓
...
 ↓
I100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em determinadas situações, podemos estabelecer checkpoints intermediários:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Initial
 ↓
I1
 ↓
...
 ↓
I20
 ↓
CHECKPOINT
 ↓
I21
 ↓
...
 ↓
I40
 ↓
CHECKPOINT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Assim reduzimos a profundidade necessária para recuperação futura.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. Falha de Executor não significa falha da aplicação
&lt;/h2&gt;

&lt;p&gt;Agora podemos enxergar resiliência de forma melhor.&lt;/p&gt;

&lt;p&gt;Temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor 1
Executor 2
Executor 3
Executor 4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Um desaparece:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor 1
Executor 2
Executor 3 X
Executor 4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso não implica necessariamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;APPLICATION FAILED
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Spark pode reagendar trabalho e reconstruir dados necessários quando a situação permite.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor failure
      ↓
Lost Tasks / Data
      ↓
Scheduler
      ↓
Re-execution
      ↓
Lineage
      ↓
Recompute
      ↓
Continue
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa capacidade é central em computação distribuída.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Task Failure
&lt;/h2&gt;

&lt;p&gt;Também podemos ter falha em uma Task específica.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Stage 10

Task 1 ✓
Task 2 ✓
Task 3 X
Task 4 ✓
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Spark pode tentar executar novamente a Task.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task 3
  ↓
Retry
  ↓
Task 3 ✓
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Talvez em outro Executor.&lt;/p&gt;

&lt;p&gt;Isso ajuda com falhas transitórias.&lt;/p&gt;

&lt;p&gt;Mas existe uma distinção importante.&lt;/p&gt;




&lt;h2&gt;
  
  
  28. Falha transitória vs. falha determinística
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task failed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;porque uma máquina teve um problema temporário.&lt;/p&gt;

&lt;p&gt;Retry pode funcionar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Attempt 1 → FAIL
Attempt 2 → SUCCESS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora imagine código:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;transform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;x = 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A Task falha.&lt;/p&gt;

&lt;p&gt;Spark tenta novamente.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Attempt 1 → divide by zero
Attempt 2 → divide by zero
Attempt 3 → divide by zero
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Não importa quantas máquinas utilizarmos.&lt;/p&gt;

&lt;p&gt;O problema está nos dados ou no código.&lt;/p&gt;

&lt;p&gt;Isso nos ensina:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Fault tolerance não corrige lógica determinística incorreta.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  29. Recomputation também não corrige dados ruins
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;corrupted input
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A partição falha.&lt;/p&gt;

&lt;p&gt;Spark tenta novamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;read
 ↓
fail
 ↓
retry
 ↓
read same corrupted data
 ↓
fail
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Lineage ajuda a recuperar computação perdida.&lt;/p&gt;

&lt;p&gt;Não transforma entrada inválida em entrada válida.&lt;/p&gt;

&lt;p&gt;Resiliência tem limites.&lt;/p&gt;




&lt;h2&gt;
  
  
  30. O Driver continua sendo especial
&lt;/h2&gt;

&lt;p&gt;Até agora falamos muito sobre Executors.&lt;/p&gt;

&lt;p&gt;Mas existe um componente central:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DRIVER
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Driver coordena a aplicação.&lt;/p&gt;

&lt;p&gt;Se um Executor falha, Spark pode frequentemente reagendar trabalho.&lt;/p&gt;

&lt;p&gt;A falha do Driver possui implicações diferentes porque ali está o processo que coordena a aplicação.&lt;/p&gt;

&lt;p&gt;A capacidade de recuperar o Driver depende também:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;do modo de implantação;&lt;/li&gt;
&lt;li&gt;do cluster manager;&lt;/li&gt;
&lt;li&gt;da configuração;&lt;/li&gt;
&lt;li&gt;da plataforma utilizada;&lt;/li&gt;
&lt;li&gt;dos mecanismos externos de supervisão/restart.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Portanto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor fault tolerance
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;não deve ser confundida automaticamente com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver high availability
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;São problemas relacionados, mas diferentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  31. Lineage não substitui durabilidade da fonte
&lt;/h2&gt;

&lt;p&gt;Imagine que nosso RDD começa em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/data/events
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source
 ↓
Map
 ↓
Filter
 ↓
RDD
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se perdemos uma partição intermediária, podemos tentar reconstruí-la a partir da fonte.&lt;/p&gt;

&lt;p&gt;Mas e se a fonte original desaparecer?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source X
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora lineage sabe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;como
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;reconstruir.&lt;/p&gt;

&lt;p&gt;Mas não possui necessariamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;de onde
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;obter os dados originais.&lt;/p&gt;

&lt;p&gt;Portanto:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Lineage não substitui armazenamento durável dos dados de origem.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Essa distinção é muito importante em arquitetura de dados.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Fault tolerance não é Backup
&lt;/h2&gt;

&lt;p&gt;Outro erro conceitual seria pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark é fault tolerant
        ↓
não preciso de backup
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Não.&lt;/p&gt;

&lt;p&gt;São problemas diferentes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fault tolerance
&lt;/h3&gt;

&lt;p&gt;Ajuda o sistema a continuar ou recuperar processamento diante de determinadas falhas.&lt;/p&gt;

&lt;h3&gt;
  
  
  Backup
&lt;/h3&gt;

&lt;p&gt;Protege dados contra cenários como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;exclusão;&lt;/li&gt;
&lt;li&gt;corrupção;&lt;/li&gt;
&lt;li&gt;desastre;&lt;/li&gt;
&lt;li&gt;erro operacional;&lt;/li&gt;
&lt;li&gt;perda da fonte.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Podemos ter um engine extremamente resiliente processando uma fonte que foi permanentemente destruída.&lt;/p&gt;

&lt;p&gt;O engine não recria magicamente dados que deixaram de existir.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. Lineage e DataFrames
&lt;/h2&gt;

&lt;p&gt;Até agora usei RDDs porque lineage é um conceito especialmente natural para explicar a arquitetura original do Spark.&lt;/p&gt;

&lt;p&gt;Mas DataFrames também participam de uma computação baseada em dependências e planos.&lt;/p&gt;

&lt;p&gt;Quando escrevemos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;vendas&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ano = 2026&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;continuamos tendo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source
 ↓
Transformations
 ↓
Dependencies
 ↓
Stages
 ↓
Tasks
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A diferença é que agora também existe a camada estruturada que vimos na Parte 6:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DataFrame
 ↓
Logical Plan
 ↓
Catalyst
 ↓
Physical Plan
 ↓
Distributed Execution
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os fundamentos não desapareceram.&lt;/p&gt;

&lt;p&gt;Eles ganharam novas abstrações.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. Lineage vs. Query Plan
&lt;/h2&gt;

&lt;p&gt;Também não devemos tratar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Lineage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Query Plan
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;como exatamente a mesma coisa.&lt;/p&gt;

&lt;p&gt;Uma forma útil de pensar é:&lt;/p&gt;

&lt;h3&gt;
  
  
  Lineage
&lt;/h3&gt;

&lt;p&gt;Relaciona datasets e dependências necessárias para derivação/recomputação.&lt;/p&gt;

&lt;h3&gt;
  
  
  Query Plan
&lt;/h3&gt;

&lt;p&gt;Representa operações lógicas e físicas usadas pelo engine estruturado para planejar e executar uma consulta.&lt;/p&gt;

&lt;p&gt;Eles se encontram na execução distribuída, mas são conceitos distintos.&lt;/p&gt;




&lt;h2&gt;
  
  
  35. DAG Scheduler
&lt;/h2&gt;

&lt;p&gt;Agora podemos encaixar outra peça da arquitetura.&lt;/p&gt;

&lt;p&gt;Spark possui componentes responsáveis por transformar a computação em trabalho executável.&lt;/p&gt;

&lt;p&gt;Uma visão conceitual:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application
    ↓
Action
    ↓
Job
    ↓
DAG Scheduler
    ↓
Stages
    ↓
Task Sets
    ↓
Executors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O DAG Scheduler consegue analisar dependências e identificar fronteiras de Stage, especialmente ao redor de shuffles.&lt;/p&gt;

&lt;p&gt;Podemos pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Narrow
 ↓
same Stage where possible

Wide
 ↓
Shuffle
 ↓
new Stage boundary
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso conecta lineage diretamente ao modelo de execução que estudamos na Parte 2.&lt;/p&gt;




&lt;h2&gt;
  
  
  36. Task Scheduling
&lt;/h2&gt;

&lt;p&gt;Depois que o trabalho é dividido em Stages e Tasks, precisamos executar essas Tasks nos Executors.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DAG Scheduler
     ↓
Stages
     ↓
Task Scheduler
     ↓
Executors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora entram preocupações como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;recursos disponíveis;&lt;/li&gt;
&lt;li&gt;localização;&lt;/li&gt;
&lt;li&gt;execução;&lt;/li&gt;
&lt;li&gt;retries;&lt;/li&gt;
&lt;li&gt;status das Tasks.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O Spark não é apenas uma biblioteca de transformação.&lt;/p&gt;

&lt;p&gt;Existe um sistema de scheduling distribuído por trás da API.&lt;/p&gt;




&lt;h2&gt;
  
  
  37. Data Locality
&lt;/h2&gt;

&lt;p&gt;Existe ainda uma ideia importante.&lt;/p&gt;

&lt;p&gt;Se os dados estão próximos de determinado Executor ou nó, pode ser vantajoso executar o processamento próximo deles.&lt;/p&gt;

&lt;p&gt;Em vez de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Move 100 GB of data
      ↓
to computation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;podemos preferir, quando a arquitetura permite:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Move computation
      ↓
to data
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa filosofia foi extremamente importante no ecossistema de Big Data.&lt;/p&gt;

&lt;p&gt;No Spark, scheduling pode considerar locality dependendo da fonte e do ambiente.&lt;/p&gt;

&lt;p&gt;Novamente encontramos o princípio:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Mover computação costuma ser mais barato do que mover grandes volumes de dados.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  38. Falhas e Shuffle
&lt;/h2&gt;

&lt;p&gt;Shuffle torna a recuperação mais interessante.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Stage 1
  ↓
Shuffle Write
  ↓
Stage 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Os outputs intermediários de shuffle são importantes para o Stage seguinte.&lt;/p&gt;

&lt;p&gt;Se dados intermediários necessários forem perdidos devido a falhas, Spark pode precisar recomputar partes anteriores.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Lost Shuffle Data
       ↓
Need previous output
       ↓
Recompute upstream tasks
       ↓
Regenerate shuffle data
       ↓
Continue downstream work
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora entendemos por que lineage continua relevante mesmo depois de falarmos sobre shuffle.&lt;/p&gt;




&lt;h2&gt;
  
  
  39. Recomputation tem custo
&lt;/h2&gt;

&lt;p&gt;Fault tolerance não significa:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;failure = free
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2-hour computation
      ↓
lost intermediate data
      ↓
recompute
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O sistema pode conseguir se recuperar.&lt;/p&gt;

&lt;p&gt;Mas isso custa:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU;&lt;/li&gt;
&lt;li&gt;tempo;&lt;/li&gt;
&lt;li&gt;rede;&lt;/li&gt;
&lt;li&gt;I/O;&lt;/li&gt;
&lt;li&gt;dinheiro.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Então existe outra pergunta arquitetural:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Qual é o custo aceitável de recomputação?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;É aqui que persistência e checkpoint podem entrar.&lt;/p&gt;




&lt;h2&gt;
  
  
  40. Cache, Checkpoint ou Recompute?
&lt;/h2&gt;

&lt;p&gt;Podemos montar um modelo mental:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Dataset intermediário
        │
        ▼
Será reutilizado?
   │           │
  sim         não
   │           │
   ▼           ▼
Cache?     Recompute?
   │
   ▼
É caro de reconstruir
e lineage é problemático?
   │
   ▼
Checkpoint?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Não é uma árvore de decisão absoluta.&lt;/p&gt;

&lt;p&gt;É apenas uma forma de pensar.&lt;/p&gt;

&lt;p&gt;A escolha depende de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cost of computation
Cost of storage
Reuse
Failure probability
Lineage length
Recovery requirements
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  41. Um exemplo prático
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;spark&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/events&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;raw&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_date &amp;gt;= &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;2026-01-01&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa transformação é usada por vários outputs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;por_cliente&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;base&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;por_evento&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;base&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;event_type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;receita&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;base&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  RAW
                   │
                   ▼
             Filter + Select
                   │
                   ▼
                  BASE
             ┌─────┼─────┐
             ▼     ▼     ▼
         Customer Event Revenue
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se &lt;code&gt;base&lt;/code&gt; for caro de produzir e reutilizado várias vezes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;pode fazer sentido.&lt;/p&gt;

&lt;p&gt;A decisão nasce do DAG.&lt;/p&gt;

&lt;p&gt;Não de uma regra genérica de que:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cache = performance
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  42. Um exemplo de Checkpoint
&lt;/h2&gt;

&lt;p&gt;Agora imagine um processamento iterativo ou uma cadeia extremamente extensa.&lt;/p&gt;

&lt;p&gt;Em RDD:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rdd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;textFile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/source&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;rdd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;expensive_transformation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rdd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A linhagem pode crescer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source
 ↓
T1
 ↓
T2
 ↓
T3
 ↓
...
 ↓
T100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos decidir estabelecer checkpoints em pontos apropriados.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source
 ↓
T1
 ↓
...
 ↓
T25
 ↓
CHECKPOINT
 ↓
T26
 ↓
...
 ↓
T50
 ↓
CHECKPOINT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Assim controlamos o tamanho da cadeia de recuperação.&lt;/p&gt;




&lt;h2&gt;
  
  
  43. &lt;code&gt;localCheckpoint()&lt;/code&gt; merece cuidado
&lt;/h2&gt;

&lt;p&gt;Spark também possui mecanismos de checkpoint local em determinados contextos.&lt;/p&gt;

&lt;p&gt;Mas existe uma diferença conceitual importante:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Reliable checkpoint
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;busca armazenamento confiável para quebrar a linhagem mantendo capacidade adequada de recuperação.&lt;/p&gt;

&lt;p&gt;Já um checkpoint baseado apenas em armazenamento local dos Executors não oferece necessariamente a mesma garantia de tolerância a falhas.&lt;/p&gt;

&lt;p&gt;Portanto:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Não devemos tratar todo tipo de checkpoint como equivalente em resiliência.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A escolha depende do objetivo.&lt;/p&gt;




&lt;h2&gt;
  
  
  44. Resiliência também depende do storage
&lt;/h2&gt;

&lt;p&gt;Se o checkpoint precisa servir como ponto confiável de recuperação, precisamos pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Onde ele está armazenado?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Um diretório local efêmero possui características muito diferentes de armazenamento distribuído e durável.&lt;/p&gt;

&lt;p&gt;Então a arquitetura completa é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark
   +
Lineage
   +
Checkpoint strategy
   +
Reliable storage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Fault tolerance não acontece isoladamente.&lt;/p&gt;




&lt;h2&gt;
  
  
  45. O que acontece quando uma máquina desaparece?
&lt;/h2&gt;

&lt;p&gt;Agora podemos responder de forma muito melhor.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor 3
    ↓
FAIL
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Spark pode identificar trabalho perdido.&lt;/p&gt;

&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Lost partition
      ↓
Find lineage
      ↓
Find required ancestors
      ↓
Schedule recomputation
      ↓
Execute elsewhere
      ↓
Rebuild data
      ↓
Continue
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa sequência explica o:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Resilient&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;de:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Resilient Distributed Dataset.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  46. O que eu olho quando existem muitas recomputações?
&lt;/h2&gt;

&lt;p&gt;Se uma aplicação está refazendo trabalho demais, quero investigar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executors estão instáveis?

Existe perda de containers?

Existe pressão de memória?

Existe GC excessivo?

Existe problema de infraestrutura?

Existe shuffle data sendo perdida?

A linhagem é cara?

Falta persistência em um ponto estratégico?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou seja, recomputation é um mecanismo de recuperação.&lt;/p&gt;

&lt;p&gt;Mas recomputation excessiva também pode ser sintoma.&lt;/p&gt;




&lt;h2&gt;
  
  
  47. Resiliência não significa ignorar infraestrutura
&lt;/h2&gt;

&lt;p&gt;Podemos ter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor failure
 ↓
Retry
 ↓
Executor failure
 ↓
Retry
 ↓
Executor failure
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Em algum momento a pergunta deixa de ser:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Spark consegue tentar novamente?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;e passa a ser:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Por que meus Executors continuam morrendo?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Talvez exista:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OOM
Disk failure
Container eviction
Network instability
Bad node
Data skew
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Fault tolerance deve absorver falhas ocasionais.&lt;/p&gt;

&lt;p&gt;Não deve ser usada para esconder um cluster permanentemente instável.&lt;/p&gt;




&lt;h2&gt;
  
  
  48. Observabilidade novamente
&lt;/h2&gt;

&lt;p&gt;No troubleshooting, quero relacionar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application
    ↓
Jobs
    ↓
Stages
    ↓
Tasks
    ↓
Executors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Failures
Retries
Recomputation
Shuffle
Memory
GC
Spill
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se Tasks estão sendo repetidas, preciso saber por quê.&lt;/p&gt;

&lt;p&gt;Se Stages estão sendo refeitos, preciso saber por quê.&lt;/p&gt;

&lt;p&gt;Se Executors desaparecem, preciso saber por quê.&lt;/p&gt;

&lt;p&gt;Resiliência é muito mais útil quando acompanhada de observabilidade.&lt;/p&gt;




&lt;h2&gt;
  
  
  49. Um modelo mental de Fault Tolerance
&lt;/h2&gt;

&lt;p&gt;Minha forma de enxergar ficou assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                   DATA SOURCE
                       │
                       ▼
                 Transformations
                       │
                       ▼
                    Lineage
                       │
                       ▼
                      DAG
                       │
                       ▼
                    Stages
                       │
                       ▼
                     Tasks
                       │
                       ▼
                   Executors
                       │
                       ▼
                    Failure?
                   /         \
                 não          sim
                 │             │
                 ▼             ▼
             Continue      What was lost?
                               │
                               ▼
                           Lineage
                               │
                               ▼
                         Recompute
                               │
                               ▼
                           Continue
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora acrescente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cache
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;quando queremos evitar recomputação frequente.&lt;/p&gt;

&lt;p&gt;E:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Checkpoint
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;quando queremos estabelecer um novo ponto materializado e truncar uma linhagem problemática.&lt;/p&gt;




&lt;h2&gt;
  
  
  50. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;A resiliência do Spark não vem de uma única configuração.&lt;/p&gt;

&lt;p&gt;Ela emerge de várias ideias trabalhando juntas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Immutable Datasets
        +
Transformations
        +
Dependencies
        +
Lineage
        +
DAG
        +
Task Scheduling
        +
Retries
        +
Recomputation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E, quando necessário:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cache / Persist
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Checkpoint
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa arquitetura permite que uma falha parcial não precise significar perda completa da computação.&lt;/p&gt;




&lt;h2&gt;
  
  
  51. O significado de RDD ficou muito mais claro
&lt;/h2&gt;

&lt;p&gt;No começo da série:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD
=
Resilient Distributed Dataset
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;podia parecer apenas um nome.&lt;/p&gt;

&lt;p&gt;Agora conseguimos decompor:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DISTRIBUTED
     ↓
Partitions across cluster
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DATASET
     ↓
Collection of data
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RESILIENT
     ↓
Lineage
     ↓
Recomputation
     ↓
Fault recovery
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa é uma das ideias fundamentais da arquitetura original do Spark.&lt;/p&gt;




&lt;h2&gt;
  
  
  52. O que essa ideia me ensinou além do Spark
&lt;/h2&gt;

&lt;p&gt;A lição mais interessante vai além da tecnologia.&lt;/p&gt;

&lt;p&gt;Em sistemas distribuídos, não podemos construir arquitetura assumindo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Everything will always work
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Precisamos assumir:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Machines fail
Processes fail
Networks fail
Tasks fail
Containers disappear
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Então a arquitetura precisa responder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;What happens next?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa pergunta vale para:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;processamento de dados;&lt;/li&gt;
&lt;li&gt;microsserviços;&lt;/li&gt;
&lt;li&gt;bancos distribuídos;&lt;/li&gt;
&lt;li&gt;streaming;&lt;/li&gt;
&lt;li&gt;Kubernetes;&lt;/li&gt;
&lt;li&gt;cloud;&lt;/li&gt;
&lt;li&gt;Machine Learning distribuído.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tolerância a falhas não é um recurso opcional adicionado no final.&lt;/p&gt;

&lt;p&gt;É uma propriedade arquitetural.&lt;/p&gt;




&lt;h2&gt;
  
  
  53. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Quando vejo uma transformação Spark:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;dados&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(...)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(...)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reduceByKey&lt;/span&gt;&lt;span class="p"&gt;(...)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;não penso apenas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;input
 ↓
transform
 ↓
output
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Penso:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Where is the source?
        ↓
What are the dependencies?
        ↓
Where are the shuffle boundaries?
        ↓
What is the lineage?
        ↓
What happens if a partition disappears?
        ↓
How expensive is recomputation?
        ↓
Should something be persisted?
        ↓
Is checkpointing justified?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa mudança de mentalidade é muito mais importante do que memorizar APIs.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora temos uma base sólida sobre:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RDD
 ↓
Partitions
 ↓
Transformations
 ↓
Lineage
 ↓
DAG
 ↓
Stages
 ↓
Tasks
 ↓
Executors
 ↓
Fault Tolerance
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas existe um tema do material original que merece ser revisitado com bastante cuidado:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;variáveis compartilhadas.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Em uma aplicação comum, podemos imaginar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;contador&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e esperar que exista apenas um contador.&lt;/p&gt;

&lt;p&gt;Mas em Spark temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver
  +
Executor 1
  +
Executor 2
  +
Executor 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Então surge uma pergunta:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;O que significa "variável compartilhada" em uma aplicação distribuída?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;No próximo artigo vamos explorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;closures;&lt;/li&gt;
&lt;li&gt;serialização de funções;&lt;/li&gt;
&lt;li&gt;variáveis capturadas;&lt;/li&gt;
&lt;li&gt;Driver vs. Executors;&lt;/li&gt;
&lt;li&gt;por que alterar uma variável dentro de uma Task não funciona como muitos esperam;&lt;/li&gt;
&lt;li&gt;Broadcast Variables;&lt;/li&gt;
&lt;li&gt;Accumulators;&lt;/li&gt;
&lt;li&gt;quando utilizar cada mecanismo;&lt;/li&gt;
&lt;li&gt;armadilhas de side effects;&lt;/li&gt;
&lt;li&gt;e como pensar em estado em computação distribuída.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Vamos sair disto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;counter = 0

for item:
    counter += 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;para isto:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;              DRIVER
                 │
          serialized closure
        ┌────────┼────────┐
        ▼        ▼        ▼
     Executor Executor Executor
        │        │        │
      Task     Task      Task
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E entender por que código aparentemente simples muda completamente quando atravessa máquinas.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;← Parte 7:&lt;/strong&gt; Memória — Executors, Driver, Cache, Spill e OutOfMemory&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 9 — Variáveis compartilhadas no Apache Spark: Broadcast, Accumulators e Closures&lt;/p&gt;

</description>
      <category>spark</category>
      <category>python</category>
      <category>dataengineering</category>
      <category>bigdata</category>
    </item>
    <item>
      <title>Como aprendi Apache Spark — Parte 7: Memória — Executors, Driver, Cache, Spill e OutOfMemory</title>
      <dc:creator>renanpyd</dc:creator>
      <pubDate>Sat, 24 Oct 2026 16:10:33 +0000</pubDate>
      <link>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-7-memoria-executors-driver-cache-spill-e-outofmemory-3e93</link>
      <guid>https://dev.to/renanpyd/como-aprendi-apache-spark-parte-7-memoria-executors-driver-cache-spill-e-outofmemory-3e93</guid>
      <description>&lt;p&gt;Nos artigos anteriores construímos uma visão cada vez mais completa do Apache Spark.&lt;/p&gt;

&lt;p&gt;Passamos por:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Data
 ↓
Partitions
 ↓
Tasks
 ↓
Executors
 ↓
Shuffle
 ↓
Joins
 ↓
Spark SQL
 ↓
Catalyst
 ↓
Physical Plan
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas existe um recurso atravessando praticamente todas essas etapas:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;memória.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Quando comecei a estudar problemas de memória no Spark, era tentador resumir tudo a:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OutOfMemory
     ↓
aumentar RAM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hoje eu vejo esse raciocínio como incompleto.&lt;/p&gt;

&lt;p&gt;Um problema de memória pode estar relacionado a:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver
Executors
Partitions
Shuffle
Broadcast
Cache
Serialization
Garbage Collection
Python workers
Off-heap/native memory
Memory Overhead
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;E aumentar memória sem descobrir qual desses componentes está causando o problema pode apenas tornar um Job caro antes de ele falhar novamente.&lt;/p&gt;

&lt;p&gt;A pergunta correta não é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Quanto de memória devo adicionar?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;É:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Quem está consumindo memória, por quê, e em qual parte da arquitetura?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  1. Memória em uma aplicação Spark
&lt;/h2&gt;

&lt;p&gt;Vamos voltar à arquitetura básica:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 DRIVER
                    │
                    ▼
             Cluster Manager
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
      Executor 1 Executor 2 Executor 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Temos dois contextos que precisam ser diferenciados imediatamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver Memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa distinção é fundamental.&lt;/p&gt;

&lt;p&gt;Porque:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver OOM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor OOM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;podem ter causas completamente diferentes.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. Driver Memory
&lt;/h2&gt;

&lt;p&gt;O Driver coordena a aplicação.&lt;/p&gt;

&lt;p&gt;Ele mantém informações relacionadas a:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;aplicação;&lt;/li&gt;
&lt;li&gt;Jobs;&lt;/li&gt;
&lt;li&gt;Stages;&lt;/li&gt;
&lt;li&gt;Tasks;&lt;/li&gt;
&lt;li&gt;scheduling;&lt;/li&gt;
&lt;li&gt;metadados;&lt;/li&gt;
&lt;li&gt;planos de execução;&lt;/li&gt;
&lt;li&gt;resultados retornados ao Driver;&lt;/li&gt;
&lt;li&gt;estado necessário para coordenar a computação.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  DRIVER
                     │
      ┌──────────────┼──────────────┐
      ▼              ▼              ▼
   Planning       Scheduling     Metadata
      │              │              │
      └──────────────┼──────────────┘
                     ▼
                  Control
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso significa que o Driver não deveria ser tratado como destino para todo o dataset distribuído.&lt;/p&gt;

&lt;p&gt;Esse ponto parece óbvio.&lt;/p&gt;

&lt;p&gt;Até alguém executar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;dados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. O perigo do &lt;code&gt;collect()&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O processamento acontece de maneira distribuída e o resultado retornado é pequeno:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;123456789
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora compare com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;dados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;collect()&lt;/code&gt; solicita que todos os registros do DataFrame sejam retornados ao Driver.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor 1 ──────┐
Executor 2 ──────┤
Executor 3 ──────┼──► DRIVER
Executor 4 ──────┤
Executor 5 ──────┘
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se o dataset possuir:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;500 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e o Driver tiver:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;temos um problema bastante previsível.&lt;/p&gt;

&lt;p&gt;O Spark é distribuído.&lt;/p&gt;

&lt;p&gt;Mas acabamos de pedir:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Traga tudo para uma única máquina.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  4. &lt;code&gt;collect()&lt;/code&gt; não é ruim — o contexto é que importa
&lt;/h2&gt;

&lt;p&gt;Não existe problema em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;quando sabemos que o resultado será pequeno.&lt;/p&gt;

&lt;p&gt;Também existem cenários legítimos em que precisamos trazer pequenas quantidades de dados ao Driver.&lt;/p&gt;

&lt;p&gt;O problema é usar &lt;code&gt;collect()&lt;/code&gt; sem compreender:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Distributed Dataset
        ↓
      Driver
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A pergunta antes de um &lt;code&gt;collect()&lt;/code&gt; deveria ser:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Qual o volume máximo que pode voltar para o Driver?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Essa pergunta evita muitos problemas.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Outros caminhos para sobrecarregar o Driver
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;collect()&lt;/code&gt; é o exemplo mais conhecido, mas não é o único.&lt;/p&gt;

&lt;p&gt;Problemas de Driver também podem aparecer com:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;resultados excessivamente grandes;&lt;/li&gt;
&lt;li&gt;estruturas locais gigantes;&lt;/li&gt;
&lt;li&gt;excesso de metadados;&lt;/li&gt;
&lt;li&gt;quantidade extrema de Tasks;&lt;/li&gt;
&lt;li&gt;operações que materializam dados no processo do Driver;&lt;/li&gt;
&lt;li&gt;código Python executado localmente;&lt;/li&gt;
&lt;li&gt;determinadas operações de visualização ou conversão.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;pdf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toPandas&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;merece atenção.&lt;/p&gt;

&lt;p&gt;O resultado precisa caber na memória do processo que recebe esse DataFrame local.&lt;/p&gt;

&lt;p&gt;Ou seja:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark DataFrame distribuído
          ↓
       toPandas()
          ↓
DataFrame local
          ↓
      DRIVER
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se os dados forem grandes, temos novamente uma concentração de memória.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Executor Memory
&lt;/h2&gt;

&lt;p&gt;Agora vamos para os Executors.&lt;/p&gt;

&lt;p&gt;Eles executam as Tasks.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor
   │
   ├── Task 1
   ├── Task 2
   ├── Task 3
   └── Task 4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Durante a execução, memória pode ser necessária para:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;processamento;&lt;/li&gt;
&lt;li&gt;agregações;&lt;/li&gt;
&lt;li&gt;joins;&lt;/li&gt;
&lt;li&gt;sorting;&lt;/li&gt;
&lt;li&gt;shuffle;&lt;/li&gt;
&lt;li&gt;estruturas intermediárias;&lt;/li&gt;
&lt;li&gt;cache;&lt;/li&gt;
&lt;li&gt;dados serializados;&lt;/li&gt;
&lt;li&gt;execução de código;&lt;/li&gt;
&lt;li&gt;componentes fora do heap JVM.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;É aqui que começam vários problemas clássicos.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Uma partição vira trabalho para uma Task
&lt;/h2&gt;

&lt;p&gt;Da Parte 3:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Em uma determinada Stage, cada partição normalmente corresponde ao trabalho de uma Task.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Então existe uma conexão direta:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Partition
    ↓
Task
    ↓
Memory pressure
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Partition 1 → 200 MB
Partition 2 → 180 MB
Partition 3 → 220 MB
Partition 4 → 18 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A Task responsável pela quarta partição terá um perfil completamente diferente das demais.&lt;/p&gt;

&lt;p&gt;Podemos observar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task 1 → OK
Task 2 → OK
Task 3 → OK
Task 4 → Memory Pressure
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Talvez o problema não seja:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Executor com pouca memória.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Talvez seja:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Uma partição absurdamente maior que as outras.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  8. Particionamento e memória são inseparáveis
&lt;/h2&gt;

&lt;p&gt;Imagine um dataset de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 TB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;distribuído em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10 partitions
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Simplificando muito:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;≈ 100 GB por partition
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1000 partitions
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Teríamos algo mais próximo de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;≈ 1 GB por partition
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso não significa que uma Task precise manter toda a partição simultaneamente em memória.&lt;/p&gt;

&lt;p&gt;Mas o exemplo mostra uma relação importante:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;O tamanho e a distribuição das partições influenciam diretamente a pressão de memória das Tasks.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;É por isso que simplesmente aumentar &lt;code&gt;spark.executor.memory&lt;/code&gt; pode esconder a causa real.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Execution Memory
&lt;/h2&gt;

&lt;p&gt;Durante operações como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Spark precisa de memória para executar estruturas intermediárias.&lt;/p&gt;

&lt;p&gt;Também em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;orderBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou em determinados joins.&lt;/p&gt;

&lt;p&gt;Podemos pensar conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Memory
      │
      ├── Execution
      │      │
      │      ├── Shuffle
      │      ├── Join
      │      ├── Sort
      │      └── Aggregation
      │
      └── Storage
             │
             ├── Cached Data
             └── Persisted Data
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No Spark moderno, execution e storage participam de um modelo de gerenciamento de memória integrado.&lt;/p&gt;

&lt;p&gt;Isso é importante porque a memória disponível não deve ser imaginada como caixas totalmente independentes e rígidas em todos os momentos.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Storage Memory
&lt;/h2&gt;

&lt;p&gt;Agora imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Quando o DataFrame é materializado, Spark pode manter partições para reutilização.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor
   │
   ├── Execution Memory
   │
   └── Storage Memory
            │
            ├── Partition A
            ├── Partition B
            └── Partition C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso pode ser extremamente útil.&lt;/p&gt;

&lt;p&gt;Se o mesmo dataset caro for utilizado várias vezes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Expensive computation
       ↓
     CACHE
       ↓
 ┌─────┼─────┐
 ▼     ▼     ▼
Use 1 Use 2 Use 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;evitamos recalcular repetidamente a mesma linhagem.&lt;/p&gt;

&lt;p&gt;Mas cache também custa memória.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. &lt;code&gt;cache()&lt;/code&gt; não significa "deixe rápido"
&lt;/h2&gt;

&lt;p&gt;Esse é outro anti-pattern comum:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ser utilizado como se significasse:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;optimize(df)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Não significa.&lt;/p&gt;

&lt;p&gt;Imagine um DataFrame de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;800 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;que será utilizado apenas uma vez.&lt;/p&gt;

&lt;p&gt;Fazer cache pode simplesmente:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;consumir memória;&lt;/li&gt;
&lt;li&gt;provocar eviction;&lt;/li&gt;
&lt;li&gt;aumentar pressão sobre o Executor;&lt;/li&gt;
&lt;li&gt;competir com execution memory;&lt;/li&gt;
&lt;li&gt;não trazer benefício relevante.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cache faz mais sentido quando existe &lt;strong&gt;reuso&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;transformacao_cara&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;resultado_a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estado&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;resultado_b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;produto&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;resultado_c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status = &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ACTIVE&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora existe um motivo claro:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Expensive Base
     ↓
   CACHE
     ↓
 ┌────┼────┐
 ▼    ▼    ▼
 A    B    C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  12. Cache também é Lazy
&lt;/h2&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso não significa necessariamente:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Neste instante, todas as partições foram computadas e armazenadas.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A materialização ocorre quando uma Action força a computação.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cache()
   ↓
mark for persistence
   ↓
count()
   ↓
compute partitions
   ↓
store according to persistence strategy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Lazy evaluation continua presente.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. &lt;code&gt;persist()&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Além de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;podemos trabalhar com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;persist&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e escolher estratégias de persistência adequadas ao contexto.&lt;/p&gt;

&lt;p&gt;Conceitualmente, a persistência pode envolver combinações de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Memory
Disk
Serialization
Replication
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A escolha depende de fatores como:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;tamanho;&lt;/li&gt;
&lt;li&gt;frequência de reuso;&lt;/li&gt;
&lt;li&gt;custo de recomputação;&lt;/li&gt;
&lt;li&gt;memória disponível;&lt;/li&gt;
&lt;li&gt;tolerância a I/O.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Novamente, não existe uma configuração universalmente melhor.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. E quando a memória não é suficiente?
&lt;/h2&gt;

&lt;p&gt;Agora chegamos a algo que apareceu na Parte 4:&lt;/p&gt;

&lt;h1&gt;
  
  
  Spill
&lt;/h1&gt;

&lt;p&gt;Imagine uma Task realizando:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Sort
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Aggregation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou processamento relacionado a:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Shuffle
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ela utiliza estruturas em memória.&lt;/p&gt;

&lt;p&gt;Mas a quantidade necessária ultrapassa o que pode ser mantido eficientemente ali.&lt;/p&gt;

&lt;p&gt;Então parte dos dados pode ser enviada para disco.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task
 ↓
Memory
 ↓
Memory pressure
 ↓
Spill
 ↓
Disk
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Disk
 ↓
Read
 ↓
Continue processing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso permite que determinadas operações continuem mesmo quando tudo não cabe simultaneamente em memória.&lt;/p&gt;

&lt;p&gt;Mas existe um preço.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Spill troca memória por I/O
&lt;/h2&gt;

&lt;p&gt;Compare:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Memory
  ↓
CPU
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;com:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Memory
  ↓
Disk Write
  ↓
Disk Read
  ↓
CPU
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Disco normalmente é muito mais lento do que memória.&lt;/p&gt;

&lt;p&gt;Então grandes volumes de spill podem transformar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Job
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Job lento
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mesmo sem ocorrer um OutOfMemory.&lt;/p&gt;

&lt;p&gt;Essa é uma lição importante:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Uma aplicação pode estar sofrendo de pressão de memória sem necessariamente falhar por OOM.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Ela pode simplesmente ficar muito mais lenta.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. Shuffle + Memory + Spill
&lt;/h2&gt;

&lt;p&gt;Agora conectamos vários artigos.&lt;/p&gt;

&lt;p&gt;Considere:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;vendas&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Podemos ter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Partitions
     ↓
Partial Aggregation
     ↓
Shuffle Write
     ↓
Network
     ↓
Shuffle Read
     ↓
Aggregation
     ↓
Memory pressure
     ↓
Spill
     ↓
Disk
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora um simples:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nf"&gt;groupBy&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ganha uma interpretação completamente diferente.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. Data Skew volta novamente
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P1 = 200 MB
P2 = 210 MB
P3 = 190 MB
P4 = 15 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Durante uma agregação:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P1 → OK
P2 → OK
P3 → OK
P4 → Spill → Spill → Spill
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou pior:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;P4 → OutOfMemory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A reação inicial pode ser:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Aumentar executor memory.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Talvez funcione.&lt;/p&gt;

&lt;p&gt;Mas talvez a pergunta correta seja:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Por que existe uma partição de 15 GB quando as demais possuem aproximadamente 200 MB?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Estamos novamente diante de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Data Skew
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A causa é distribuição.&lt;/p&gt;

&lt;p&gt;O sintoma aparece na memória.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. OutOfMemory não é um único problema
&lt;/h2&gt;

&lt;p&gt;Quando alguém diz:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Spark deu OOM."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Ainda sabemos muito pouco.&lt;/p&gt;

&lt;p&gt;Precisamos perguntar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Onde?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver OOM?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor OOM?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou pressão fora do heap/container?&lt;/p&gt;

&lt;p&gt;Também precisamos perguntar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Em qual Stage?

Em qual operação?

Em todas as Tasks?

Em uma única Task?

Depois de um Shuffle?

Durante Broadcast?

Durante collect()?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essas perguntas mudam completamente a investigação.&lt;/p&gt;




&lt;h2&gt;
  
  
  19. Um Driver OOM pode indicar problema de arquitetura
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;dados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;sobre um dataset enorme.&lt;/p&gt;

&lt;p&gt;Podemos aumentar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.driver.memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;4 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;para:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;16 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;32 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;64 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Talvez o Job finalmente funcione.&lt;/p&gt;

&lt;p&gt;Mas a pergunta continua:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Por que estamos trazendo um dataset distribuído gigantesco para um único processo?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Aumentar memória resolveu o sintoma.&lt;/p&gt;

&lt;p&gt;Talvez não tenha resolvido o design.&lt;/p&gt;




&lt;h2&gt;
  
  
  20. Executor OOM também pode esconder outro problema
&lt;/h2&gt;

&lt;p&gt;Agora imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor OOM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A reação:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.executor.memory
8 GB → 16 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas talvez exista:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;uma partition = 40 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;join skewed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;broadcast excessivo
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou várias Tasks concorrentes disputando recursos.&lt;/p&gt;

&lt;p&gt;Ou consumo significativo fora do heap principal.&lt;/p&gt;

&lt;p&gt;Então:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Executor OOM não significa automaticamente que o Executor precisa apenas de mais heap.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  21. &lt;code&gt;spark.executor.memory&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Uma configuração conhecida é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.executor.memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ela define a quantidade de memória heap disponibilizada ao processo Executor JVM.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nt"&gt;--executor-memory&lt;/span&gt; 8g
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas existe uma armadilha conceitual importante:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A memória total necessária por um Executor não é necessariamente igual ao valor de &lt;code&gt;spark.executor.memory&lt;/code&gt;.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Existe memória utilizada fora desse heap.&lt;/p&gt;

&lt;p&gt;E isso nos leva diretamente ao tema do material original que estudei:&lt;/p&gt;

&lt;h1&gt;
  
  
  Memory Overhead
&lt;/h1&gt;




&lt;h2&gt;
  
  
  22. O que é Memory Overhead?
&lt;/h2&gt;

&lt;p&gt;Uma aplicação Spark não utiliza memória apenas para o heap principal da JVM.&lt;/p&gt;

&lt;p&gt;Podemos pensar de forma simplificada:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Container / Process Environment
│
├── JVM Heap
│
│    └── spark.executor.memory
│
└── Additional / Non-Heap Memory
     │
     ├── JVM overhead
     ├── native memory
     ├── Python processes
     ├── direct buffers
     └── other runtime overhead
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;É essa segunda região que torna &lt;code&gt;memoryOverhead&lt;/code&gt; tão importante.&lt;/p&gt;

&lt;p&gt;Dependendo do ambiente e do workload, um Executor pode morrer mesmo quando aparentemente existe heap suficiente.&lt;/p&gt;




&lt;h2&gt;
  
  
  23. Um exemplo conceitual
&lt;/h2&gt;

&lt;p&gt;Imagine uma configuração:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Heap
8 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso não significa necessariamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor total
=
8 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O processo também precisa de espaço para outras necessidades.&lt;/p&gt;

&lt;p&gt;Podemos pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Container / Pod / Executor allocation

┌───────────────────────────────┐
│                               │
│ JVM Heap                      │
│ spark.executor.memory         │
│                               │
├───────────────────────────────┤
│                               │
│ Memory Overhead               │
│                               │
│ Native / Python / Direct /    │
│ JVM non-heap / other          │
│                               │
└───────────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se a região adicional for insuficiente, o processo pode exceder os limites impostos pelo gerenciador de recursos.&lt;/p&gt;




&lt;h2&gt;
  
  
  24. Por que isso aparece muito com PySpark?
&lt;/h2&gt;

&lt;p&gt;Em PySpark existe uma característica importante.&lt;/p&gt;

&lt;p&gt;Nosso código Python não transforma a JVM inteira em Python.&lt;/p&gt;

&lt;p&gt;Existe interação entre:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;JVM
 ↕
Python worker
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor JVM
     │
     ├── Spark execution
     │
     └── Python worker
             │
             └── Python code
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esses processos também consomem memória.&lt;/p&gt;

&lt;p&gt;Dependendo do workload, bibliotecas e operações utilizadas, isso pode se tornar relevante para o consumo total do Executor.&lt;/p&gt;

&lt;p&gt;Portanto, em PySpark precisamos olhar além de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;JVM Heap
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  25. Memory Overhead no material histórico
&lt;/h2&gt;

&lt;p&gt;Uma das anotações que motivou este capítulo tratava especificamente de problemas de &lt;code&gt;memoryOverhead&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Esse tipo de problema continua conceitualmente relevante.&lt;/p&gt;

&lt;p&gt;O que mudou ao longo dos anos foram:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;versões do Spark;&lt;/li&gt;
&lt;li&gt;nomes e comportamento de determinadas configurações;&lt;/li&gt;
&lt;li&gt;resource managers;&lt;/li&gt;
&lt;li&gt;defaults;&lt;/li&gt;
&lt;li&gt;cálculo automático de overhead;&lt;/li&gt;
&lt;li&gt;integração com containers;&lt;/li&gt;
&lt;li&gt;Kubernetes;&lt;/li&gt;
&lt;li&gt;execução Python;&lt;/li&gt;
&lt;li&gt;mecanismos internos de memória.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Por isso não considero uma boa prática pegar uma configuração antiga e simplesmente reproduzi-la hoje.&lt;/p&gt;

&lt;p&gt;O conceito continua válido.&lt;/p&gt;

&lt;p&gt;A configuração precisa ser analisada no contexto da versão e do ambiente atuais.&lt;/p&gt;




&lt;h2&gt;
  
  
  26. Heap vs. Off-Heap vs. Overhead
&lt;/h2&gt;

&lt;p&gt;Uma simplificação útil é:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PROCESS MEMORY
     │
     ├── JVM Heap
     │
     ├── JVM Non-Heap
     │
     ├── Direct / Native Memory
     │
     ├── Python Worker Memory
     │
     └── Runtime Overhead
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso é mais próximo da realidade do que pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;memory = spark.executor.memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Quando um container é encerrado por ultrapassar limite de memória, precisamos descobrir &lt;strong&gt;qual parte do consumo cresceu&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  27. Garbage Collection
&lt;/h2&gt;

&lt;p&gt;Agora vamos entrar em outro sintoma importante:&lt;/p&gt;

&lt;h1&gt;
  
  
  GC
&lt;/h1&gt;

&lt;p&gt;A JVM gerencia automaticamente memória de objetos.&lt;/p&gt;

&lt;p&gt;Objetos são criados:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Objects
 ↓
Heap
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois deixam de ser necessários.&lt;/p&gt;

&lt;p&gt;O Garbage Collector precisa recuperar esse espaço.&lt;/p&gt;

&lt;p&gt;Em um cenário saudável:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application Work ███████████████████

GC               █
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas sob forte pressão podemos encontrar algo mais próximo de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application ███████

GC          █████████████
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ou seja:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A aplicação passa tempo demais tentando gerenciar memória em vez de processar dados.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  28. GC alto é sintoma
&lt;/h2&gt;

&lt;p&gt;Assim como OOM, GC excessivo não deveria gerar automaticamente:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Aumente a heap.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Talvez tenhamos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;objetos demais;&lt;/li&gt;
&lt;li&gt;partições grandes;&lt;/li&gt;
&lt;li&gt;agregações pesadas;&lt;/li&gt;
&lt;li&gt;estruturas intermediárias enormes;&lt;/li&gt;
&lt;li&gt;cache excessivo;&lt;/li&gt;
&lt;li&gt;serialização inadequada;&lt;/li&gt;
&lt;li&gt;concorrência excessiva por Executor.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Precisamos descobrir por que a memória está sob pressão.&lt;/p&gt;




&lt;h2&gt;
  
  
  29. Mais cores por Executor também afetam memória
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor
8 cores
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Dependendo da configuração, várias Tasks podem executar simultaneamente nesse Executor.&lt;/p&gt;

&lt;p&gt;Conceitualmente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Memory
      │
 ┌────┼────┬────┬────┬────┐
 ▼    ▼    ▼    ▼    ▼    ▼
T1   T2   T3   T4   T5   ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cada Task pode precisar de memória.&lt;/p&gt;

&lt;p&gt;Então existe uma relação:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;mais Tasks concorrentes por Executor
              ↓
mais competição pela memória disponível
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso mostra por que tuning não deveria analisar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;executor memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;isoladamente.&lt;/p&gt;

&lt;p&gt;Precisamos olhar para:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Memory
      +
Executor Cores
      +
Partition Size
      +
Task Behavior
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  30. Executors gigantes também não são automaticamente melhores
&lt;/h2&gt;

&lt;p&gt;Poderíamos pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 Executor
64 cores
256 GB RAM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Parece poderoso.&lt;/p&gt;

&lt;p&gt;Mas Executors gigantes podem trazer trade-offs relacionados a:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;concorrência;&lt;/li&gt;
&lt;li&gt;garbage collection;&lt;/li&gt;
&lt;li&gt;blast radius de falha;&lt;/li&gt;
&lt;li&gt;utilização;&lt;/li&gt;
&lt;li&gt;scheduling;&lt;/li&gt;
&lt;li&gt;memória compartilhada entre muitas Tasks.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No outro extremo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;64 Executors
1 core
4 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;também possui trade-offs.&lt;/p&gt;

&lt;p&gt;Não existe um tamanho universal.&lt;/p&gt;

&lt;p&gt;Precisamos equilibrar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cores
memory
parallelism
partition size
workload
cluster manager
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  31. Broadcast também consome memória
&lt;/h2&gt;

&lt;p&gt;Na Parte 5 vimos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pyspark.sql.functions&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;broadcast&lt;/span&gt;

&lt;span class="n"&gt;resultado&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;grande&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pequena&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A estratégia pode evitar um shuffle grande.&lt;/p&gt;

&lt;p&gt;Mas existe um custo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Small Table
    ↓
Broadcast
 ┌──┼──┐
 ▼  ▼  ▼
E1 E2 E3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cada Executor precisa conseguir trabalhar com a estrutura transmitida.&lt;/p&gt;

&lt;p&gt;Se tentarmos broadcast de algo grande demais, podemos criar pressão significativa de memória.&lt;/p&gt;

&lt;p&gt;Novamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Network optimization
        ↕
Memory cost
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sistemas distribuídos são cheios de trade-offs.&lt;/p&gt;




&lt;h2&gt;
  
  
  32. Cache também compete por recursos
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df_a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;df_b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;df_c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;df_d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pode parecer:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Quanto mais cache, melhor.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Mas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cache A
Cache B
Cache C
Cache D
     ↓
Storage Pressure
     ↓
Eviction / recomputation / resource competition
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cache deveria ser uma decisão baseada em:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;reuse frequency
×
recomputation cost
×
available resources
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e não um reflexo automático.&lt;/p&gt;




&lt;h2&gt;
  
  
  33. &lt;code&gt;unpersist()&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Se um dataset não será mais reutilizado, podemos liberar sua persistência:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unpersist&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso comunica:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Não preciso mais manter essas partições persistidas.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Em pipelines longos isso pode ser importante.&lt;/p&gt;

&lt;p&gt;Por exemplo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Phase A
 ↓
cache intermediate
 ↓
reuse several times
 ↓
phase completed
 ↓
unpersist
 ↓
free resources for Phase B
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Gerenciamento de memória também faz parte do design do pipeline.&lt;/p&gt;




&lt;h2&gt;
  
  
  34. O Spark UI volta a ser essencial
&lt;/h2&gt;

&lt;p&gt;Se existe problema de memória, não quero começar alterando configurações.&lt;/p&gt;

&lt;p&gt;Quero observar.&lt;/p&gt;

&lt;p&gt;No Spark UI e métricas relacionadas, procuro sinais como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task Duration
GC Time
Shuffle Read
Shuffle Write
Spill
Storage
Executor behavior
Failed Tasks
Skew
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Também observo logs.&lt;/p&gt;

&lt;p&gt;A pergunta é:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;O que aconteceu antes da falha?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  35. Um padrão de investigação
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Job falhou
   ↓
Executor lost
   ↓
Memory-related error
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Meu raciocínio começaria assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual Stage?
    ↓
Qual operação?
    ↓
Join?
Aggregation?
Sort?
UDF?
Cache?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Todas as Tasks falharam?
        ↓
ou apenas algumas?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Se apenas algumas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Possível skew?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Depois:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual tamanho das partitions?
Quanto shuffle?
Quanto spill?
Quanto GC?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Só então:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Heap insuficiente?
Overhead insuficiente?
Particionamento ruim?
Broadcast inadequado?
Concorrência excessiva?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  36. Caso 1 — Driver morrendo
&lt;/h2&gt;

&lt;p&gt;Sintoma:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Driver OutOfMemory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Código:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;dados&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Dataset:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;120 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Driver:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Diagnóstico provável:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Distributed Data
      ↓
collect()
      ↓
Single Process
      ↓
OOM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Solução arquitetural:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Não coletar o dataset inteiro.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Talvez:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/data/output&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou continuar o processamento distribuído.&lt;/p&gt;

&lt;p&gt;A solução principal não é necessariamente:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 GB → 128 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  37. Caso 2 — Uma Task morre repetidamente
&lt;/h2&gt;

&lt;p&gt;Sintoma:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Stage 12

199 Tasks → OK
1 Task    → FAIL
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso é extremamente informativo.&lt;/p&gt;

&lt;p&gt;Talvez tenhamos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Data Skew
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou uma partição muito maior.&lt;/p&gt;

&lt;p&gt;Se aumentarmos memória de todos os Executors:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cluster cost ↑
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;para resolver uma única partição anormal, talvez estejamos atacando o lugar errado.&lt;/p&gt;

&lt;p&gt;Precisamos investigar a distribuição.&lt;/p&gt;




&lt;h2&gt;
  
  
  38. Caso 3 — Todas as Tasks sofrem Spill
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Task 1 → heavy spill
Task 2 → heavy spill
Task 3 → heavy spill
Task 4 → heavy spill
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora talvez tenhamos um problema sistêmico.&lt;/p&gt;

&lt;p&gt;Possíveis causas:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Partitions muito grandes
Memory insuficiente
Join pesado
Sort pesado
Aggregation pesada
Task concurrency
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nesse cenário, ajustar recursos pode realmente fazer parte da solução.&lt;/p&gt;

&lt;p&gt;Mas ainda precisamos medir.&lt;/p&gt;




&lt;h2&gt;
  
  
  39. Caso 4 — OOM depois de Broadcast
&lt;/h2&gt;

&lt;p&gt;Imagine que:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Job funcionava
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e alguém adicionou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nf"&gt;broadcast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dim&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora Executors começam a morrer.&lt;/p&gt;

&lt;p&gt;Pergunta:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Qual o tamanho real de dim?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Talvez:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;"small table"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;tenha crescido.&lt;/p&gt;

&lt;p&gt;Ontem:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;50 MB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hoje:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;6 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Uma otimização válida no passado pode se tornar um problema quando os dados mudam.&lt;/p&gt;

&lt;p&gt;Isso é importante:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Performance depende também da evolução dos dados.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  40. Caso 5 — PySpark e memória externa ao heap
&lt;/h2&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.executor.memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;aparentemente confortável.&lt;/p&gt;

&lt;p&gt;Mas containers continuam sendo encerrados por memória.&lt;/p&gt;

&lt;p&gt;Agora precisamos olhar para:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;JVM Heap
     +
Python workers
     +
Native memory
     +
Direct buffers
     +
Other overhead
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;É exatamente nesse tipo de cenário que compreender &lt;code&gt;memoryOverhead&lt;/code&gt; deixa de ser detalhe e passa a ser parte do diagnóstico.&lt;/p&gt;




&lt;h2&gt;
  
  
  41. Não ajuste &lt;code&gt;memoryOverhead&lt;/code&gt; às cegas
&lt;/h2&gt;

&lt;p&gt;Se um Executor morre por ultrapassar limites externos ao heap, aumentar overhead pode ser apropriado.&lt;/p&gt;

&lt;p&gt;Mas ainda precisamos perguntar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Por que esse consumo existe?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pode ser:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Python;&lt;/li&gt;
&lt;li&gt;bibliotecas nativas;&lt;/li&gt;
&lt;li&gt;buffers;&lt;/li&gt;
&lt;li&gt;comportamento da JVM;&lt;/li&gt;
&lt;li&gt;volume de dados;&lt;/li&gt;
&lt;li&gt;operação específica.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A configuração correta depende do workload e do ambiente.&lt;/p&gt;

&lt;p&gt;Por isso evito recomendações como:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sempre configure X GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sempre use Y%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;sem contexto.&lt;/p&gt;

&lt;p&gt;Defaults e comportamentos também evoluem entre versões.&lt;/p&gt;




&lt;h2&gt;
  
  
  42. Memória em Kubernetes
&lt;/h2&gt;

&lt;p&gt;Em ambientes containerizados, a questão fica ainda mais visível.&lt;/p&gt;

&lt;p&gt;Podemos pensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Kubernetes Pod
      │
      └── Executor
            │
            ├── JVM Heap
            ├── Non-Heap
            ├── Native
            ├── Python
            └── Overhead
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O Pod possui limites.&lt;/p&gt;

&lt;p&gt;Não basta a JVM achar que possui heap suficiente.&lt;/p&gt;

&lt;p&gt;O consumo total do processo e componentes associados precisa respeitar os recursos disponíveis.&lt;/p&gt;

&lt;p&gt;Esse é um dos motivos pelos quais problemas modernos de Spark precisam ser analisados também no nível da infraestrutura.&lt;/p&gt;




&lt;h2&gt;
  
  
  43. Memória em YARN
&lt;/h2&gt;

&lt;p&gt;Em ambientes YARN, existe raciocínio semelhante.&lt;/p&gt;

&lt;p&gt;O Executor executa dentro de recursos controlados pelo gerenciador do cluster.&lt;/p&gt;

&lt;p&gt;Historicamente, muitos erros de &lt;code&gt;memoryOverhead&lt;/code&gt; ficaram conhecidos justamente em ambientes onde o container ultrapassava seu limite mesmo quando a heap configurada parecia aceitável.&lt;/p&gt;

&lt;p&gt;O princípio permanece:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;JVM Heap
≠
Total Process Memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  44. A configuração depende do ambiente
&lt;/h2&gt;

&lt;p&gt;Hoje Spark pode rodar em diferentes contextos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Standalone
YARN
Kubernetes
Managed Platforms
Cloud Services
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso significa que recomendações de memória precisam considerar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Spark version;&lt;/li&gt;
&lt;li&gt;cluster manager;&lt;/li&gt;
&lt;li&gt;container limits;&lt;/li&gt;
&lt;li&gt;JVM;&lt;/li&gt;
&lt;li&gt;Python;&lt;/li&gt;
&lt;li&gt;workload;&lt;/li&gt;
&lt;li&gt;plataforma gerenciada.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Uma configuração copiada de um artigo antigo pode estar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;correta historicamente
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;inadequada atualmente
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ao mesmo tempo.&lt;/p&gt;

&lt;p&gt;Esse é exatamente o tipo de cuidado que estou aplicando ao revisitar meu material antigo nesta série.&lt;/p&gt;




&lt;h2&gt;
  
  
  45. Um modelo mental melhor para OOM
&lt;/h2&gt;

&lt;p&gt;Em vez de:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OOM
 ↓
More Memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;prefiro:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                    OOM
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
       Driver?               Executor?
          │                     │
          ▼                     ▼
      collect()?          Which Stage?
      toPandas()?              │
      metadata?                ▼
                         Which Operation?
                               │
                    ┌──────────┼──────────┐
                    ▼          ▼          ▼
                  Skew      Shuffle    Broadcast
                    │          │          │
                    └──────────┼──────────┘
                               ▼
                         Partition Size
                               │
                               ▼
                      Heap or Overhead?
                               │
                               ▼
                         Root Cause
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse modelo evita tratar todos os erros de memória como iguais.&lt;/p&gt;




&lt;h2&gt;
  
  
  46. Performance tuning começa com observabilidade
&lt;/h2&gt;

&lt;p&gt;Imagine dois engenheiros.&lt;/p&gt;

&lt;p&gt;O primeiro vê:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OutOfMemoryError
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e imediatamente altera:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spark.executor.memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O segundo investiga:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Stage
Tasks
Partition distribution
Shuffle
Spill
GC
Executor logs
Physical plan
Data volume
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;O segundo talvez leve alguns minutos a mais antes de alterar a configuração.&lt;/p&gt;

&lt;p&gt;Mas a chance de resolver a causa real é muito maior.&lt;/p&gt;

&lt;p&gt;Essa é uma diferença importante entre:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;configuration tuning
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;performance engineering
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  47. Um checklist para problemas de memória
&lt;/h2&gt;

&lt;p&gt;Quando encontro um problema de memória em Spark, quero responder:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;O erro aconteceu no Driver ou Executor?&lt;/li&gt;
&lt;li&gt;Qual Stage estava executando?&lt;/li&gt;
&lt;li&gt;Qual operação estava acontecendo?&lt;/li&gt;
&lt;li&gt;Existe &lt;code&gt;collect()&lt;/code&gt;?&lt;/li&gt;
&lt;li&gt;Existe &lt;code&gt;toPandas()&lt;/code&gt;?&lt;/li&gt;
&lt;li&gt;Existe broadcast?&lt;/li&gt;
&lt;li&gt;Existe cache excessivo?&lt;/li&gt;
&lt;li&gt;As partições estão equilibradas?&lt;/li&gt;
&lt;li&gt;Existe data skew?&lt;/li&gt;
&lt;li&gt;Qual o tamanho das maiores Tasks?&lt;/li&gt;
&lt;li&gt;Existe Shuffle?&lt;/li&gt;
&lt;li&gt;Quanto?&lt;/li&gt;
&lt;li&gt;Existe Spill?&lt;/li&gt;
&lt;li&gt;Quanto?&lt;/li&gt;
&lt;li&gt;Quanto tempo está sendo gasto em GC?&lt;/li&gt;
&lt;li&gt;Todas as Tasks sofrem ou apenas algumas?&lt;/li&gt;
&lt;li&gt;Existem Python workers?&lt;/li&gt;
&lt;li&gt;Existe consumo nativo relevante?&lt;/li&gt;
&lt;li&gt;O limite do container está sendo ultrapassado?&lt;/li&gt;
&lt;li&gt;O problema é heap ou overhead?&lt;/li&gt;
&lt;li&gt;A configuração corresponde à versão e ao cluster manager utilizados?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Só depois:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Qual configuração preciso alterar?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  48. O que eu evitaria fazer automaticamente
&lt;/h2&gt;

&lt;p&gt;Se o Job estiver lento ou falhando, eu evitaria começar por:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;dobrar executor memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;dobrar driver memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;aumentar memoryOverhead arbitrariamente
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cachear tudo
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;reduzir o número de partitions sem medir
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;aumentar o cluster
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Todas essas ações podem ser válidas.&lt;/p&gt;

&lt;p&gt;Mas nenhuma delas deveria substituir diagnóstico.&lt;/p&gt;




&lt;h2&gt;
  
  
  49. Memória, performance e custo
&lt;/h2&gt;

&lt;p&gt;Em cloud, existe ainda outra dimensão.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Executor Memory
8 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e simplesmente dobramos para:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;16 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;em dezenas ou centenas de Executors.&lt;/p&gt;

&lt;p&gt;Talvez o Job fique estável.&lt;/p&gt;

&lt;p&gt;Mas talvez tenhamos acabado de aumentar recursos para compensar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Data Skew
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;repartition ruim
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;join inadequado
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cache desnecessário
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Então temos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Problema arquitetural
       ↓
Mais infraestrutura
       ↓
Maior custo
       ↓
Problema mascarado
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Performance tuning também é FinOps.&lt;/p&gt;




&lt;h2&gt;
  
  
  50. A conexão entre tudo que vimos até aqui
&lt;/h2&gt;

&lt;p&gt;Agora podemos montar uma visão muito mais completa:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                    DATA
                      │
                      ▼
                 PARTITIONS
                      │
                      ▼
                    TASKS
                      │
                      ▼
                  EXECUTORS
                      │
          ┌───────────┼───────────┐
          ▼           ▼           ▼
       Memory       CPU        Network
          │                       │
          ▼                       ▼
      Aggregation              Shuffle
          │                       │
          └───────────┬───────────┘
                      ▼
                    Spill
                      │
                      ▼
                     Disk
                      │
                      ▼
                 Performance
                      │
                      ▼
                    Cost
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora adicione:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Data Skew
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e podemos gerar:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Uneven Partition
      ↓
Huge Task
      ↓
Memory Pressure
      ↓
Spill / GC
      ↓
Slow Task
      ↓
Straggler
      ↓
Slow Stage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Isso mostra por que os capítulos anteriores estavam conectados.&lt;/p&gt;




&lt;h2&gt;
  
  
  51. A principal lição deste capítulo
&lt;/h2&gt;

&lt;p&gt;Se eu pudesse resumir memória no Spark em uma frase:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Problemas de memória frequentemente são sintomas de como os dados e a computação foram distribuídos.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Nem sempre.&lt;/p&gt;

&lt;p&gt;Às vezes realmente precisamos de mais memória.&lt;/p&gt;

&lt;p&gt;Mas antes de aumentar recursos, quero saber:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Quem está consumindo?
        ↓
Onde?
        ↓
Durante qual operação?
        ↓
Por quê?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Só então:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Architecture
     +
Configuration
     +
Resources
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;entram na solução.&lt;/p&gt;




&lt;h2&gt;
  
  
  52. Como penso hoje
&lt;/h2&gt;

&lt;p&gt;Meu modelo mental deixou de ser:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark OOM
   ↓
Increase Memory
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;e passou a ser:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spark OOM
   ↓
Driver or Executor?
   ↓
Heap or Overhead?
   ↓
Which Stage?
   ↓
Which Task?
   ↓
Which Operation?
   ↓
Partition Size?
   ↓
Skew?
   ↓
Shuffle?
   ↓
Spill?
   ↓
Broadcast?
   ↓
Cache?
   ↓
Python / Native Memory?
   ↓
Root Cause
   ↓
Then Tune
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse raciocínio é muito mais útil do que decorar valores de configuração.&lt;/p&gt;




&lt;h2&gt;
  
  
  Próximo artigo
&lt;/h2&gt;

&lt;p&gt;Agora entendemos:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Partitions
    ↓
Tasks
    ↓
Memory
    ↓
Shuffle
    ↓
Spill
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mas existe outro conceito fundamental que apareceu desde os primeiros artigos:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lineage.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Quando uma partição é perdida, como Spark sabe reconstruí-la?&lt;/p&gt;

&lt;p&gt;Por que RDD significa:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Resilient Distributed Dataset
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;?&lt;/p&gt;

&lt;p&gt;Como Spark obtém tolerância a falhas sem simplesmente replicar toda computação intermediária?&lt;/p&gt;

&lt;p&gt;E o que acontece quando a linhagem fica grande demais?&lt;/p&gt;

&lt;p&gt;No próximo artigo vamos explorar:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;RDD Lineage;&lt;/li&gt;
&lt;li&gt;DAG;&lt;/li&gt;
&lt;li&gt;fault tolerance;&lt;/li&gt;
&lt;li&gt;recomputation;&lt;/li&gt;
&lt;li&gt;cache vs. recomputation;&lt;/li&gt;
&lt;li&gt;checkpoint;&lt;/li&gt;
&lt;li&gt;persistência;&lt;/li&gt;
&lt;li&gt;falha de Executor;&lt;/li&gt;
&lt;li&gt;recuperação de partições;&lt;/li&gt;
&lt;li&gt;por que Spark é "resilient";&lt;/li&gt;
&lt;li&gt;e como essas ideias evoluíram nas APIs modernas.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Vamos partir deste modelo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source
  ↓
Map
  ↓
Filter
  ↓
Join
  ↓
Aggregate
  ↓
Result
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;para enxergar algo mais profundo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;              LINEAGE
                 │
                 ▼
Source → Transform → Transform → Shuffle → Result
  │          │          │
  └──────────┴──────────┘
             │
      information needed
       for recomputation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Essa é uma das ideias fundamentais que fizeram o Spark ser Spark.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;← Parte 6:&lt;/strong&gt; Spark SQL e DataFrames — do código ao plano de execução&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Próximo:&lt;/strong&gt; Parte 8 — RDD Lineage e tolerância a falhas: como o Spark se recupera quando algo dá errado&lt;/p&gt;

</description>
      <category>cluster</category>
      <category>spark</category>
      <category>python</category>
      <category>bigdata</category>
    </item>
  </channel>
</rss>
