<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Matheus Soares</title>
    <description>The latest articles on DEV Community by Matheus Soares (@mrsoares).</description>
    <link>https://dev.to/mrsoares</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4248136%2F82175b88-702c-4464-9d8a-cc4d5722dacb.png</url>
      <title>DEV Community: Matheus Soares</title>
      <link>https://dev.to/mrsoares</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mrsoares"/>
    <language>en</language>
    <item>
      <title>Um modelo de 1.7B consegue igualar um LLM 4,7x maior? Destilando o Qwen3-8B com QLoRA em avaliações reais em PT-BR</title>
      <dc:creator>Matheus Soares</dc:creator>
      <pubDate>Mon, 16 Nov 2026 16:39:21 +0000</pubDate>
      <link>https://dev.to/mrsoares/um-modelo-de-17b-consegue-igualar-um-llm-47x-maior-destilando-o-qwen3-8b-com-qlora-em-avaliacoes-41b4</link>
      <guid>https://dev.to/mrsoares/um-modelo-de-17b-consegue-igualar-um-llm-47x-maior-destilando-o-qwen3-8b-com-qlora-em-avaliacoes-41b4</guid>
      <description>&lt;p&gt;&lt;em&gt;&lt;a href="https://dev.to/mrsoares/can-a-17b-model-match-a-47x-larger-llm-distilling-qwen3-8b-with-qlora-on-real-brazilian-n66"&gt;Read this article in English&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Todo time de atendimento de e-commerce tem o mesmo problema: milhares de avaliações de clientes em texto livre e nenhum tempo para ler. Quais falam de atraso na entrega? Quais clientes estão irritados? Quais precisam de alguém agindo hoje?&lt;/p&gt;

&lt;p&gt;Eu queria saber se um modelo de linguagem &lt;em&gt;pequeno&lt;/em&gt; consegue fazer essa triagem tão bem quanto um bem maior, por uma fração do custo. Então montei um experimento controlado com avaliações reais em português, um conjunto de teste rotulado à mão, intervalos de confiança e orçamento zero.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Resposta curta:&lt;/strong&gt; um Qwen3-1.7B ajustado com QLoRA empata com um Qwen3-8B few-shot em dois dos três campos, perde 4 pontos de acurácia em sentimento e roda 3,7× mais rápido em lote. O experimento inteiro rodou nas GPUs T4 grátis do Kaggle; a preço de nuvem sob demanda, custaria cerca de US$ 1,30.&lt;/p&gt;

&lt;p&gt;As descobertas mais interessantes estão nos detalhes: onde os modelos erram, e o campo em que um simples baseline TF-IDF é estatisticamente tão bom quanto os dois LLMs.&lt;/p&gt;

&lt;h2&gt;
  
  
  A tarefa
&lt;/h2&gt;

&lt;p&gt;Cada avaliação vira uma linha de JSON:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"categoria"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"nao_recebido"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"sentimento"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"negativo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"urgencia"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"alta"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Campo&lt;/th&gt;
&lt;th&gt;Valores&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;categoria&lt;/code&gt; (problema principal)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;atraso_entrega&lt;/code&gt;, &lt;code&gt;nao_recebido&lt;/code&gt;, &lt;code&gt;produto_errado_ou_incompleto&lt;/code&gt;, &lt;code&gt;defeito_ou_qualidade&lt;/code&gt;, &lt;code&gt;atendimento_vendedor&lt;/code&gt;, &lt;code&gt;elogio_sem_problema&lt;/code&gt;, &lt;code&gt;outro&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;sentimento&lt;/code&gt; (tom geral)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;positivo&lt;/code&gt;, &lt;code&gt;neutro&lt;/code&gt;, &lt;code&gt;negativo&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;urgencia&lt;/code&gt; (a empresa precisa agir?)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;alta&lt;/code&gt;, &lt;code&gt;media&lt;/code&gt;, &lt;code&gt;baixa&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Um guia de rotulagem escrito define cada classe e as regras de desempate: se a avaliação cita vários problemas, vale o mais grave; urgência &lt;code&gt;alta&lt;/code&gt; só quando o cliente pede uma ação.&lt;/p&gt;

&lt;h2&gt;
  
  
  A montagem
&lt;/h2&gt;

&lt;p&gt;Os dados são o &lt;a href="https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce" rel="noopener noreferrer"&gt;dataset público de e-commerce da Olist&lt;/a&gt;: 99.224 avaliações, 40.977 com comentário e 36.709 depois de remover duplicatas. Os textos são curtos (mediana de 62 caracteres), informais e cheios de erros de digitação.&lt;/p&gt;

&lt;p&gt;Comparei três abordagens:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Abordagem&lt;/th&gt;
&lt;th&gt;O que é&lt;/th&gt;
&lt;th&gt;Treinada com&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A. Baseline&lt;/td&gt;
&lt;td&gt;TF-IDF + regressão logística, um classificador por campo&lt;/td&gt;
&lt;td&gt;5.000 rótulos silver&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B. Professor&lt;/td&gt;
&lt;td&gt;Qwen3-8B em 4-bit, prompt de sistema + 6 exemplos (few-shot)&lt;/td&gt;
&lt;td&gt;nada&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C. Aluno&lt;/td&gt;
&lt;td&gt;Qwen3-1.7B com QLoRA, só o prompt de sistema&lt;/td&gt;
&lt;td&gt;5.000 rótulos silver&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fk6x0l6c1l4n9mdsr4lci.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fk6x0l6c1l4n9mdsr4lci.png" alt="Pipeline: as avaliações da Olist se dividem em 300 gold rotuladas à mão e 5.000 sem rótulo; o professor rotula as 5.000, o baseline e o aluno treinam com esses rótulos, e os três são avaliados no teste gold" width="800" height="264"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A decisão mais importante do projeto foi sobre os rótulos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Gold:&lt;/strong&gt; 300 avaliações que eu rotulei à mão, &lt;em&gt;às cegas&lt;/em&gt; (só o texto, sem a nota em estrelas e sem a saída de nenhum modelo), divididas em 100 de dev e 200 de teste.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Silver:&lt;/strong&gt; 5.000 outras avaliações rotuladas pelo professor.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;O baseline e o aluno aprendem com os &lt;em&gt;mesmos&lt;/em&gt; rótulos silver. Isso faz do aluno uma destilação do professor e garante que a comparação isola o modelo, não os dados. Os três são julgados contra os rótulos gold humanos, nunca contra o silver: medir o aluno no mesmo tipo de rótulo com que ele treinou e chamar isso de acurácia é o jeito mais fácil de se enganar.&lt;/p&gt;

&lt;p&gt;Mais um detalhe: o gold e o silver têm a mesma quantidade de avaliações por nota. Nos dados brutos, 48% das avaliações são 5 estrelas, então as reclamações estão super-representadas de propósito. As métricas descrevem uma mistura balanceada, não o tráfego real de produção.&lt;/p&gt;

&lt;h2&gt;
  
  
  A disciplina de avaliação
&lt;/h2&gt;

&lt;p&gt;É isto que separa um experimento de uma demo.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;O conjunto de teste foi usado uma única vez&lt;/strong&gt;, depois que todas as decisões (prompt, modelo, rodada de treino) estavam congeladas. Toda a iteração aconteceu nas 100 avaliações de dev.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Intervalos de confiança por bootstrap pareado.&lt;/strong&gt; Com 200 avaliações de teste, cada avaliação vale 0,5 ponto de acurácia, e diferenças pequenas podem ser ruído. Reamostrei o teste 2.000 vezes (as mesmas avaliações para todas as abordagens) e calculei um IC de 95% para cada diferença.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Um único parser&lt;/strong&gt; para as três abordagens, então "JSON válido" significa a mesma coisa em todas.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checagens contra fatos que nenhum rotulador viu.&lt;/strong&gt; A nota em estrelas e as datas reais de entrega ficaram escondidas de mim e dos modelos. Por isso servem como checagem independente, inclusive nas 5.000 avaliações silver, que não têm referência humana.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Resultados
&lt;/h2&gt;

&lt;p&gt;Teste gold, n = 200. Acurácia / macro-F1:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Abordagem&lt;/th&gt;
&lt;th&gt;categoria&lt;/th&gt;
&lt;th&gt;sentimento&lt;/th&gt;
&lt;th&gt;urgencia&lt;/th&gt;
&lt;th&gt;JSON válido&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A. Baseline TF-IDF&lt;/td&gt;
&lt;td&gt;0,775 / 0,70&lt;/td&gt;
&lt;td&gt;0,795 / 0,73&lt;/td&gt;
&lt;td&gt;0,745 / 0,74&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B. Professor Qwen3-8B&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;0,870&lt;/strong&gt; / &lt;strong&gt;0,78&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;0,840&lt;/strong&gt; / &lt;strong&gt;0,77&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;0,820 / 0,81&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C. Aluno Qwen3-1.7B&lt;/td&gt;
&lt;td&gt;0,855 / 0,77&lt;/td&gt;
&lt;td&gt;0,800 / 0,69&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;0,825&lt;/strong&gt; / &lt;strong&gt;0,82&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A tabela sozinha convida a tirar conclusões demais. Estas são as diferenças com IC de 95% (em negrito, quando o intervalo exclui o zero):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Diferença de acurácia (pontos)&lt;/th&gt;
&lt;th&gt;categoria&lt;/th&gt;
&lt;th&gt;sentimento&lt;/th&gt;
&lt;th&gt;urgencia&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Professor − aluno&lt;/td&gt;
&lt;td&gt;+1,5 [−1,5; +4,5]&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+4,0 [+1,0; +7,5]&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;−0,5 [−4,0; +3,0]&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Professor − baseline&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+9,5 [+4,0; +15,0]&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+4,5 [−1,5; +10,5]&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+7,5 [+2,0; +13,0]&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Aluno − baseline&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+8,0 [+3,0; +13,5]&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+0,5 [−4,5; +5,5]&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+8,0 [+3,0; +13,5]&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;O que isso diz:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;O aluno empata com o professor em categoria e urgência.&lt;/strong&gt; A única perda clara é em sentimento.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Os dois LLMs batem o TF-IDF por 7 a 10 pontos em categoria e urgência.&lt;/strong&gt; São campos que exigem alguma compreensão do texto.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Em sentimento, o TF-IDF é estatisticamente tão bom quanto qualquer um dos LLMs.&lt;/strong&gt; Essa foi a surpresa. Mais capacidade de modelo rende pouco aqui, e a análise de erros abaixo mostra por quê: o gargalo é como &lt;code&gt;neutro&lt;/code&gt; foi definido e rotulado, não o tamanho do modelo.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;O aluno concorda com o professor em 93–94% das avaliações de teste em cada campo, e nenhuma saída deixou de virar JSON válido (zero falhas em 300 gold e 5.000 silver), sem geração restrita (&lt;em&gt;constrained decoding&lt;/em&gt;).&lt;/p&gt;

&lt;h3&gt;
  
  
  Velocidade e custo
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Professor 8B&lt;/th&gt;
&lt;th&gt;Aluno 1.7B&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Hardware&lt;/td&gt;
&lt;td&gt;1× T4, 4-bit&lt;/td&gt;
&lt;td&gt;1× T4, 4-bit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vazão em lote&lt;/td&gt;
&lt;td&gt;0,71 avaliação/s&lt;/td&gt;
&lt;td&gt;2,6 avaliações/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Latência, 1 avaliação (p50)&lt;/td&gt;
&lt;td&gt;3,82 s&lt;/td&gt;
&lt;td&gt;3,09 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Custo de GPU por 1.000 avaliações*&lt;/td&gt;
&lt;td&gt;US$ 0,137&lt;/td&gt;
&lt;td&gt;US$ 0,037&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;* Só o tempo de GPU, ao preço sob demanda de US$ 0,35 por hora de T4 no Google Cloud. As execuções em si usaram a cota grátis do Kaggle. Custos únicos no mesmo preço: rotular as 5.000 avaliações silver com o professor, US$ 0,69; treinar o aluno (96 minutos), US$ 0,56.&lt;/p&gt;

&lt;p&gt;Uma ressalva que eu não esperava: &lt;strong&gt;a latência de uma requisição isolada quase não muda&lt;/strong&gt; (3,8 s → 3,1 s). Numa T4, uma resposta de uns 30 tokens é dominada pelo custo fixo de gerar cada token, não pelo tamanho do modelo. Parte do ganho em lote também vem do prompt mais curto: o ajuste fino substitui os exemplos few-shot do professor. O modelo pequeno compensa em processamento em lote, não em requisições uma a uma.&lt;/p&gt;

&lt;h2&gt;
  
  
  Onde os modelos erram
&lt;/h2&gt;

&lt;p&gt;Li todas as avaliações de teste em que pelo menos um modelo errou algum campo (84 de 200). Alguns padrões apareceram várias vezes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Atalhos de palavra-chave.&lt;/strong&gt; Os modelos reagem a palavras, não ao sentido:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Recebi antes do prqzo" (com o erro de digitação original)&lt;/p&gt;

&lt;p&gt;Gold: &lt;code&gt;elogio_sem_problema&lt;/code&gt; · Professor: &lt;strong&gt;&lt;code&gt;atraso_entrega&lt;/code&gt;&lt;/strong&gt; · Aluno: &lt;strong&gt;&lt;code&gt;atraso_entrega&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A palavra "prazo" puxa para atraso na entrega mesmo quando o pacote chegou antes. O aluno acertou uma avaliação parecida e caiu em outra: a destilação copia o viés do professor, não cada resposta individual.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;neutro&lt;/code&gt; é a classe mais difícil.&lt;/strong&gt; Das 40 avaliações neutras do teste, o professor acerta 16 e o aluno só 10. Confirmações simples como "Recebi corretamente." viram &lt;code&gt;positivo&lt;/code&gt;; pedidos calmos como "Gostaria de um vale troca! Aguardando número de postagem." viram &lt;code&gt;negativo&lt;/code&gt;. Seis das oito avaliações por trás da diferença de 4 pontos do aluno em sentimento estão aqui: &lt;strong&gt;ele aprendeu o viés do professor e o ampliou.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Essa fronteira é ambígua de verdade; dá para defender &lt;code&gt;positivo&lt;/code&gt; para "chegou como prometido". É um problema do guia de rotulagem tanto quanto dos modelos, e é exatamente por isso que o TF-IDF acompanha os LLMs em sentimento.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A urgência sobe pelo tom.&lt;/strong&gt; A regra diz &lt;code&gt;alta&lt;/code&gt; só quando o cliente pede uma ação. Cerca de 20 das 68 avaliações de urgência &lt;code&gt;media&lt;/code&gt; são previstas como &lt;code&gt;alta&lt;/code&gt; por cada modelo. "Quase um mês de atraso na entrega" é grave, mas não pede nada. Numa fila de atendimento, o erro é para o lado seguro, mas a fila prevista de urgência alta fica uns 45% maior que a real.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;O viés negativo aparece em escala.&lt;/strong&gt; A checagem contra as estrelas confirma isso nas 5.000 avaliações silver, sem nenhum humano no processo: o professor rotula avaliações de 4–5 estrelas como &lt;code&gt;negativo&lt;/code&gt; em 15% dos casos, contra 9% nos meus rótulos gold. No geral, o sentimento ainda acompanha bem as estrelas (média de 2,1 estrelas para &lt;code&gt;negativo&lt;/code&gt; e 4,4 para &lt;code&gt;positivo&lt;/code&gt;), então o silver é sólido no agregado, só inclinado.&lt;/p&gt;

&lt;h2&gt;
  
  
  O que não funcionou
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Meu primeiro prompt era ruim em &lt;code&gt;neutro&lt;/code&gt;.&lt;/strong&gt; Acertou 1 de 8 avaliações neutras no dev (macro-F1 de 0,69 em sentimento). Reescrever a definição de neutro com frases concretas de exemplo levou a 0,82 no dev. O teste mostra que o problema só foi resolvido em parte.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Não testei um aluno menor.&lt;/strong&gt; O plano era escolher entre dois tamanhos com um teste rápido no dev. Para poupar cota de GPU, fiquei com o Qwen3-1.7B sem testar o Qwen3-0.6B, que talvez fosse suficiente em categoria e urgência com um terço do tamanho.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Uma rodada de treino, sem busca de hiperparâmetros.&lt;/strong&gt; Foi uma escolha consciente, mas significa que os números do aluno são um piso, não um teto.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A demo pública foi cortada.&lt;/strong&gt; Planejei um Space grátis no Hugging Face, mas os Spaces com Gradio passaram a exigir plano pago. A demo roda localmente.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Um único rotulador.&lt;/strong&gt; Uma segunda passada às cegas em 30 avaliações gold bateu com a primeira em todas as 30. Parece ótimo, mas foi feita no mesmo dia (a memória ajuda), com 30 avaliações o limite inferior de 95% fica em torno de 88%, e isso mede consistência, não acerto. Relendo, pelo menos um rótulo gold parece errado, e eu teria rotulado igual nas duas vezes. O teste de verdade seria um segundo rotulador independente.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Lições
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;O teto do aluno é o professor.&lt;/strong&gt; O aluno não consegue ser muito melhor que os rótulos com que aprende. Rótulos silver melhores (autoconsistência do professor, ou algumas centenas de rótulos humanos misturados) provavelmente ajudariam mais do que qualquer ajuste de hiperparâmetro.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Definição de classe importa mais que tamanho de modelo.&lt;/strong&gt; A maior lacuna do experimento inteiro (sentimento &lt;code&gt;neutro&lt;/code&gt;) não se resolve com um modelo maior. Precisa de uma definição mais precisa e de rótulos melhores.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sempre rode o baseline sem graça.&lt;/strong&gt; Sem o TF-IDF na tabela, eu teria concluído que "LLMs são melhores em sentimento" e estaria errado.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mostre intervalos, não só números pontuais.&lt;/strong&gt; Quatro das nove diferenças da tabela de acurácia estão dentro do ruído. O bootstrap é barato e muda as conclusões.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Modelos pequenos ganham em vazão, não em latência.&lt;/strong&gt; Se a sua carga é uma requisição por vez numa T4, um modelo 4,7× menor economiza menos do que parece.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Código, divisões dos dados e todas as previsões: &lt;a href="https://github.com/SoaresCRF/ai-engineer-portfolio/tree/main/review-triage-llm" rel="noopener noreferrer"&gt;github.com/SoaresCRF/ai-engineer-portfolio/tree/main/review-triage-llm&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Modelo (adaptador LoRA): &lt;a href="https://huggingface.co/SoaresCRF/qwen3-1.7b-review-triage-ptbr-lora" rel="noopener noreferrer"&gt;huggingface.co/SoaresCRF/qwen3-1.7b-review-triage-ptbr-lora&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffan2xu9msmfu5lfju0w1.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffan2xu9msmfu5lfju0w1.gif" alt="Demo local em Gradio: uma avaliação entra e o aluno devolve categoria, sentimento e urgência em JSON" width="799" height="260"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;As avaliações e os rótulos derivados vêm do dataset da Olist (CC BY-NC-SA 4.0), então os rótulos gold e o adaptador são publicados para uso não comercial, com atribuição.&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>llm</category>
      <category>python</category>
      <category>nlp</category>
    </item>
    <item>
      <title>Can a 1.7B model match a 4.7x larger LLM? Distilling Qwen3-8B with QLoRA on real Brazilian Portuguese reviews</title>
      <dc:creator>Matheus Soares</dc:creator>
      <pubDate>Mon, 16 Nov 2026 16:35:02 +0000</pubDate>
      <link>https://dev.to/mrsoares/can-a-17b-model-match-a-47x-larger-llm-distilling-qwen3-8b-with-qlora-on-real-brazilian-n66</link>
      <guid>https://dev.to/mrsoares/can-a-17b-model-match-a-47x-larger-llm-distilling-qwen3-8b-with-qlora-on-real-brazilian-n66</guid>
      <description>&lt;p&gt;&lt;em&gt;&lt;a href="https://dev.to/mrsoares/um-modelo-de-17b-consegue-igualar-um-llm-47x-maior-destilando-o-qwen3-8b-com-qlora-em-avaliacoes-41b4"&gt;Leia este artigo em português&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Every e-commerce support team has the same problem: thousands of free-text customer reviews and no time to read them. Which ones are about a late delivery? Which customers are angry? Which ones need someone to act today?&lt;/p&gt;

&lt;p&gt;I wanted to know whether a &lt;em&gt;small&lt;/em&gt; language model could do that triage as well as a much larger one, at a fraction of the cost. So I set up a controlled experiment with real Brazilian Portuguese reviews, a hand-labeled test set, confidence intervals and a budget of zero.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; a Qwen3-1.7B fine-tuned with QLoRA ties a few-shot Qwen3-8B on two of three fields, loses 4 accuracy points on sentiment, and runs 3.7× faster in batch. The whole experiment ran on Kaggle's free T4 GPUs; at on-demand cloud prices it would cost about $1.30.&lt;/p&gt;

&lt;p&gt;The most interesting findings are in the details: where the models fail, and the field where a plain TF-IDF baseline is statistically as good as both LLMs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The task
&lt;/h2&gt;

&lt;p&gt;Each review becomes one line of JSON:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"categoria"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"nao_recebido"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"sentimento"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"negativo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"urgencia"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"alta"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;Values&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;categoria&lt;/code&gt; (main problem)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;atraso_entrega&lt;/code&gt; (late delivery), &lt;code&gt;nao_recebido&lt;/code&gt; (not received), &lt;code&gt;produto_errado_ou_incompleto&lt;/code&gt; (wrong or incomplete product), &lt;code&gt;defeito_ou_qualidade&lt;/code&gt; (defect or quality), &lt;code&gt;atendimento_vendedor&lt;/code&gt; (seller service), &lt;code&gt;elogio_sem_problema&lt;/code&gt; (praise, no problem), &lt;code&gt;outro&lt;/code&gt; (other)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;sentimento&lt;/code&gt; (overall tone)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;positivo&lt;/code&gt;, &lt;code&gt;neutro&lt;/code&gt;, &lt;code&gt;negativo&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;urgencia&lt;/code&gt; (does the company need to act?)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;alta&lt;/code&gt; (high), &lt;code&gt;media&lt;/code&gt; (medium), &lt;code&gt;baixa&lt;/code&gt; (low)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Keys and labels stay in Portuguese on purpose: they are the data contract a Brazilian product team would consume. A written labeling guide defines each class and the tie-break rules: if a review mentions several problems, the most severe wins; &lt;code&gt;alta&lt;/code&gt; urgency only when the customer asks for an action.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The data is the &lt;a href="https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce" rel="noopener noreferrer"&gt;Olist public e-commerce dataset&lt;/a&gt;: 99,224 reviews, 40,977 with a comment, and 36,709 after removing duplicates. The texts are short (median of 62 characters), informal and full of typos.&lt;/p&gt;

&lt;p&gt;I compared three approaches:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Approach&lt;/th&gt;
&lt;th&gt;What it is&lt;/th&gt;
&lt;th&gt;Trained on&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A. Baseline&lt;/td&gt;
&lt;td&gt;TF-IDF + logistic regression, one classifier per field&lt;/td&gt;
&lt;td&gt;5,000 silver labels&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B. Teacher&lt;/td&gt;
&lt;td&gt;Qwen3-8B in 4-bit, system prompt + 6 examples (few-shot)&lt;/td&gt;
&lt;td&gt;nothing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C. Student&lt;/td&gt;
&lt;td&gt;Qwen3-1.7B with QLoRA, system prompt only&lt;/td&gt;
&lt;td&gt;5,000 silver labels&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F57vm3xiqr0sj3lohlvmv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F57vm3xiqr0sj3lohlvmv.png" alt="Pipeline: Olist reviews split into 300 hand-labeled gold reviews and 5,000 unlabeled ones; the teacher labels the 5,000, the baseline and the student train on those labels, and all three are scored on the gold test set" width="800" height="264"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The most important decision in the project was about labels:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Gold:&lt;/strong&gt; 300 reviews I labeled by hand, &lt;em&gt;blind&lt;/em&gt; (text only, no star rating and no model output), split into 100 dev and 200 test.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Silver:&lt;/strong&gt; 5,000 other reviews labeled by the teacher.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The baseline and the student learn from the &lt;em&gt;same&lt;/em&gt; silver labels. That makes the student a distillation of the teacher and ensures the comparison isolates the model, not the data. All three are judged against the human gold labels, never against silver: measuring the student on the same kind of labels it trained on and calling it accuracy is the easiest way to fool yourself.&lt;/p&gt;

&lt;p&gt;One more detail: gold and silver have the same number of reviews per star rating. In the raw data, 48% of reviews are 5 stars, so complaints are over-represented on purpose. The metrics describe a balanced mix, not real production traffic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Evaluation discipline
&lt;/h2&gt;

&lt;p&gt;This is what separates an experiment from a demo.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The test set was used exactly once&lt;/strong&gt;, after every decision (prompt, model, training run) was frozen. All iteration happened on the 100 dev reviews.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paired bootstrap confidence intervals.&lt;/strong&gt; With 200 test reviews, each review is worth 0.5 accuracy points, and small differences can be noise. I resampled the test set 2,000 times (the same reviews for every approach) and computed a 95% CI for each difference.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One shared parser&lt;/strong&gt; for all three approaches, so "valid JSON" means the same thing for each.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checks against facts no labeler saw.&lt;/strong&gt; The star rating and the real delivery dates were hidden from me and from the models. That makes them an independent check, even on the 5,000 silver reviews, which have no human reference.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;Gold test set, n = 200. Accuracy / macro-F1:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Approach&lt;/th&gt;
&lt;th&gt;categoria&lt;/th&gt;
&lt;th&gt;sentimento&lt;/th&gt;
&lt;th&gt;urgencia&lt;/th&gt;
&lt;th&gt;Valid JSON&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A. TF-IDF baseline&lt;/td&gt;
&lt;td&gt;0.775 / 0.70&lt;/td&gt;
&lt;td&gt;0.795 / 0.73&lt;/td&gt;
&lt;td&gt;0.745 / 0.74&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B. Qwen3-8B teacher&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;0.870&lt;/strong&gt; / &lt;strong&gt;0.78&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;0.840&lt;/strong&gt; / &lt;strong&gt;0.77&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;0.820 / 0.81&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;C. Qwen3-1.7B student&lt;/td&gt;
&lt;td&gt;0.855 / 0.77&lt;/td&gt;
&lt;td&gt;0.800 / 0.69&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;0.825&lt;/strong&gt; / &lt;strong&gt;0.82&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The table alone invites over-reading. Here are the differences with 95% CIs (bold when the interval excludes zero):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Difference in accuracy (points)&lt;/th&gt;
&lt;th&gt;categoria&lt;/th&gt;
&lt;th&gt;sentimento&lt;/th&gt;
&lt;th&gt;urgencia&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Teacher − student&lt;/td&gt;
&lt;td&gt;+1.5 [−1.5, +4.5]&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+4.0 [+1.0, +7.5]&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;−0.5 [−4.0, +3.0]&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Teacher − baseline&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+9.5 [+4.0, +15.0]&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+4.5 [−1.5, +10.5]&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+7.5 [+2.0, +13.0]&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Student − baseline&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+8.0 [+3.0, +13.5]&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+0.5 [−4.5, +5.5]&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+8.0 [+3.0, +13.5]&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;What this says:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The student ties the teacher on category and urgency.&lt;/strong&gt; Its only clear loss is sentiment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Both LLMs beat TF-IDF by 7 to 10 points on category and urgency.&lt;/strong&gt; Those fields require some understanding of the text.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On sentiment, TF-IDF is statistically as good as either LLM.&lt;/strong&gt; That was the surprise. Extra model capacity buys little here, and the error analysis below shows why: the bottleneck is how &lt;code&gt;neutro&lt;/code&gt; was defined and labeled, not model size.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The student agrees with the teacher on 93–94% of test reviews in each field, and not a single output failed to parse as JSON (zero failures across 300 gold and 5,000 silver reviews), without constrained decoding.&lt;/p&gt;

&lt;h3&gt;
  
  
  Speed and cost
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Teacher 8B&lt;/th&gt;
&lt;th&gt;Student 1.7B&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Hardware&lt;/td&gt;
&lt;td&gt;1× T4, 4-bit&lt;/td&gt;
&lt;td&gt;1× T4, 4-bit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Batched throughput&lt;/td&gt;
&lt;td&gt;0.71 reviews/s&lt;/td&gt;
&lt;td&gt;2.6 reviews/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Latency, 1 review (p50)&lt;/td&gt;
&lt;td&gt;3.82 s&lt;/td&gt;
&lt;td&gt;3.09 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPU cost per 1,000 reviews*&lt;/td&gt;
&lt;td&gt;$0.137&lt;/td&gt;
&lt;td&gt;$0.037&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;* GPU time only, at Google Cloud's on-demand price of $0.35 per T4-hour. The runs themselves used Kaggle's free quota. One-off costs at the same price: labeling the 5,000 silver reviews with the teacher, $0.69; training the student (96 minutes), $0.56.&lt;/p&gt;

&lt;p&gt;A caveat I did not expect: &lt;strong&gt;single-request latency barely moves&lt;/strong&gt; (3.8 s → 3.1 s). On a T4, a ~30-token answer is dominated by the fixed cost of generating each token, not by model size. Part of the batch gain also comes from the shorter prompt: fine-tuning replaces the teacher's few-shot examples. The small model pays off in batch processing, not in one-at-a-time requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the models fail
&lt;/h2&gt;

&lt;p&gt;I read every test review where at least one model got a field wrong (84 of 200). A few patterns came up again and again.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Keyword shortcuts.&lt;/strong&gt; The models react to words, not meaning:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Recebi antes do prqzo" (&lt;em&gt;"Got it before the deadline"&lt;/em&gt;, original typo included)&lt;/p&gt;

&lt;p&gt;Gold: &lt;code&gt;elogio_sem_problema&lt;/code&gt; · Teacher: &lt;strong&gt;&lt;code&gt;atraso_entrega&lt;/code&gt;&lt;/strong&gt; · Student: &lt;strong&gt;&lt;code&gt;atraso_entrega&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The word &lt;em&gt;prazo&lt;/em&gt; (deadline) pulls toward late delivery even when the package arrived early. The student got a similar review right and fell for another one: distillation copies the teacher's bias, not each individual answer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;neutro&lt;/code&gt; is the hardest class.&lt;/strong&gt; Of the 40 neutral test reviews, the teacher gets 16 right and the student only 10. Plain confirmations like "Recebi corretamente." (&lt;em&gt;"Received it correctly."&lt;/em&gt;) become &lt;code&gt;positivo&lt;/code&gt;; calm requests like "Gostaria de um vale troca! Aguardando número de postagem." (&lt;em&gt;"I'd like an exchange voucher! Waiting for the shipping number."&lt;/em&gt;) become &lt;code&gt;negativo&lt;/code&gt;. Six of the eight reviews behind the student's 4-point sentiment gap are here: &lt;strong&gt;it learned the teacher's bias and amplified it.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This boundary is genuinely ambiguous; you could defend &lt;code&gt;positivo&lt;/code&gt; for "it arrived as promised". It is a labeling-guide problem as much as a model problem, and that is exactly why TF-IDF keeps up with the LLMs on sentiment.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Urgency rises with tone.&lt;/strong&gt; The rule says &lt;code&gt;alta&lt;/code&gt; only when the customer asks for an action. About 20 of the 68 &lt;code&gt;media&lt;/code&gt; urgency reviews are predicted &lt;code&gt;alta&lt;/code&gt; by each model. "Quase um mês de atraso na entrega" (&lt;em&gt;"Almost a month late"&lt;/em&gt;) is serious, but it asks for nothing. In a support queue the error leans to the safe side, but the predicted high-urgency queue ends up about 45% larger than the real one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The negativity bias shows up at scale.&lt;/strong&gt; The star-rating check confirms it on the 5,000 silver reviews, with no human in the loop: the teacher labels 4–5 star reviews &lt;code&gt;negativo&lt;/code&gt; 15% of the time, against 9% in my gold labels. Overall, sentiment still tracks the stars well (a mean of 2.1 stars for &lt;code&gt;negativo&lt;/code&gt; and 4.4 for &lt;code&gt;positivo&lt;/code&gt;), so the silver labels are sound in aggregate, just tilted.&lt;/p&gt;

&lt;h2&gt;
  
  
  What did not work
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;My first prompt was bad at &lt;code&gt;neutro&lt;/code&gt;.&lt;/strong&gt; It got 1 of 8 neutral dev reviews right (0.69 macro-F1 on sentiment). Rewriting the neutral definition with concrete example phrasings brought it to 0.82 on dev. The test set shows the problem was only partly solved.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;I did not try a smaller student.&lt;/strong&gt; The plan was to choose between two sizes with a quick dev test. To save GPU quota, I went with Qwen3-1.7B without trying Qwen3-0.6B, which might have been good enough on category and urgency at a third of the size.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One training run, no hyperparameter search.&lt;/strong&gt; That was a deliberate choice, but it means the student's numbers are a floor, not a ceiling.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The public demo got cut.&lt;/strong&gt; I planned a free Hugging Face Space, but Gradio Spaces now require a paid plan. The demo runs locally.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A single labeler.&lt;/strong&gt; A second blind pass over 30 gold reviews matched the first on all 30. That sounds great, but it was done the same day (memory helps), with 30 reviews the 95% lower bound is around 88%, and it measures consistency, not correctness. On re-reading, at least one gold label looks wrong, and I would have labeled it the same way both times. The real test would be a second, independent labeler.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Lessons
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The student's ceiling is the teacher.&lt;/strong&gt; The student cannot be much better than the labels it learns from. Better silver labels (teacher self-consistency, or a few hundred human labels mixed in) would likely help more than any hyperparameter tuning.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Class definitions matter more than model size.&lt;/strong&gt; The biggest gap in the whole experiment (&lt;code&gt;neutro&lt;/code&gt; sentiment) is not solved by a bigger model. It needs a sharper definition and better labels.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Always run the boring baseline.&lt;/strong&gt; Without TF-IDF in the table, I would have concluded that "LLMs are better at sentiment" and been wrong.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Report intervals, not just point estimates.&lt;/strong&gt; Four of the nine differences in the accuracy table are within noise. The bootstrap is cheap and it changes the conclusions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small models win on throughput, not latency.&lt;/strong&gt; If your workload is one request at a time on a T4, a 4.7× smaller model saves less than you would think.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Code, data splits and every prediction: &lt;a href="https://github.com/SoaresCRF/ai-engineer-portfolio/tree/main/review-triage-llm" rel="noopener noreferrer"&gt;github.com/SoaresCRF/ai-engineer-portfolio/tree/main/review-triage-llm&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Model (LoRA adapter): &lt;a href="https://huggingface.co/SoaresCRF/qwen3-1.7b-review-triage-ptbr-lora" rel="noopener noreferrer"&gt;huggingface.co/SoaresCRF/qwen3-1.7b-review-triage-ptbr-lora&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffan2xu9msmfu5lfju0w1.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffan2xu9msmfu5lfju0w1.gif" alt="Local Gradio demo: a review goes in and the student returns category, sentiment and urgency as JSON" width="799" height="260"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The reviews and derived labels come from the Olist dataset (CC BY-NC-SA 4.0), so the gold labels and the adapter are released for non-commercial use, with attribution.&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>llm</category>
      <category>python</category>
      <category>nlp</category>
    </item>
  </channel>
</rss>
