<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: lenevo02487</title>
    <description>The latest articles on DEV Community by lenevo02487 (@lenevo02487).</description>
    <link>https://dev.to/lenevo02487</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4222156%2F4383d380-2d8d-4906-8418-130083f18c2f.png</url>
      <title>DEV Community: lenevo02487</title>
      <link>https://dev.to/lenevo02487</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/lenevo02487"/>
    <language>en</language>
    <item>
      <title>GST Invoice Math in Pure Python — CGST, SGST, IGST Explained with Code</title>
      <dc:creator>lenevo02487</dc:creator>
      <pubDate>Mon, 02 Nov 2026 11:06:49 +0000</pubDate>
      <link>https://dev.to/lenevo02487/gst-invoice-math-in-pure-python-cgst-sgst-igst-explained-with-code-2kl9</link>
      <guid>https://dev.to/lenevo02487/gst-invoice-math-in-pure-python-cgst-sgst-igst-explained-with-code-2kl9</guid>
      <description>&lt;p&gt;If you've ever built billing software for India, you've hit the question: &lt;strong&gt;when does GST split into CGST + SGST, and when is it a single IGST?&lt;/strong&gt; Get it wrong and the invoice is wrong — even if the total looks right.&lt;/p&gt;

&lt;p&gt;I built a small GST invoice generator in pure Python (standard library only) to get this math exactly right. Here's the logic, with a real worked example whose grand total is &lt;strong&gt;₹15,522&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one rule that decides everything
&lt;/h2&gt;

&lt;p&gt;Compare two state codes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Seller's state code == place-of-supply state code → intra-state&lt;/strong&gt; → tax splits in half: &lt;strong&gt;CGST + SGST&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;They differ → inter-state&lt;/strong&gt; → one single tax: &lt;strong&gt;IGST&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's it. The customer's address doesn't decide it — the &lt;em&gt;place of supply&lt;/em&gt; does (for goods, usually where they're delivered). In code:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;intra&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;shop_state_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;place_of_supply_state_code&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Per-line math
&lt;/h2&gt;

&lt;p&gt;For each item: &lt;code&gt;taxable = qty × rate&lt;/code&gt;, then &lt;code&gt;tax = taxable × GST%&lt;/code&gt;. The split is where people slip:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;decimal&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ROUND_HALF_UP&lt;/span&gt;

&lt;span class="n"&gt;D&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Decimal&lt;/span&gt;
&lt;span class="n"&gt;CENT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;D&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;0.01&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Round to 2 decimals, half-up (the way shops actually bill).&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;D&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;quantize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;CENT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rounding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ROUND_HALF_UP&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;taxable&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;qty&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;tax&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;taxable&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;gst_percent&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;intra&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;cgst&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;money&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tax&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;sgst&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tax&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;cgst&lt;/span&gt;          &lt;span class="c1"&gt;# &amp;lt;-- the important line
&lt;/span&gt;    &lt;span class="n"&gt;igst&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;D&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;cgst&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sgst&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;D&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;igst&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tax&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things to notice:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Never use floats for money.&lt;/strong&gt; &lt;code&gt;0.1 + 0.2 != 0.3&lt;/code&gt; in floating point, and those errors &lt;em&gt;will&lt;/em&gt; surface on an invoice. &lt;code&gt;Decimal&lt;/code&gt; with explicit half-up rounding is how real billing works.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. The odd-paisa trick.&lt;/strong&gt; Splitting ₹111.75 in half gives ₹55.875 — not a real amount. Round one half up (CGST ₹55.88) and set the other half to &lt;code&gt;tax − cgst&lt;/code&gt; (SGST ₹55.87), so the two halves always add back to the exact tax. If you round both halves independently, invoices drift by a paisa and accountants come looking for you.&lt;/p&gt;

&lt;h2&gt;
  
  
  The worked example (real output)
&lt;/h2&gt;

&lt;p&gt;An intra-state invoice — seller and place of supply in the same state — with five items at three different GST rates:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Item&lt;/th&gt;
&lt;th&gt;Qty&lt;/th&gt;
&lt;th&gt;Rate&lt;/th&gt;
&lt;th&gt;GST&lt;/th&gt;
&lt;th&gt;Taxable&lt;/th&gt;
&lt;th&gt;Tax&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;CTC Tea (1 kg)&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;₹420&lt;/td&gt;
&lt;td&gt;5%&lt;/td&gt;
&lt;td&gt;₹4,200.00&lt;/td&gt;
&lt;td&gt;₹210.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Earthen Cups (pack of 12)&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;₹60&lt;/td&gt;
&lt;td&gt;12%&lt;/td&gt;
&lt;td&gt;₹1,200.00&lt;/td&gt;
&lt;td&gt;₹144.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chai Masala Combo Pack&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;₹149&lt;/td&gt;
&lt;td&gt;5%&lt;/td&gt;
&lt;td&gt;₹2,235.00&lt;/td&gt;
&lt;td&gt;₹111.75&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Electric Kettle 1.8L&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;₹1,299&lt;/td&gt;
&lt;td&gt;18%&lt;/td&gt;
&lt;td&gt;₹3,897.00&lt;/td&gt;
&lt;td&gt;₹701.46&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LED Bulb 9W (set of 4)&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;₹299&lt;/td&gt;
&lt;td&gt;18%&lt;/td&gt;
&lt;td&gt;₹2,392.00&lt;/td&gt;
&lt;td&gt;₹430.56&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Because it's intra-state, every tax above splits into CGST + SGST halves — including the chai masala's ₹111.75, which becomes CGST ₹55.88 + SGST ₹55.87 (spot the odd-paisa trick doing its job).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tax summary by rate&lt;/strong&gt; (this grouping is what goes on the invoice and into GST returns):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;GST rate&lt;/th&gt;
&lt;th&gt;Taxable value&lt;/th&gt;
&lt;th&gt;CGST&lt;/th&gt;
&lt;th&gt;SGST&lt;/th&gt;
&lt;th&gt;Total tax&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;5% (2.5% + 2.5%)&lt;/td&gt;
&lt;td&gt;₹6,435.00&lt;/td&gt;
&lt;td&gt;₹160.88&lt;/td&gt;
&lt;td&gt;₹160.87&lt;/td&gt;
&lt;td&gt;₹321.75&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12% (6% + 6%)&lt;/td&gt;
&lt;td&gt;₹1,200.00&lt;/td&gt;
&lt;td&gt;₹72.00&lt;/td&gt;
&lt;td&gt;₹72.00&lt;/td&gt;
&lt;td&gt;₹144.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;18% (9% + 9%)&lt;/td&gt;
&lt;td&gt;₹6,289.00&lt;/td&gt;
&lt;td&gt;₹566.01&lt;/td&gt;
&lt;td&gt;₹566.01&lt;/td&gt;
&lt;td&gt;₹1,132.02&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Totals:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Subtotal (taxable value)   ₹13,924.00
Total GST                  ₹ 1,597.77
Grand total                ₹15,521.77  →  rounded ₹15,522 (round-off +₹0.23)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the same five items were supplied to a customer in a &lt;em&gt;different&lt;/em&gt; state, nothing about the taxable values changes — the same ₹1,597.77 simply appears as IGST instead of the CGST/SGST split, and the total stays ₹15,522. Same money, different tax heads — which is exactly why getting the split right matters for filing, not for the customer.&lt;/p&gt;

&lt;h2&gt;
  
  
  One more detail: the round-off
&lt;/h2&gt;

&lt;p&gt;Invoices are paid in whole rupees, so the generator rounds the grand total (&lt;code&gt;₹15,521.77 → ₹15,522&lt;/code&gt;) and shows the &lt;code&gt;+₹0.23&lt;/code&gt; round-off as its own line. Hiding the difference inside an item rate is how totals stop reconciling — display it openly instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  Full code
&lt;/h2&gt;

&lt;p&gt;The complete generator — JSON or CSV in, print-ready HTML invoice out, with a per-rate tax summary and the amount in words (Indian numbering: lakh/crore) — is on GitHub: &lt;strong&gt;&lt;a href="https://github.com/lenevo02487/gst-invoice-generator" rel="noopener noreferrer"&gt;gst-invoice-generator&lt;/a&gt;&lt;/strong&gt;. Run it with the bundled sample and you'll get exactly the ₹15,522 invoice above.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Demo first — on YOUR sample — before you pay anything.&lt;/strong&gt; Need invoice automation, spreadsheet cleanup, or a scraping/automation tool for your business? Send me a sample of your data and I'll show you a working demo first.&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://www.fiverr.com/lenevo509/build-a-python-web-scraper-or-automation-script-for-you" rel="noopener noreferrer"&gt;My Python web scraping &amp;amp; automation gig on Fiverr&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;— Abhishek, Python Automation Developer&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>tutorial</category>
      <category>gst</category>
      <category>india</category>
    </item>
    <item>
      <title>I Scraped a 1,000-Book Catalogue with Python — Full Walkthrough</title>
      <dc:creator>lenevo02487</dc:creator>
      <pubDate>Mon, 02 Nov 2026 11:04:52 +0000</pubDate>
      <link>https://dev.to/lenevo02487/i-scraped-a-1000-book-catalogue-with-python-full-walkthrough-2f8i</link>
      <guid>https://dev.to/lenevo02487/i-scraped-a-1000-book-catalogue-with-python-full-walkthrough-2f8i</guid>
      <description>&lt;p&gt;Every web scraping tutorial stops at "here's how to grab one page." Real catalogues don't fit on one page — so for this practice project I scraped the &lt;strong&gt;entire&lt;/strong&gt; books.toscrape.com catalogue: 1,000 books across 50 pages, into one clean CSV with title, price, star rating, and stock status.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://books.toscrape.com" rel="noopener noreferrer"&gt;books.toscrape.com&lt;/a&gt; exists specifically for scraping practice, which makes it the perfect place to learn the full workflow — pagination included. Here's exactly how it works, with the real code.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we're building
&lt;/h2&gt;

&lt;p&gt;A script that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Opens the catalogue's first page&lt;/li&gt;
&lt;li&gt;Extracts every book card on the page&lt;/li&gt;
&lt;li&gt;Follows the "next" link until there isn't one&lt;/li&gt;
&lt;li&gt;Writes everything to &lt;code&gt;books.csv&lt;/code&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Final output (real rows from the run):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;title,price_gbp,rating,availability
A Light in the Attic,51.77,3,In stock
Tipping the Velvet,53.74,1,In stock
Soumission,50.10,1,In stock
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Setup
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;requests beautifulsoup4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's it. No browser automation, no Selenium — the site serves plain HTML, so &lt;code&gt;requests&lt;/code&gt; + &lt;code&gt;BeautifulSoup&lt;/code&gt; is enough. Reaching for a headless browser when the HTML is right there is the most common beginner mistake I see.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Understand one book card
&lt;/h2&gt;

&lt;p&gt;Every book on the site lives in an &lt;code&gt;&amp;lt;article class="product_pod"&amp;gt;&lt;/code&gt; element. Inside it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the &lt;strong&gt;title&lt;/strong&gt; is in the &lt;code&gt;title&lt;/code&gt; attribute of the &lt;code&gt;&amp;lt;h3&amp;gt;&amp;lt;a&amp;gt;&lt;/code&gt; link (the visible text is truncated with "…", the attribute has the full title — grab the attribute!)&lt;/li&gt;
&lt;li&gt;the &lt;strong&gt;price&lt;/strong&gt; is in &lt;code&gt;.price_color&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;the &lt;strong&gt;rating&lt;/strong&gt; is a word — One to Five — hidden in the class list of &lt;code&gt;.star-rating&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;availability&lt;/strong&gt; is the text of &lt;code&gt;.availability&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;RATING_WORDS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;One&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Two&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Three&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Four&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Five&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;parse_book&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Pull title, price, rating and availability out of one book card.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;h3&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;price&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.price_color&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;get_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;rating_word&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.star-rating&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;class&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;availability&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.availability&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;get_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price_gbp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;price&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;£&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rating&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;RATING_WORDS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rating_word&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;availability&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;availability&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 2: Follow the pagination
&lt;/h2&gt;

&lt;p&gt;Instead of guessing page URLs (&lt;code&gt;page-2.html&lt;/code&gt;, &lt;code&gt;page-3.html&lt;/code&gt;…), let the site tell you where to go next. Each page has a &lt;code&gt;li.next a&lt;/code&gt; link — follow it until it disappears:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;bs4&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BeautifulSoup&lt;/span&gt;

&lt;span class="n"&gt;BASE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://books.toscrape.com/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;scrape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_pages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;books&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="n"&gt;page_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;BASE&lt;/span&gt;
    &lt;span class="n"&gt;page_no&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Session&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0 (demo scraper)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;page_url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;page_no&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;encoding&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;  &lt;span class="c1"&gt;# see lesson 1 below
&lt;/span&gt;
        &lt;span class="n"&gt;soup&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;BeautifulSoup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;html.parser&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;article&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;soup&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;article.product_pod&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;books&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;parse_book&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;page &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;page_no&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;books&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; books so far&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;max_pages&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;page_no&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;max_pages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;break&lt;/span&gt;
        &lt;span class="n"&gt;next_link&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;soup&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;li.next a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;page_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;next_link&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;href&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;next_link&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;page_url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# be polite to the server
&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;books&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details matter here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;urljoin&lt;/code&gt;&lt;/strong&gt; — the next-page href is relative (&lt;code&gt;catalogue/page-2.html&lt;/code&gt;), and &lt;code&gt;urljoin&lt;/code&gt; resolves it correctly against whatever page you're on. Concatenating strings by hand breaks the moment the site structure changes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The small sleep&lt;/strong&gt; — 0.3s between pages costs you 15 seconds over 50 pages and keeps you a good citizen. On real projects this matters even more.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: Write the CSV
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;

&lt;span class="n"&gt;books&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;scrape&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;books.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;writer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DictWriter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fieldnames&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price_gbp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rating&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;availability&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeheader&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;books&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Done: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;books&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; books written to books.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output: &lt;code&gt;Done: 1000 books written to books.csv&lt;/code&gt; — exactly 1,000, which matches the catalogue's own count. Always check your row count against a number the site itself reports; "it ran without errors" is not the same as "it got everything."&lt;/p&gt;

&lt;h2&gt;
  
  
  3 lessons this project teaches
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Encoding will bite you.&lt;/strong&gt; The site is UTF-8, but &lt;code&gt;requests&lt;/code&gt; sometimes guesses Latin-1 from the headers — and the &lt;code&gt;£&lt;/code&gt; sign turns into &lt;code&gt;Â£&lt;/code&gt; in your CSV. One line (&lt;code&gt;resp.encoding = "utf-8"&lt;/code&gt;) fixes it. I found this the fun way: by opening the CSV.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Attributes beat visible text.&lt;/strong&gt; The visible title is cut off ("A Light in the …"); the full title sits in the &lt;code&gt;title&lt;/code&gt; attribute. When scraped data looks truncated, inspect the element before blaming your code.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Ratings hide in class names.&lt;/strong&gt; There's no "3 stars" text anywhere — the rating is a CSS class. Real sites hide data in classes, data attributes, and JSON blobs all the time; reading the raw HTML is a core scraping skill.&lt;/p&gt;

&lt;h2&gt;
  
  
  Full code
&lt;/h2&gt;

&lt;p&gt;The complete script (with a &lt;code&gt;--pages&lt;/code&gt; flag for quick tests) is on GitHub: &lt;strong&gt;&lt;a href="https://github.com/lenevo02487/python-web-scraper" rel="noopener noreferrer"&gt;python-web-scraper&lt;/a&gt;&lt;/strong&gt; — the repo includes the actual &lt;code&gt;books.csv&lt;/code&gt; output with all 1,000 rows.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Demo first — on YOUR sample — before you pay anything.&lt;/strong&gt; That's how I work: if you need a catalogue, price list, or directory scraped, send me the site and a few rows of the output you want, and I'll build a small working demo on your sample before you spend anything.&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://www.fiverr.com/lenevo509/build-a-python-web-scraper-or-automation-script-for-you" rel="noopener noreferrer"&gt;My Python web scraping &amp;amp; automation gig on Fiverr&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;— Abhishek, Python Automation Developer&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>tutorial</category>
      <category>beginners</category>
    </item>
  </channel>
</rss>
