DEV Community: Reinaldo Assis

Multi-language Web Novel Scraper in Python

Reinaldo Assis — Fri, 24 May 2024 19:48:22 +0000

Introduction

TL;DR Today we will code a web scraper to scrap multiple sources of web novels, download chapters and combine them into a .epub file.

I like reading a specific web novel called Overgeared, it has more than 2000 chapters! I used to use a website that downloaded the web novel chapters for me and combined it all in a .epub file that I could send to my kindle. Unfortunately, the site went down and I was left with no choice but to code my own solution.

Overview: Here's the breakdown of the process we have to follow to achieve our goal:

Find sources
Identify ways to scrap each source
Write a Scraper Interface
Write a subclass for each source and implement the methods
Combine it all together and be happy reading my lil novel

Fiding Sources

Since I'm currently bettering my French and studying Italian I'd like to have sources other than English, fortunately Overgeared has translation to multiple languages so this step wasn't hard. Here are the sources I'll be using:

🇫🇷 French
🇺🇸 English
🇮🇹 Italian

Inspecting the structure of each source

Let's first start by taking a look at the French source since it poses an interesting challenge:

<ul class="lcp_catlist" id="lcp_instance_0">
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-16/">OG Chapitre 16</a></li>
   <li><a href="https://xiaowaz.fr/articles/overgeared-chapitre-1/">Overgeared Chapitre 1</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-2/">OG Chapitre 2</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-3/">OG Chapitre 3</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-4/">OG Chapitre 4</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-5/">OG Chapitre 5</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-6/">OG Chapitre 6</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-7/">OG Chapitre 7</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-8/">OG Chapitre 8</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-9/">OG Chapitre 9</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-10/">OG Chapitre 10</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-11/">OG Chapitre 11</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-12/">OG Chapitre 12</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-13/">OG Chapitre 13</a></li>
   <li><a href="https://xiaowaz.fr/articles/og-chapitre-14-rattrapage/">OG Chapitre 14 [Rattrapage]</a></li>
</ul>

If you take a look at the links for each chapter you will see that some of them are different, that means there's no pattern we can iterate on such as ".../og-chapitre-{chapter number we want}" (this link would work for some but not all of them). As such, we have to come up with a strategy, there are some options but I think the most straight forward one is to simply divide the scraping process into two steps:

First, we call the web novel home page and extract all of the available chapter's link into an array.
Second, we iterate in the generated array of links, extracting the text and combining into a book.

Generality

Another fun aspect we have to take into consideration is generality, in reality it is nearly impossible to write one code that can scrap any page we want, instead it's easier to write an interface class and implement the methods for scraping in each one. For example, both scrapers of French and English would have the same method scraper, but they would have different implementations.

The scraper Interface I wrote is quite big, but here are the main methods to get the idea across:


     def scraper(self, chapter_number : str, override_link: bool = False) -> chapter:
        """Responsible for requesting the html page and extracting the wanted text from it.

        :param int chapter_number: the number of the chapter to be requested and scraped.
        :param str language: default to english.
        """
        pass

    def create_book_from_chapters(self, book_name : str, out : str, chapters : List[chapter]):
        """Turns an array of chapters into an ebook."""
        pass

    def get_multiple_chapters(self, start: int, end : int) -> List[chapter]:
        pass

    def get_multiple_chapters_from_list(self, chapters: List[chapter]) -> List[chapter]:
        """In case the links from the source are not standard, this function can be used in
        conjunction with search_available_chapters, the chapters list containing the links will be
        downloaded and stored in a new list that can be used with create_book_from_chapters function."""
        pass

The ideal flow is
- [user input chapter range of download e.g. 100-200]
- [user select language i.e. FR/EN/IT]
- [calls function start(range)]

The start(range : str) function is responsible for calling the needed functions, for example, if the scraper in the selected language can't predict how the url works, it should use the function search_available_chapters to compile a list of links it can use to download chapters from.

In short, for each language we want to add we will create a class that implements the method of our interface. Here are some methods of the English implementation:

def scraper(self, chapter_number: str) -> chapter:
        try:
            response = requests.get(self._partial_link + chapter_number).text
        except:
            click.echo("[ERROR] Error while getting request.")
            click.echo(f"[ERROR] Request {self._partial_link+chapter_number} has failed.")
            raise self.FAILED_REQUEST

        soup = BeautifulSoup(response, 'html.parser')

        chr_content_div = soup.find('div', id='chr-content')

        text = ""

        if chr_content_div:
            paragraphs = chr_content_div.find_all('p')

            for p in paragraphs:
                text += f"<p>{p.get_text()}</p>\n\n"
        else:
            text = ""

        ch = chapter(text,f"Chapter {chapter_number}", "en", self.partial_link+chapter_number, len(text.split(" ")), chapter_number)

        return ch

 def create_book_from_chapters(self, book_name: str, out: str, chapters: List[chapter]):
        book = epub.EpubBook()
        book.set_identifier('overgeared')
        book.set_title('Overgeared Novel')
        book.set_language('en')
        book.add_author('Park Saenal')

        # Cover image
        book.set_cover("cover.jpg", open(self._cover_image_path, "rb").read())

        total_words = sum(ch.word_count for ch in chapters)

        info_page = epub.EpubHtml(title="Information", file_name="info.xhtml", lang='en')
        info_content = """
        <h1>Information</h1>
        <p>This is a collection of chapters compiled by me but all credits go to the author (Park Saenal) and translator (rainbowturtle).</p>
        <p>This book contains approximately {num_pages} pages.</p>
        """.format(num_pages=round(total_words/250))
        info_page.content = info_content
        book.add_item(info_page)
        book.toc.append((info_page, "Information"))

        for ch in chapters:
            book.add_item(ch.epub_info)
            book.toc.append((ch.epub_info, ch.title))

        # Define CSS
        style = 'body { font-family: Times, Times New Roman, serif; text-align: justify; }'
        nav_css = epub.EpubItem(uid="style_nav", file_name="style/nav.css", media_type="text/css", content=style)
        book.add_item(nav_css)

        book.add_item(epub.EpubNcx())
        book.add_item(epub.EpubNav())

        # Adds CSS to the book
        book.spine = [info_page, 'nav', *book.items]

        epub.write_epub(f'Overgeared {chapters[0].number} to {chapters.pop().number}.epub', book)

Combining all together

All that is left is to combine everything we've made, my plan is to use this module as a sort of "plugin" for my personal CLI (where I have a bunch of tools I've coded for myself), as such you can see here two important methods extension_name (tells the main program what is this extension's name) and start (called when the extension is selected).

def extension_name():
    return "Overgeared Ebook Novel"

def start():
    click.echo("Module: og.py")
    click.echo("Module Version: 0.0.1")
    click.echo("Created in: 14.05.24")
    click.echo("")
    lg = click.prompt("Language code: ")
    ch = click.prompt("Chapter to download: ")

    if lg == "en":
        en = OG_Novel_Downloader_EN()
        en.start(ch)
        # en.sanity_check(ch, verbose=True)

    if lg == "fr":
        fr = OG_Novel_Downloader_FR()
        fr.start(ch)

Usage

Now all we've to do is enjoy reading our novel in multiple languages 🥳.

Conclusion

I still struggle a bit with the concept of interfaces and reusable code, in hindsight I think my code could look a lot cleaner (maybe I'll refactor it?). Unfortunately, I won't be able to provide you guys with the source code this time, I don't have permission to scrap this sites much less share a code that does, but I hope this post was of some help and feel free to contact me if you have any suggestions or questions (my contacts can be found bellow ;).

That's it for today, thank you for reading so far!

About the Author

Computer Engineering Student making cool projects for fun (:

You can find more about me on my website (I’ll put the link here when available), TikTok, YouTube and Instagram.

Coding a manga downloader with NodeJS

Reinaldo Assis — Fri, 25 Mar 2022 20:43:46 +0000

Overview

I am currently studying French on my own and one of the strategies I deployed for that is reading mangas in French. But! French manga is something extremely hard to find - legally, that is. You see, the oficial manga publishers have little to no incentive on translating their mangas to French since the target audience is so scarce, it is different on the community side though, mangas are often translated to French by the community and - although technically illegal - are the only way to properly consume mangas in the language.

How

Alright, alright, we have seen the motivation. But how exactly can we do that? The technique I'm going to use here is the famous and controversial Web Scraping.

Web scraping is the process of collecting structured web data in an automated fashion. It’s also called web data extraction. Some of the main use cases of web scraping include price monitoring, price intelligence, news monitoring, lead generation, and market research among many others.
- Colm Kenny

The goal is simple: download individual images from a given manga of the French website sushi scan and convert them to PDF so that I can read it on my phone afterwards.

Structure

Our app should follow roughly something like this:

↳ Ask which manga we want to download and were to save it
↳ Connect to sushi scan and load all images from the manga
↳ Download all images from the given chapter/volume
↳ Convert all images to a single pdf file
↳ Done!

The Interface

Since it's a simple tool that is going to be used only by me, I'll not bother to implement a whole UI, a simple CLI (Command Line Interface) will suffice. For that I'll be using the following libraries: InquirerJS for the CLI itself and NanoSpinner for loading animation.

Main menu

The main menu consists of three options:

Download manga
Search mangas
Combine images into a single pdf

I'll focus only in the first one since it incorporates all of the others.

Downloading the manga

First, let's ask the user for some information:



await askLink();
await askDestination();
await askName();

These three functions are self-explanatory, they ask for the sushi scan manga link, were does the user want the manga to be saved, and give the final pdf a name. I'll show just one of the functions here, if you want to peek the whole code you can go to my repo here.



async function askLink() {
    const promptlink = await inquirer.prompt({
    name: "link",
    type: "input",
    message: "Chapter link",
    default() {
        return chaplink;
    },
 });

    chaplink = promptlink.link;
}

This is a basic example of who to use inquirerJS, again, I want to reinforce that I am not using best practices here - far from it - since it's just a simple and personal project. But I highly recommend checking the docs for how to properly use the library to its full potential.

Let's say I want to download the 14th volume of the manga Magi from this link. How does the program scrap the images? To achieve that, I'll be using the awesome scraping library Puppeteer.

First, we create a browser, puppeteer mimics a real browser to bypass anti-scraping strategies.



const brow = await puppeteer.launch();

Now, we can create a page, set a viewport and go to our target page.



const page = await brow.newPage();
page.setViewport({ width: 414, height: 896 });

//goes to our target page and waits for a delay
//chaplink -> chapter link
await page.goto(chaplink, { waitUntil: "load" });
await page.waitForTimeout(delay);

We loaded our page, awesome! Let's start scraping! The things is, web scraping - 90% of the time - has to be tailored to perform a specific task, so my solution - although useful for my use case - is probably worthless for other scenarios. Having said that, I'll just give you a quick look behind the code.



const spinFullPage = createSpinner("Loading images...").start();

//select reading mode to be page by page and not full scroling
await page.select("#readingmode", "single");
//wait for 500ms
await page.waitForTimeout(500);

//loading complete
spinFullPage.success({ text: "Loaded images." });

const spinImages = createSpinner("Processing images...").start();

//wait until page selector apears in the screen
await page.waitForSelector("#select-paged", { timeout: 100 });


//to scrap the images, we need to know how many pages there are in the chapter/volume, so we get the inner html from the page selector and count how many values there are inside it
let innerHTML = await page.$eval("#select-paged", (e) => {
 return e.innerHTML;
});

//this is the length of the chapter/volume
let tamanho = innerHTML.toString().split("value").length - 1;
 console.log(`TAMANHO ->> ${tamanho}`);

images = [];

//for each page of the manga, get the image link and store it in images
for (let i = 0; i < tamanho; i++) {
    let atual = await page.$eval("#readerarea > .ts-main-image", (e) => {
        return e.src;
    });

    images.push(atual);

    //push to downloads a promise, the link and where to save it (we got that from the user before)
    downloads.push(downloadImage(images[i], `./${out}/${name}-${i}.jpg`));

    //wait a while before passing to the next page and repeating
    await page.select("#select-paged", `${i + 2}`);
    await page.waitForTimeout(100);

 }


 spinImages.success({ text: "Got all the links." });

That's the gist of it, there's a lot more to it than that but I hope that gives you an idea on how it's done.

Creating the PDF

Now that we have all the image links, we just need to download all the images and combine them into a single pdf file. I accomplished that using the library PDFKit. Here's a quick look into how I add an image.



const manga = new PDFDocument({
    autoFirstPage: false,
    size: [960, 1481],
    compress: true,
 });

I recommend setting autoFirstPage to false, but that's up to you. The size is not always 960x1481, that's just the default I set it to, every time I download a manga I check it's size and set the pdf accordingly.



for (let i = 0; i < l; i++) {
    manga
    .addPage({ margin: 0, size: [width, height] })
    .image(`./${out}/${name}-${i}.jpg`, 0, 0, {
        height: manga.page.height,
    });

    if (autodelete)
    fs.unlinkSync(`./${out}/${name}-${i}.jpg`,         () => {});

 }

Adding the image is simple enough, we add a page, then an image to the page giving the file path and size. After adding the image, we delete it from disk automaticaly.

Conclusion

We have achieved our goal, we downloaded a manga from a website and turned it into a single pdf file! This is extremely helpful for my French studies and I hope it inspired you to do something related as well. I'll end here, as always, stay curious!

P.S: You can subscribe to my youtube channel for simmilar content and to my other social medias at my website.