DEV Community: Rattlingshinuiba

Debug puppeteer in an interactive way

Rattlingshinuiba — Sun, 05 Jan 2020 14:15:24 +0000

Recently I play around with Puppeteer as an attempt to build a simple back-end service. Puppeteer is an awesome tool to emulate the browser's behaviors, which makes web scraping possible. It is sponsored by Google, and Javascript has more strong ties to HTML document(that is, the crawler's target) than Python does, though Python is a friendly language to get your feet wet.

As the title suggests, I would like to show you how to debug puppeteer just like the way we use Ipython or Jupyter notebooks.

I pretty like debug-driven learning which offers an overview of a particular object or function. For Javascript, there is console in the browser to meet my need, but when we run or debug puppeteer-based script, we might not know the way to access puppeteer's API in browser's console.

After I messed around with it, I discovered the node --inspect option which would add a node logo in devtools if you run your script with headless mode turned off.

Therefore, if you console.log(page)(page is an object representing the page in the browser) in the script, run it with --inspect option and click the node logo in the devtools, you would see the page object in the console and in turn you can access its API. For me, it is enough to learn puppeteer step by step.

Basically, the simple steps are as follows:

const puppeteer = require("puppeteer");
(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    console.log(page);
// And you can access the page object in the browser console if you follow the steps I specified above.
}
)();

Thanks for reading!

Google Keywords When I Learn Programming

Rattlingshinuiba — Wed, 27 Nov 2019 08:05:10 +0000

Sometimes you can not wrap your head around concepts from official doc when you just wanna try it out.

Sometimes you are confused when people are engaged in lively conversations in the post or comments.

Sometimes you google a topic and jargon pops up one after another.

Isn't there a simplified version of page where author talk with you, not explaining something with terminology from another world?

Seeking answers, I've been collecting these keywords whenever I wanna learn something without fuss.

Note that we can understand concepts more deeply in official doc after we grasp basics of them. So official doc is pretty important. After all, some docs are also awesome for us to learn something quickly.

p.s. Following list is not limited to programming.

Get an overview of a subject to get your feet wet

basics
introduction
starter/beginner
- get started in xxx
learn xxx in x minutes
learn xxx the right way
glossary/terminology
ipynb/notebook
hello world

Learn a concept quickly

explained
understand
⭐eli5
- Reddit has many eli5-related threads. So does dev.to!
visualize
- If you struggle to understand git commands, you can learn it visually.
- There is an awesome project for algorithm visualization.
layman
⭐gist

Miscellaneous

curated
best practices
project
hands-on
hand book
hitchhiker
intuitive
roadmap
guide
⭐blog
- There are a ton of personal blogs waiting for you to find!
⭐101
⭐cheatsheet
- You can revise something quickly with cheatsheet.
step by step
end to end
from scratch
template
boilerplate
review
crash course
infographic
snippets
kit

bonus-tips

Check out image tab in google search results to visualize something.
Make sure you know exactly what are you gonna search for before you dive into a concept.
⭐Type thank, work, or solve in CTRL+F input field in a community question thread.
- save my day
- awesome
kaggle and datacamp is awesome places to find nice tutorials for data science.
⭐xxx site:github.io

Explain Offset in CSS Like I'm Five

Rattlingshinuiba — Wed, 06 Nov 2019 03:13:24 +0000

Hello there! I have come across this term offset in CSS for many times. In my view, I treat it as a way of describing a element position in box model but I'm not sure whether it's accurate or not. I have googled it but every page seems to assume readers have grasped this concept. Also, I looked it up in dictionary but I hope there is a definition specifically for CSS. Forgive my asking but English is my second language.
Thanks in advance!

Thoughts after I rm my home directory

Rattlingshinuiba — Tue, 29 Oct 2019 08:09:57 +0000

Recently I have committed a big mistake: I rm my home directory. One good aspect of this bad situation is that I have backed up the code directory with borg. However, my hexo personal blog is not in the list. To make matters worse, most posts of this blog were stored locally.

The mess is attributed to a simple command: rm. Although rm are usually treated as a source of memes at which even newbies would smile knowingly, this worst case happened to me. Here is the command which cause a disaster: rm -rf * '/home/<user_name>/.local/share/Trash/files'.

Well, I thought I specified a folder to delete everything(*) in the absolute path(path in the single quote) and expected so. But my home directory was empty after that.

There is an another joke when we talk about rm: You should install adobe series in your filesystem because it would hold its own against rm for a while when you delete your / root stupidly.

So the conclusion is that you need to rm files in the current working directory. If not, be aware of *, like the path you specify in the single quote, is part of argument to tell rm to delete files. So the command should be rm -rf '/home/<user_name>/.local/share/Trash/files/*'. I learned this lesson the hard way, and hope you never encounter this kind of issue.

With hindsight, I think I ought to back up files on a regular basis.

A use case for decorator in Python: retry a function

Rattlingshinuiba — Mon, 28 Oct 2019 12:02:33 +0000

Life is short, use Python. Python is easy to learn for newbies like me who desire both functionality and performance without complex logic, until advanced concepts come in front of us including asynchronous programming, descriptor, decorator, among many others.

Admittedly, I have used these advanced features in the way of importing those awesome open source packages, but I didn't type them very much myself. It's OK to only import packages because we just want functionality and retrieve results without wasting time.

Beyond these basic needs, however, we sometimes also need to write the same function as the one from certain awesome package for the purpose of learning to deepen our understanding of programming principles behind these interfaces.

Let's jump right into the theme of today, decorator.

Let's take an example when we don't use decorator. There are times when you make a request for response from remote server but failed because your connection to network is not stable. So you might try, say, twice more.

for i in range(1,4):
    try:
        request_something()
    except:
        print('Bite the dust!') # try again
        continue
    else:
        break       
else:
    print('Killer Queen failed to do that') # we failed after 3 attempts

This will try 3 times when request_something() failed and print a message when it all failed. If you are a fan of jojo you would get the point of these messages but you can just pay attention to the comments if you are not.

But this code block is not good enough in that we have to duplicate the code around request_something whenever we make another request.

Decorator comes to rescue. In this case, we focus on one of its characteristic, reusability. Decorator comes in handy when you are going to modify your definition of a particular function.

Consider the following example:

def retry(request_sth):
    def wrapper():
        for i in range(1,4):
            try:
                request_sth()
            except:
                print('Bite the dust!') # try again
                continue
            else:
                break       
        else:
            print('Killer Queen failed to do that') # we failed after 3 attempts
    return wrapper

It would be much clearer if we use diff to see what changes are made:

+ def retry(request_sth):
+    def wrapper():
        for i in range(1,4):
            try:
                request_sth()
            except:
                print('Bite the dust!') # try again
                continue
            else:
                break       
        else:
            print('Killer Queen failed to do that') # we failed after 3 attempts
+       return wrapper

To summarize these changes:

We define a new function called retry and pass in a function, in this case, request_sth.
- It allows to use @retry syntax to modify a function.
We define inner function wrapper and return it.
- Why bother? We will talk about it soon.
The rest of code is the same as the first code block in this post.

To answer why we define a inner function, we can just test it:

def retry(request_sth):
-    def wrapper():
        for i in range(1,4):
            try:
                request_sth()
            except:
                print('Bite the dust!') # try again
                continue
            else:
                break       
        else:
            print('Killer Queen failed to do that') # we failed after 3 attempts
-       return wrapper 
+ @retry
+ def request_sth():
+   import requests
+   response = requests.get(<url>)

request_sth()

After you run it, it will work as expected until we hit the final line request_sth(), it will raise an error. The code would run normally If we didn't remove these lines I marked as red. Why?

Under the hood, to put it simply, when we hit lines from @retry to response = requests.get(<url>), an invisible line is executed: request_sth = retry(request_sth). If you want to dive deeper into it, I recommend checking out Simple Decorator section from an awesome article Primer on Python Decorators.

Reminder: Some lines in code block might not indent properly in diff view.