DEV Community

Discussion on: Crawling Simply with Scrapy

Collapse
 
yelluw profile image
Pablo Rivera

Great post! I hope you don't mind if I suggest a little tweak to the code. :)

This method:

def parse_item(self, response):
    x = HtmlXPathSelector(response)
    filename = "output.txt"
    open(filename, 'ab').write(response.url + "\n")

Enter fullscreen mode Exit fullscreen mode

To this:

def parse_item(self, response):
    """
    Parses response and saves
    response.url to file output.txt
    """
    x = HtmlXPathSelector(response)

    # this makes sure the file is closed
    # when the context of the operation
    # is no more.

    # f.writeline() writes each value in a newline
    # no need to format the string. 

    with open("output.txt", "ab") as f:
        f.writeline(response.url)
Enter fullscreen mode Exit fullscreen mode

Happy coding!