DEV Community

Cover image for What is Nokogiri?
Jesse vB
Jesse vB

Posted on


What is Nokogiri?


A Nokogiri is a Japanese pull saw. Oh... didn't help? Keep reading.

The Gem

Nokogiri comes standard in the Gemfiles of Rails apps. I first noticed it because it showed up in a Bundler error.

Nokogiri is a low-dependency module written in C, Java, and Ruby that parses XML and HTML so that these protocols can be used by Ruby. Let's get started.

Using Nokogiri to Parse HTML

Do you want to use Ruby to extract data from a webpage?

First run nokogiri -v and see if you get a response with a verion number. If not, you'll have to run gem i nokogiri

➜  ~ irb --simple-prompt
>> require 'nokogiri'
=> true
>> Nokogiri
=> Nokogiri
Enter fullscreen mode Exit fullscreen mode

Don't forget to require it, otherwise you'll get a NameError for an uninitialized constant.

First, we will start by parsing HTML we create, then we will parse an actual webpage.

# First, put some valid html into a string.
>> html = "<h1>Hello World!</h1>
<section><h2>A List of Stuff</h2>
# Now pass that string into the HTML5 module as an argument
>> document = Nokogiri::HTML5(html)
Enter fullscreen mode Exit fullscreen mode

You should get a response like this

#(Document:0x4c4 {
  name = "document",
  children = [
    #(Element:0x4d8 {
      name = "html",
      children = [
        #(Element:0x4ec { name = "head" }),
        #(Element:0x500 {
          name = "body",
          children = [
            #(Element:0x514 {
              name = "h1",
              children = [ #(Text "Hello World!")]
            #(Element:0x528 {
              name = "section",
              children = [
                #(Element:0x53c {
                  name = "h2",
                  children = [ #(Text "A List of Stuff")]
                #(Element:0x550 {
                  name = "ul",
                  children = [
                    #(Element:0x564 {
                      name = "li",
                      children = [ #(Text "Camera")]
                    #(Element:0x578 {
                      name = "li",
                      children = [ #(Text "Computer")]
                    #(Element:0x58c {
                      name = "li",
                      children = [ #(Text "Television")]
Enter fullscreen mode Exit fullscreen mode

Nokogiri's HTML5 module took our HTML string and returned a Document class to us that has it's own set of methods. Let's look at a couple. (Remember you can always list all the methods by using Document.methods.sort.)


This will will return an array-like XML::NodeSet. It only has a length of one because it's the <html>...</html> element that we never included in our string. This child has two children, the <head>...</head> and the <body>...</body> tags.

>> body = document.children.children.last
#(Element:0x80c {
>> ul = body.children.children.children
=> [#<Nokogiri::XML::Text:0x8c0 "A List of Stuff">, #<Nokogiri::XML::Elemen...
?> => [#<Nokogiri::XML::Text:0x8c0 "A List of Stuff">, #<Nokogiri::XML::Elemen..
>> >> ul.children.length
>> => 3
?> ?> ul.children.each do |child|
>> ?>   puts child.text
>> >> end
>> Camera
>> Computer
>> Television
Enter fullscreen mode Exit fullscreen mode

Parsing HTML from the Internet

I've heard the Internet is a good place to find HTML documents to parse. You can capture some of this HTML by using the Client URL command in your terminal. is a great resource to learn about all Ruby objects.

Capture it like this.

➜  ~ curl ''
Enter fullscreen mode Exit fullscreen mode

While it's cool to see the HTML populate the terminal, it's even better when it comes back as a Ruby object for us to use the data. To do so, let's use Nokogiri!

# You'll need the open-uri gem in addition to Nokogiri
# Open-uri allows you to open a URI as if it was a file in Ruby

>> require 'nokogiri'
=> true
>> require 'open-uri'
=> true

>> doc = Nokogiri::HTML5(''))
#(Document:0x9998 {
Enter fullscreen mode Exit fullscreen mode

Now we can use the #css method to query the html by elements and classes.

>> puts doc.css('h1')
<h1><a href="/"></a></h1>
=> nil
>> puts doc.css('p').first
<p>Help and documentation for the Ruby programming language.</p>
Enter fullscreen mode Exit fullscreen mode

Query class this way.

>> rails = Nokogiri::HTML5(URI(''))
#(Document:0x1f1bc {
>> headline = rails.css('div.heading__headline')
=> [#<Nokogiri::XML::Element:0x1fb94 name="div" attributes=[#<Nokogiri::XML...
>> puts headline.text

        Compress the complexity of modern web apps.
        Learn just what you need to get started, then keep leveling up as you go. Ruby on Rails scales from HELLO WORLD to IPO.

        Youre in good company.
        Over the past two decades, Rails has taken countless companies to millions of users and billions in market valuations.

        Building it together.

        Over six thousand people have contributed code to Rails, and many more have served the community through evangelism, documentation, and bug reports. Join us!

        Everything you need.
        Rails is a full-stack framework. It ships with all the tools needed to build amazing web apps on both the front and back end.

        Optimized for happiness.

        Lets get started.
Enter fullscreen mode Exit fullscreen mode

Great! Now you have a general idea of how Nokogiri works!

And now for free bonus content...

Let's search a webpage's content. For instance, it's basketball season at the time of this writing, so let's see if the homepage to ESPN talks about basketball.

>> espn = Nokogiri::HTML5(URI(''))
#(Document:0x259e0 {
# For this we will use a Regular Expression (RegEx)
# The Regular Expression goes between forward slashes
# And the 'i' flag makes the search case-insensitive
>> espn.text.scan(/basketball/i).count
=> 32
Enter fullscreen mode Exit fullscreen mode

There you have it! And we didn't even have to visit the website to find out that ESPN refers to basketball 32 times on it's homepage! Wow...

Top comments (0)

An Animated Guide to Node.js Event Loop

Node.js doesn’t stop from running other operations because of Libuv, a C++ library responsible for the event loop and asynchronously handling tasks such as network requests, DNS resolution, file system operations, data encryption, etc.

What happens under the hood when Node.js works on tasks such as database queries? We will explore it by following this piece of code step by step.