DEV Community: Hans

How to make a simple webcrawler with JAVA ….(and jsoup)

Hans — Mon, 17 May 2021 09:05:30 +0000

While Python is arguably the numero uno language to use when it comes to webscraping, good ole JAVA has it’s perks. At least for a JAVA developer like me who hasn’t quite yet delved in Python. If you are in a hurry, dont’t worry. The complete code is found at the end of this post.

Anywho, I wanted to figure out how to make a webcrawler w/JAVA, just for the lulz really. Turns out. It was way easier than expected. First of all you need to download jsoup(that is, you need to start a new JAVA project as well)

Link

Now as soon as IntelliJ has done its magic making your project you put the downloaded jsoup .jar file in the project root.

Now time to add som nice programming principles right? imports needed for this project are:


import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import java.io.IOException;
import java.util.ArrayList;

Next — its constant’s time. I chose to index links (hrefs) on the CNN website. I chose CNN bc i am not a big fan of FOX news.


public class Crawler {
    public static final String CNN = "https://edition.cnn.com/";
public static void main(String[] args) {
System.out.println("Web Crawler ")
 }
}

And then we need two methods utilising a cluster methods from the jsoup library
First one is a recursive method which indexes said page’s “a href=”´s. To not make the the method ramble on in infinity like a good recursive method will do I chose to make it stop two levels down.

private static void crawl (int level, String url, ArrayList<String> visited) {
    if(level <=2 ) {
        Document doc = request(url, visited);
        if (doc!= null) {
            for (Element link : doc.select("a[href]")) {
                 String next_link = link.absUrl("href");
                 if(visited.contains(next_link) == false) {
                     crawl(level++, next_link, visited);
                 }
            }
        }
    }
}

And then we need a Document method that checks for connection and returns a string of “Link “ + the url that is indexed.

private static Document request(String url, ArrayList<String> v) {
    try {
        Connection con = Jsoup.connect(url);
        Document doc = con.get();
        if(con.response().statusCode() == 200) {
            System.out.println("Link: " + url);
            System.out.println(doc.title());
            v.add(url);
            return doc;
        }
        return null;
    } catch (IOException e) {
        return null;
    }
}

Finally the full code is ready to be run now

package.com.company
import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;


public class Crawler {
    public static final String CNN = "https://edition.cnn.com/";


    public static void main(String[] args) {
      String url = CNN;
        crawl(1, url, new ArrayList<String>());

    }

    private static void crawl (int level, String url, ArrayList<String> visited) {
        if(level <=2 ) {
            Document doc = request(url, visited);
            if (doc!= null) {
                for (Element link : doc.select("a[href]")) {
                     String next_link = link.absUrl("href");
                     if(visited.contains(next_link) == false) {
                         crawl(level++, next_link, visited);
                     }
                }
            }
        }
    }
    private static Document request(String url, ArrayList<String> v) {
        try {
            Connection con = Jsoup.connect(url);
            Document doc = con.get();
            if(con.response().statusCode() == 200) {
                System.out.println("Link: " + url);
                System.out.println(doc.title());
                v.add(url);
                return doc;
            }
            return null;
        } catch (IOException e) {
            return null;
        }
    }
}

Cheers
https://boreatech.medium.com/

Workstation 2.0

Hans — Sat, 13 Mar 2021 20:15:14 +0000

Difference between react props vs. state

Hans — Sun, 28 Feb 2021 08:45:23 +0000

One of the core concepts of react is the difference between props and state. Only changes in props and state trigger react to re-render components and update the DOM.

The biggest difference is that re-rendering of the component based on input in state is done entirely within the component, whereas you with using props can receive new data from outside the component and re-render.

Props

props allows you to pass data from a parent component to a child component.

//Parent Component
const books = () => {
    return (
<div>
 <Book title = "Data structures and algorithms with JAVA" />
 </div>
   );
}
//Child component
const book = (props) => {
    return ( 
        <div>
            <h1>{props.title}</h1>
        </div>
    )
}

Explanation: Now. ‘props’ is passed in to the child component and the functional component the passes the ‘props’ as an argument which in turn will be handled as an object. The property ‘title’ is accessible in the child component from the parent component.

State

Only class-based react components can define and use state. It’s although possible to pass state to a functional component but functional components can’t edit them directly.

class NewBook extends Component {
    state = {
        number: ''
    };
    render() {
        return ( 
            <div>{this.state.number}</div>
        );
    }
}

As you can see the NewBook component contains a defined state. This state is accessible through this.state.number and can be returned in the render() method.

Connect to postgresdatabase with JAVA

Hans — Thu, 16 Jan 2020 06:48:07 +0000

Feel free to put variables and dbConnection() in another class.

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.swing.JOptionPane;

Need library Postgresql -> installed in library
Simple connection to postgres
https://www.tutorialspoint.com/postgresql/postgresql_java.htm */

public class DatabaseConnection {

Connection connection;
String url = "jdbc:postgresql://localhost:5432/databasename";
String user = "username";
String password = "yourpassword";

public static void main(String[] args) throws ClassNotFoundException {
    DatabaseConnection db = new DatabaseConnection();
    db.dbConnection();
}

public Connection dbConnection() {
    try {
        Class.forName("org.postgresql.Driver");
    } catch (ClassNotFoundException ex) {
        Logger.getLogger(DatabaseConnection.class.getName()).log(Level.SEVERE, null, ex);
    }

    try {
        DriverManager.getConnection(url, user, password);
        JOptionPane.showMessageDialog(null, "connected");

    } catch (SQLException ex) {
        Logger.getLogger(DatabaseConnection.class.getName()).log(Level.SEVERE, null, ex);
        JOptionPane.showMessageDialog(null, "Failed to connect");
    }
    return connection;
}

}

Connect your node backend to postgresql database

Hans — Tue, 14 Jan 2020 20:22:39 +0000

npm init -> entry point: server.js
create file "server.js" in project root
npm install express pg nodemon
in package.json add script "start":"nodemon"

{
"name": "server",
"version": "1.0.0",
"description": "",
"main": "server.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start":"nodemon"
},
"author": "",
"license": "ISC",
"dependencies": {
"express": "^4.17.1",
"nodemon": "^2.0.2",
"pg": "^7.17.1"
}
}

in project root -> make folder "config"
make folder db.js and add the following

const { Pool, Client } = require("pg");

const pool = new Pool({
user: 'postgres',
host: 'localhost',
database: '',
password: '',
port: 5432
});

pool.connect(err => {
if (err) {
console.error('connection error', err.stack)
} else {
console.log('connected')
}
});

in file server.js add following

const express = require('express');
const app = express();
const pool = require ('./config/db');

const PORT = process.env.PORT || 5000;
app.listen(PORT, () => {
console.log(Listening to port: ${PORT});
});

npm start

Make aliases in ubuntu

Hans — Tue, 14 Jan 2020 12:12:43 +0000

Easiest way to make aliases of often used commands is to open file manager and then hit ctrl-h to show the hidden files. Open the .bashrc file and scroll down to the bottom of the file

The syntax is: alias='commands'

For example you can combine these two commands with a simple alias

alias ago='sudo apt-get update && sudo apt-get upgrade'

Now save the file, open a terminal window and hit 'ago'

I like to hit 'ls' everytime I cd into a folder just to get a view of the newly entered folder so therefore i like to combine the cd command with a ls. However, this kind of command needs a function as opposed to an alias

Type:

function cd {
builtin cd "$@" && ls
}

hit save and open up a new terminal window and test it

Connect your node app to MongoDB in ten easy steps

Hans — Tue, 14 Jan 2020 12:06:49 +0000

npm init -> entry point: server.js
create file "server.js" in project root
npm install mongoose express config nodemon
in package.json add script "start":"nodemon"

{
"name": "server",
"version": "1.0.0",
"description": "",
"main": "server.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start":"nodemon"
},
"author": "",
"license": "ISC",
"dependencies": {
"config": "^3.2.4",
"express": "^4.17.1",
"mongoose": "^5.8.7",
"nodemon": "^2.0.2"
}
}

in project root -> make folder "config"
in folder "config" make two files -> "db.js" and "default.json"
in file "default.json" add the link with your mongodb credentials

{
"MongoURI":"mongodb+srv://username:yourpassword@aquacontrol-atxjt.mongodb.net/test?retryWrites=true&w=majority"
}

in file "db.js" - add the following:

const mongoose = require('mongoose');
const config = require('config');
const database = config.get("MongoURI");

const connectDB = async () => {
try {
await mongoose.connect(database, {
useNewUrlParser: true,
useUnifiedTopology: true
});
console.log('Connected to mongoDB');
} catch (err) {
console.error(err.message);
process.exit(1);

};

module.exports = connectDB;

in "server.js" - add the following:

const express = require('express');
const app = express();

const PORT = process.env.PORT || 5000;
app.listen(PORT, () => console.log(Server started on port ${PORT}));
/*

Connecting to mongoDB */ const connectDB = require('./config/database'); connectDB(); app.use(express.json({ extended: false }));

module.exports = connectDB;

npm start