DEV Community: moose

waxy - Part 1 of my attempt to build a community driven search engine

moose — Mon, 03 Jan 2022 21:25:20 +0000

Waxy is the crawler portion, and it's still very much in its infancy. Here is where its at:
https://github.com/salugi/waxy
https://crates.io/crates/waxy

What does it do?
Well, it crawls. It can blindly crawl any (well, every site I've tested) site to a blind depth and provide statistics on the site. It can crawl only specific URLs based on a regex. You can crawl 100, 1000, or 27.
With the pages it can parse specific tags and grab text to better index a site. Currently I am working on a scraper wrapper (https://crates.io/crates/scraper) for parsing selectors. It provides hashmaps of meta data.

Pages equal Records, and records get pressed by crawling pages. I made a design decision to store the pages alone and just change the way they get parsed. I also mainly deal in strings. String hashsets, hashmaps. String inputs, outputs. I convert data to strings as I have found it easier to translate when dealing with higher level abstractions. Something to consider is that it is not a fast process, this generally takes time when sequentially producing records.

I have provided functionality to produce multiple records as a way to better run a worker thread. This thread/server is dedicated to crawling. It's not meant to be a speedy process as reindexing a lot and a fast crawl speed likely throws alarms off sites you want to crawl. Hook it up as a producer over a socket, you will likely be able to go real quick. But it's not a fruitful process or one that is really consolidated to a single function.

This is my first crate, and it has been an interesting experience. I've learned a lot about planning. Learning about versioning, when to and when not to.
Crates io is a real slick system and I fully intend to make use of all the free functionality that comes with the platform. This is just the crawler. I'm realizing what a large endeavor it is.

As to it being part of a community driven search engine, I'm going to attempt to make this a part of FOSS suite people/communities can use to make search engines that index retrieve, archive based on what a community finds important and not what google finds important.

I'm toying with ideas for the oddjob server, as well as the indexing engine, the algorithms. Theres a lot of reading and a lot of failure. Tiring.

But, it's nice to have something published.
I intend to incorporate chrome headless at some point. because unfortunately js

heres the rundown

waxy = "0.1.1"
tokio = { version = "1", features = ["full"] }

main.rs

use waxy::pressers::HtmlPresser;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {


    //Wax worker

    /*

    create a single record from url

    */
    match HtmlPresser::press_record("https://example.com").await{
        Ok(res)=>{
            println!("{:?}", res);
        },
        Err(..)=>{
            println!("went bad")
        }

    }

    println!();
    println!("----------------------");
    println!();

    /*

    crawl a vector or urls for a vector of documents

    */

    match HtmlPresser::press_records(vec!["https://example.com"]).await{
        Ok(res)=>{
            println!("{:?}", res.len());
        },
        Err(..)=>{
            println!("went bad")
        }

    }

    println!();
    println!("----------------------");
    println!();

   /*

    crawl a domain, the "1" is the limit of pages you are willing to crawl. 
    This is a slow process and created a return array of HtmlRecords

    */

    match HtmlPresser::press_records_blind("https://funnyjunk.com",1).await{
        Ok(res)=>{
            println!("{:?}", res.len());
        },
        Err(..)=>{
            println!("went bad")
        }

    }

    /*
    blind crawl a domain for links, 
    inputs:
    url to site
    link limit, limit of the number of links willing to be grabbed
    page limit, limit of the number of pages to crawl for links

    This will just url collect, use case is getting a blind depth or a blind crawl of the site. 
    */

    match HtmlPresser::press_urls("https://example.com",1,1).await{
        Ok(res)=>{
            println!("{:?}", res.len());
        },
        Err(..)=>{
            println!("went bad")
        }

    }

    println!();
    println!("----------------------");
    println!();

    /*
    blind crawl a domain for links that match a pattern, 
    inputs:
    url to site
    pattern the  url should match
    link limit, limit of the number of links willing to be grabbed
    page limit, limit of the number of pages to crawl for links

    This would be useful when wanting to crawl a thread. This is NOT domain name specific
    */
    match HtmlPresser::press_curated_urls("https://example.com", ".", 1,1).await{
        Ok(res)=>{
            println!("{:?}", res);
        },
        Err(..)=>{
            println!("went bad")
        }

    }

    println!();
    println!("----------------------");
    println!();

        /*
    blind crawl a domain for document whose urls that match a pattern, 
    inputs:
    url to site
    pattern the  url should match
    page limit, limit of the number of pages to crawl for links

    Does the URL thing, but just with records

    */
    match HtmlPresser::press_curated_records("https://example.com", ".", 1).await{
        Ok(res)=>{
            println!("{:?}", res);
        },
        Err(..)=>{
            println!("went bad")
        }

    }

    println!();
    println!("----------------------");
    println!();


    /*

    section illustrating the way records parse themselves. 

    */
    //get doc
    let record = HtmlPresser::press_record("https://funnyjunk.com").await.unwrap();

    //get anchors
    println!("{:?}",record.anchors());
    println!();
    println!("{:?}",record.anchors_curate("."));
    println!();
    println!("{:?}",record.domain_anchors());
    println!();
    //call headers
    println!("{:?}",record.headers);
    println!();
    //call meta data
    println!("{:?}",record.meta_data());
    println!();
    //tag text and html
    println!("{:?}",record.tag_html("title"));
    println!();
    println!("{:?}",record.tag_text("div"));
    println!();
    println!();

    Ok(())


}

Rust Host Lookup and Port Scanner

moose — Sun, 26 Dec 2021 16:14:27 +0000

This is just the remnants of a project I started and am tabling. Its ANOTHER port scanner in rust. I was planning on mimicking nmap functionality.
tl;dr
https://github.com/salugi/port_scanner_demo

The project build is meant for building bigger pieces. The conductor is the actual actor. The project was going to be separated up into scans. Since it is just the TCP part, here is how it works.

We are going to structure the project using cargo workspaces. I decided to build it this way since it better illustrates how exactly to structure files in order for rust to see them all.

First, make a directory

mkdir port_scanner_demo

in that directory make a Cargo.toml file and copypaste the following into it:

[workspace]
members = [
    "example_bin",
    "lib"
    ]

This sets up the directory to handle multiple project files and makes it easier to break up files. Right here, we are basically saying "I have these two project folders". Now, since the toml file declares this, cargo knows not to make multiple target directories. Once you have the Cargo.toml file created, do the following in your project directory:
1 . Create the example_bin binary space:

cargo new --bin example_bin

2 . Create the library to go along with it:

cargo new --lib lib

3 . To make sure, in the project root directory do:

cargo run

If it runs, things went right, you should have only one shared target directory because cargo found the root directory toml file and knew there were shared workspaces.

First in the ./lib/Cargo.toml file, change the dependencies to look like the following:

[dependencies]
dns-lookup = "1.0.2"

Second, let us make a function to look up the ip of a given host.
In the lib directory create a file called conductor.rs

touch ./lib/conductor.rs

The conductor will "conduct" the different port scan variation. This will be the main abstraction and is done because there are two components to the TCP scan that have to do with creating SocketAddr's for a timeout scan. This means we need to be able to take in a "google.com" or an "8.8.8.8" address.

In the ./lib/conductor.rs file, copy and paste the following:

use std::io;
use std::net::IpAddr;
use std::str::FromStr;
use dns_lookup::lookup_host;

pub fn conduct_host_lookup(addr:&str)->Result<Vec<String>, io::Error>{

    let ip_vecs : Vec<IpAddr> = lookup_host(addr).unwrap();
    let mut return_array: Vec<String> = vec![];

    for ip in ip_vecs {
        return_array.push(ip.to_string())
    }


    Ok(return_array)

}

now, in the ./lib/lib.rs file, copy past the following:

pub mod conductor;

next in the ./example_bin/Cargo.toml file, change the dependency section to look like the following:

[dependencies]
lib = {path = "../lib"}

Now in the ./example_bin/main.rs file, import the conduct_host_lookup function by copypasting the following in:

use lib::conductor::{conduct_host_lookup};

fn main() {

    let hosts = conduct_host_lookup("google.com");
    println!("{:?}", hosts);

}

in the root of the ./port_scanner_demo/ directory run it by

cargo run

your output should yield a similar result:

Ok(["142.250.190.78"])

The host lookup part is done. Now, lets start on the first tcp scan. In the ./lib/ directory create the ./lib/tcp_scans.rs file. Copy paste the following in:

use std::io;
use std::net::{SocketAddr, TcpStream};
use std::time::Duration;



/*

connect scan

 */
pub fn tcp_connect_scan(address:&str, ports:Vec<i32>)->Result<Vec<i32>, io::Error>{

    let mut open_port_vector:Vec<i32> = vec![];


    for port in ports {

        let addr_string = format!("{}:{}", address, port);
        let server_details:SocketAddr = addr_string.parse().expect("unable to parse");

        if TcpStream::connect(&server_details).is_ok() {

            open_port_vector.push(port);

        }

    }

    Ok(open_port_vector)

}

Since the project revolved around records, we will create a data struct to have a consolidated way to transport the host as well as the open ports on the host.
In the ./lib/ directory, create the ./lib/records.rs file and copypaste the following into it:

pub struct TcpScanRecord {
    pub host : String,
    pub open_ports : Vec<i32>
}

To make sure it is available to main, go to the ./lib/lib.rs file and change it to look like the following:

pub mod conductor;
mod tcp_scans;
pub mod records;

Switch back over to the ./lib/conductor.rs file change it to the following

use crate::tcp_scans::{tcp_connect_scan};
use crate::records::TcpScanRecord;

use std::io;
use std::net::IpAddr;
use std::str::FromStr;
use dns_lookup::lookup_host;

pub fn conduct_connect_scan(addr:&str, ports:Vec<i32>) ->Result<TcpScanRecord,io::Error> {

    let sanitized_host = host_sanitizer(addr);
    let open_ports = tcp_connect_scan(&sanitized_host, ports)?;
    let record = TcpScanRecord {
        host : sanitized_host,
        open_ports
    };

    Ok(record)

}

pub fn conduct_host_lookup(addr:&str)->Result<Vec<String>, io::Error>{

    let ip_vecs : Vec<IpAddr> = lookup_host(addr).unwrap();
    let mut return_array: Vec<String> = vec![];

    for ip in ip_vecs {
        return_array.push(ip.to_string())
    }


    Ok(return_array)

}

fn host_sanitizer(addr:&str)->String{

    match IpAddr::from_str(addr) {

        Err(_) => {

            let ip_vecs : Vec<std::net::IpAddr> = lookup_host(addr).unwrap();

            ip_vecs[0].to_string()

        },

        Ok(IpAddr::V4(..)) => addr.to_string(),
        Ok(IpAddr::V6(..)) => addr.to_string(),

     }

}

The host_sanitizer function allows us to take in a host string, an IPv4 address and an IPv6 address to feed to the the TCP connect scan.

To test, switch to the ./example_bin/main.rs file and change it to look like the following:


use lib::conductor::{conduct_connect_scan, conduct_host_lookup};

fn main() {

    let hosts = conduct_host_lookup("google.com");
    println!("{:?}", hosts);

    let record = conduct_connect_scan("google.com",vec![80]);
    println!("{:?}", record);
}

To run, go back to the root folder and do a

cargo run

Your output should be similar to the following:

Ok(["142.250.191.238"])
Ok(TcpScanRecord { host: "142.250.191.238", open_ports: [80]})

Now, this is a very basic port scanner. The issue is, if we want to then scan a bunch of ports we would have to wait a long time if a port is not open. So we want a timeout. Luckily for us, there is a Rust function for that. Back in the ./lib/tcp_scans.rs, change it to look like the following:

use std::io;
use std::net::{SocketAddr, TcpStream};
use std::time::Duration;



/*

connect scan

 */
pub fn tcp_connect_scan(address:&str, ports:Vec<i32>)->Result<Vec<i32>, io::Error>{

    let mut open_port_vector:Vec<i32> = vec![];


    for port in ports {

        let addr_string = format!("{}:{}", address, port);
        let server_details:SocketAddr = addr_string.parse().expect("unable to parse");

        if TcpStream::connect(&server_details).is_ok() {

            open_port_vector.push(port);

        }

    }

    Ok(open_port_vector)

}


/*

connect timeout scan

 */
pub fn tcp_timeout_scan(address:&str, ports:Vec<i32>, timeout:u64)->Result<Vec<i32>, io::Error>{

    let mut open_port_vector:Vec<i32> = vec![];
    let duration = Duration::from_millis(timeout);

    for port in ports {

        let addr_string = format!("{}:{}", address, port);
        let server_details:SocketAddr = addr_string.parse().expect("unable to parse");

        if TcpStream::connect_timeout(&server_details,duration).is_ok() {

            open_port_vector.push(port);

        }

    }

    Ok(open_port_vector)

}

This is basically saying "we are only willing to wait n number of milliseconds to be able to connect".

Switch back to the ./lib/conductor.rs file and modify it to look like the following:

use crate::tcp_scans::{tcp_connect_scan, tcp_timeout_scan};
use crate::records::TcpScanRecord;

use std::io;
use std::net::IpAddr;
use std::str::FromStr;
use dns_lookup::lookup_host;

pub fn conduct_connect_scan(addr:&str, ports:Vec<i32>) ->Result<TcpScanRecord,io::Error> {

    let sanitized_host = host_sanitizer(addr);
    let open_ports = tcp_connect_scan(&sanitized_host, ports)?;
    let record = TcpScanRecord {
        host : sanitized_host,
        open_ports
    };

    Ok(record)

}

pub fn conduct_timeout_scan(addr:&str, ports:Vec<i32>, timeout:u64) ->Result<TcpScanRecord,io::Error> {

    let sanitized_host = host_sanitizer(addr);
    let open_ports = tcp_timeout_scan(&sanitized_host, ports,timeout)?;
    let record = TcpScanRecord {
        host : sanitized_host,
        open_ports
    };

    Ok(record)

}

pub fn conduct_host_lookup(addr:&str)->Result<Vec<String>, io::Error>{

    let ip_vecs : Vec<IpAddr> = lookup_host(addr).unwrap();
    let mut return_array: Vec<String> = vec![];

    for ip in ip_vecs {
        return_array.push(ip.to_string())
    }


    Ok(return_array)

}

fn host_sanitizer(addr:&str)->String{

    match IpAddr::from_str(addr) {

        Err(_) => {

            let ip_vecs : Vec<std::net::IpAddr> = lookup_host(addr).unwrap();

            ip_vecs[0].to_string()

        },

        Ok(IpAddr::V4(..)) => addr.to_string(),
        Ok(IpAddr::V6(..)) => addr.to_string(),

     }

}

To test, switch back to ./example_bin/main.rs and modify it to look like the following:


use lib::conductor::{conduct_timeout_scan, conduct_connect_scan, conduct_host_lookup};

fn main() {

    let hosts = conduct_host_lookup("google.com");
    println!("{:?}", hosts);

    let record = conduct_connect_scan("google.com",vec![80]);
    println!("{:?}", record);

    let record = conduct_timeout_scan("google.com",vec![80], 80);
    println!("{:?}", record);

}

Run

cargo run

The output should resemble the following:

Ok(["142.250.190.110"])
Ok(TcpScanRecord { host: "142.250.190.110", open_ports: [80]})
Ok(TcpScanRecord { host: "142.250.190.110", open_ports: [80]})

Where to go from here? It is an interesting question. These types of scans can run independent of sudo/admin access. When getting into the ip layer, you will run into the need to have admin/sudo access. A lighter-weight is a syn scan.
This will require the user to build partial packets, but this could also be a way to trigger alarms at a target address because this is a symptom of a type of DDOS/DOS attack. ICMP or ping scans also require sudo/admin access.

These are scans I decided to not build out.

Another thing to build out is a way to analyze the OS. Techniques to do this are banner grabbing and open port OS analysis.

IF you wanted to send strings to try and do some banner grabbing, you could do something like

/*

tcp send string

 */
pub fn tcp_send_string(address:String, port:i32, msg:String) ->Result<String, io::Error>{


        let host_string = format!("{}:{}", address, port.to_string());
        let mut stream =  TcpStream::connect(host_string).expect("could not connect");
        let mut response_shell = String::new();
        let req_bytes =     msg.as_bytes();

        stream
            .write_all(req_bytes)
            .expect("couldnt write bytes");

        stream
            .read_to_string(&mut response_shell)
            .expect("could not read string");

    Ok(response_shell)
}

its basically sending data and parsing the return strings or packet data to verify the service/protocol running on a given port.
You'd probably want to try a scraper to parse returned document data. Rust has great options for that.

If no banner is able to be grabbed, the open port analysis basically takes all the open ports and will attempt to match against know schemas of different systems. Basically if A, B, and C are open there is an X% chance of being linux.

Some things to consider:
Rate limiting: You can send a shit ton of data and throw a bunch of DOS flags because, you can DOS/DDOS a host by sending too many packets (especially partial packets like syn packets that don't get acknowledged.)
Gettin' Listed: you can get blocked by a number of hosts if you send them a bunch of weird traffic. Avoid unnecessary checks.
I wrote this to be able to pass core functionality strings. There are a lot of different types, and this was aiming to build basic API's with Deno for an intelligence platform.

Remember, when collecting intelligence slow and steady wins the race. If you get a response indicating a connection was reset: time to change IPs.

A port scanner is a perfect exercise to learn about protocols. It's a great opportunity to learn about concurrent programming and how to utilize multiple threads to optimize performance.

If you know how to build the tools used, you know how to benefit from their use.

If you have any issues, feel free to leave a comment. The working product is at
https://github.com/salugi/port_scanner_demo

peace

Building a crawler

moose — Sat, 20 Nov 2021 19:57:58 +0000

So I haven't written in a while. It's a slow day so I thought, how about we write a web crawler tutorial?

so, this code can easily be switched to js, and you can find a deno_dom replacement in cheerio with node js.

The main thing a crawler needs is valid markup.

tl;dr
[https://github.com/salugi/smooth_crawl]

I am currently building out my project for a deno/rust search engine with mellisearch as part of a larger project.

Since this is a deno backend, I'm taking advantage of the ability to use typescript OOB. Its handy, especially with the ability to create interfaces that can extend others.

What should a web crawler do?

Crawl to a depth blindly
Be rate limited in the calling
Crawl unique links
Parse URLs
Handle edge cases
probably some other stuff

We are going to make one that does all of those.
First, we will look at building a self contained crawler which is painfully slow (but, doesn't DOS a site).

Since this tutorial is using Deno (https://deno.land)
https://deno.land/#installation

super easy.

1 . Let's start with creating the folder structure
mine:

smooth_crawl
|_src
  |models

2 . Open up the smooth crawl directory in your ide of choice

3 . Create entry.ts file

./smooth_crawl/entry.ts

4 . Create base objects.

In our case, the base object will be a ts interface named RecordKey and HttpRecord.

Create RecordKey.ts file

./smooth_crawl/models/RecordKey.ts

copy paste the following code


export interface RecordKey {

    id:string,
    creation_date:any,
    archive_object:any

 }

Create HttpRecord.ts file

./smooth_crawl/models/HttpRecord.ts

Copy paste the following into it

import { RecordKey } from "./RecordKey.ts";

export interface HttpRecord extends RecordKey {

    url:URL,
    response:any,
    response_text:string

}

These serve has data models, they hold data, they do nothing outside of that.

5 . Build http client

Deno uses the fetch API. It is worth noting that since JS is widely used in some sites, a puppeteer implementation would be needed for some sites. We won't focus on puppeteer in this tutorial, mainly relying on the fetch API to handle the HTTP reqs. Below I have annotated some of the code to explain what is going on

Create http_client.ts file

./smooth_crawl/http_client.ts

Copy and paste the following code:

// @ts-ignore
import {v4} from "https://deno.land/std/uuid/mod.ts";
import {HttpRecord} from "./models/HttpRecord.ts";

/*

returns http text (normally html)

*/

const controller = new AbortController()
const timeoutId = setTimeout(() => controller.abort(), 5000)

export async function get_html_text(unparsed_url:string) : Promise<string> {

    return new Promise(async function (resolve, reject) {

        //parse url cuz symbols

                let parsed_url = new URL(unparsed_url)

        //send get
                await fetch(parsed_url.href,{signal:controller.signal}).then(function (result) {
                    clearTimeout(timeoutId)
                    if (result !== undefined) {

                        //turn result to text.
                        result.text().then(function (text) {

                                resolve(text)

                        }).catch(error => {

                            console.error("get_html_text result.text errored out")

                            reject(error)

                        })

                    }
                }).catch(error => {

                    console.error("get_html_text fetch errored out")

                    reject(error)

                })
    })
}

/*

returns http record

*/

export async function get_http_record(unparsed_url:string) : Promise<HttpRecord> {

    return new Promise(async function (resolve, reject) {

        let parsed_url = new URL(unparsed_url)

        let record : HttpRecord ={
            id:v4.generate(),
            creation_date : Date.now(),
            url:parsed_url,
            response:{},
            response_text:"",
            archive_object:{}
        }

        await fetch(record.url.href,{signal:controller.signal}).then(function (result) {

            clearTimeout(timeoutId)

            if (result !== undefined && result !== null) {

                record.response = result

                // turn result to text.
                result.text().then(function (text) {

                    if (text.length > 1){

                        record.response_text = text

                    }

                    resolve(record)

                }).catch(error => {

                    console.error("get_http_record result.text errored out")

                    reject(error)

                })

            }
        }).catch(error => {

            console.error("get_http_record fetch errored out")

            reject(error)

        })
    })
}

The explanation of these two methods is that they do very similar things, sans one is just returning the text of an HTTP request and the other is returning an HTTP record. The text return, or get_html_text() is lower weight and doesn't unnecessarily create objects. The reason behind this is because this acts as a depth check

Some pages could have, say 50,000 links on a single page right? Sounds ludicrously high, but it is out there if not just to make people work harder. But this first html function acts as a way to be as non committal as possible to depth check the site so we don't blow out.

Now that we have the file created, we need to test it. go back to the entry.ts file in root of the smooth_crawl directory and copy paste this code:

import {get_html_text} from "./src/http_client.ts";

// @ts-ignore
let smoke = Deno.args

// @ts-ignore
let html_text = await get_html_text(smoke[0])

console.log(html_text)

then from the smooth_crawl directory run the command:

deno run --allow-net ./entry.ts https://example.com

if it doesn't error out, we are ready to move into conductor concept and parsing the html.

6 . Create the conductor.ts file

Create conductor.ts file

./smooth_crawl/src/conductor.ts

Copy paste the following in


import {get_html_text, get_http_record} from "./http_client.ts"
import {catalogue_basic_data, catalogue_links} from "./cataloguer.ts";

import {HttpRecord} from "./models/HttpRecord.ts";
const non_crawl_file = ["jpg", "pdf", "gif", "webm", "jpeg","css","js","png"]

/*

returns http record

archival objects:
link data (all links on a page parsed)
metadata (all metadata tages)

*/

export function conduct_basic_archive(unparsed_url:string) : Promise<HttpRecord> {

    return new Promise<HttpRecord>(async(resolve,reject)=> {

        try {

            let parsed_url = new URL(unparsed_url)
            let record = await get_http_record(parsed_url.href)
            let archival_data : any = await catalogue_basic_data(parsed_url.origin, record.response_text)

            record.archive_object.links = archival_data.link_data
            record.archive_object.meta = archival_data.meta_data

            resolve(record)

        } catch (error) {

            reject(error)

        }

    })

}


/*

harvests links, number_to_gather === number of links to gather

*/

export async function conduct_link_harvest(link:string, link_limit:number, page_limit:number) : Promise<Array<string>> {

    return new Promise<Array<string>>(async (resolve, reject)=>{

        try {

            let links = Array();

            links.push(link)

            for (let i = 0; i < links.length; i++) {

                let url : URL = new URL(links[i])
                // @ts-ignore
                let text : string = await get_html_text(url.href)
                let unharvested_links : Array<URL> = await catalogue_links(url.origin, text)
                let harvested_links : Array<string> = await harvest_links(links, unharvested_links)
                let stop : number = 0;

                if (links.length + harvested_links.length > link_limit){

                    stop = link_limit - links.length

                }else{

                    stop = harvested_links.length

                }

                for (let j = 0; j < stop; j++) {

                    links.push(harvested_links[j])

                }

                if(i >= page_limit){

                    break;

                }

            }

            resolve(links)

        } catch (error) {

            reject(error)

        }

    })

}

function harvest_links(gathered_links: Array<string>, links:Array<any>) : Promise<Array<string>> {

    return new Promise( (resolve, reject) => {

        try {

            let return_array = Array()

            for (const link of links) {

                let should_add = !gathered_links.includes(link.href)
                let file_extension = get_url_extension(link.href)
                let not_in_list = !non_crawl_file.includes(file_extension)

                if (
                    should_add
                    &&
                    not_in_list
                ) {

                    return_array.push(link.href)

                }

            }



            resolve(return_array)

        } catch (error) {

            console.error(error)

        }

    })

}


function get_url_extension( url: string ) {
    //@ts-ignore
    return url.split(/[#?]/)[0].split('.').pop().trim();
}



export async function conduct_worker_harvest(link:string, link_limit:number, page_limit:number) : Promise<Array<string>> {

    return new Promise<Array<string>>(async (resolve, reject)=>{

        try {

            let links = Array();
            let should_break = false

            links.push(link)

            for (let page_index = 0; page_index < links.length; page_index++) {

                let url : URL = new URL(links[page_index])
                // @ts-ignore
                let text : string = await get_html_text(url.href)
                let unharvested_links : Array<URL> = await catalogue_links(url.origin, text)
                let harvested_links : Array<string> = await harvest_links(links, unharvested_links)
                let stop : number = 0;

                if (links.length + harvested_links.length > link_limit){

                    stop = link_limit - links.length
                    should_break = true

                }else{

                    stop = harvested_links.length

                }

                if(page_index >= page_limit ){

                    should_break = true

                }

                for (let j = 0; j < stop; j++) {

                    links.push(harvested_links[j])

                    //publisher.publish_message( { url : harvested_links[j] } )

                }

                if(should_break){

                    break;

                }

            }

            resolve(links)

        } catch (error) {

            reject(error)

        }

    })

}

This will error out. We have to add to it. However, the concept is that the conductor conducts actions. A mishmash on smaller actions that build bigger things.

Example
http call -> crawl -> return object

on to the next step.

7 . create the cataloguer.ts file

Create file cataloguer.ts

./smooth_crawl/src/cataloguer.ts

Copy paste the following code into it

import {DOMParser} from 'https://deno.land/x/deno_dom/deno-dom-wasm.ts';
import {v4} from "https://deno.land/std/uuid/mod.ts";

export async function catalogue_links(origin:string, text:string):Promise<any>{

    return new Promise(function(resolve) {

        try {

            let link_set = Array();

            if(text.length > 1) {

                const document: any = new DOMParser().parseFromString(text, 'text/html');

                if (document === undefined) {

                    let funnel_point = "cataloguer.ts"
                    let funk = "crawl"
                    let error = "unable to interchange gen_object"
                    let id = v4.generate()

                    resolve(link_set)

                } else {

                    let link_jagged_array = Array<Array<any>>(
                        document.querySelectorAll('a'),
                        document.querySelectorAll('link'),
                        document.querySelectorAll('base'),
                        document.querySelectorAll('area')
                    )

                    for (let i = 0; i < link_jagged_array.length; i++) {

                        for (let j = 0; j < link_jagged_array[i].length; j++) {

                            if (link_jagged_array[i][j].attributes.href !== undefined
                                &&
                                link_jagged_array[i][j].attributes.href.length > 0) {

                                link_set.push(link_jagged_array[i][j].attributes.href)

                            }

                        }

                    }

                    link_set = [...new Set(link_set)]

                    // @ts-ignore
                    let fully_parsed_links = link_parse(origin, link_set)

                    resolve(fully_parsed_links)

                }

            }else{
                resolve(link_set)
            }

        }catch(error){

            console.error(error)

        }

    })

}
export async function catalogue_basic_data(origin:string, text:string):Promise<any>{

    return new Promise(function(resolve) {

        try {

            let link_set = Array();

            if(text.length > 1) {

                const document: any = new DOMParser().parseFromString(text, 'text/html');

                if (document === undefined) {

                    console.error("document not defined")

                } else {

                    let link_jagged_array = Array<Array<any>>(
                        document.querySelectorAll('a'),
                        document.querySelectorAll('link'),
                        document.querySelectorAll('base'),
                        document.querySelectorAll('area')
                    )

                    let meta_information = document.querySelectorAll('meta')

                    for (let i = 0; i < link_jagged_array.length; i++) {

                        for (let j = 0; j < link_jagged_array[i].length; j++) {

                            if (link_jagged_array[i][j].attributes.href !== undefined
                                &&
                                link_jagged_array[i][j].attributes.href.length > 0) {

                                link_set.push(link_jagged_array[i][j].attributes.href)

                            }

                        }

                    }

                    link_set = [...new Set(link_set)]

                    // @ts-ignore
                    let fully_parsed_links = link_parse(origin, link_set)
                    let parsed_meta_information = meta_parse(meta_information)

                    let archives = {
                        link_data:fully_parsed_links,
                        meta_data:parsed_meta_information
                    }

                    resolve(archives)

                }

            }else{
                resolve(link_set)
            }

        }catch(error){

            let funnel_point = "cataloguer.ts"
            let funk = "crawl"
            let id = v4.generate()

            console.error(error)
        }

    })

}

function meta_parse(a:Array<any>):Array<string>{
    try {

        let out = Array<any>();

        for (let i = 0; i < a.length; i++) {

            if (a[i].attributes.content !== undefined
                &&
                a[i].attributes.content !== null) {

                let meta_tag = {
                    name: "",
                    content: Array()
                }

                if (a[i].attributes.charset !== undefined) {

                    meta_tag.name = "charset"
                    meta_tag.content.push(a[i].attributes.charset)

                    out.push(meta_tag)

                    continue

                } else if (a[i].attributes.property !== undefined) {

                    meta_tag.name = a[i].attributes.property
                    meta_tag.content = a[i].attributes.content.split(",")

                    out.push(meta_tag)

                    continue

                } else if (a[i].attributes["http-equiv"] !== undefined) {

                    meta_tag.name = a[i].attributes["http-equiv"]
                    meta_tag.content = a[i].attributes.content.split(",")
                    out.push(meta_tag)

                    continue

                } else if (a[i].attributes.name !== undefined) {

                    meta_tag.name = a[i].attributes.name
                    meta_tag.content = a[i].attributes.content.split(",")

                    out.push(meta_tag)

                    continue

                }else {

                    out.push({

                        "meta-related":a[i].attributes.content

                    })

                }

            }

        }

        return out

    }catch(error){

        console.log("crawler-tools.ts")
        Deno.exit(2)

    }

}


function meta_check(a:string):Boolean{

    if (
        /^\w+([\.-]?\w+)*@\w+([\.-]?\w+)*(\.\w{2,3})+$/.test(a) ||
        /(tel:.*)/.test(a)                                      ||
        /(javascript:.*)/.test(a)                               ||
        /(mailto:.*)/.test(a)
    ) {
        return true

    }else{

        return false

    }
}

function link_parse(domain:string, lineage_links: Array<string>):any{

    try {

        let c: Array<any> = new Array()

        if (lineage_links.length > 1) {

            for (let i = 0; i < lineage_links.length; i++) {

                if (
                    !/\s/g.test(lineage_links[i])
                    &&
                    lineage_links[i].length > 0
                ) {

                    let test = lineage_links[i].substring(0, 4)

                    if (meta_check(lineage_links[i])) {

                        continue

                    } else if (/[\/]/.test(test.substring(0, 1))) {

                        if (/[\/]/.test(test.substring(1, 2))) {

                            let reparse_backslash = lineage_links[i].slice(1, lineage_links[i].length)
                            lineage_links[i] = reparse_backslash

                        }


                        c.push(new URL(domain + lineage_links[i]))

                        continue

                    } else if (
                        (/\.|#|\?|[A-Za-z0-9]/.test(test.substring(0, 1))
                            &&
                            !/(http)/.test(test))
                    ) {

                        try {

                            //weed out potential non http protos
                            let url = new URL(lineage_links[i])


                        } catch {

                            let url = new URL("/" + lineage_links[i], domain)

                            c.push(url)

                        }

                        continue

                    } else if (/\\\"/.test(test)) {

                        let edge_case_split_tester = lineage_links[i].split(/\\\"/)
                        lineage_links[i] = edge_case_split_tester[0]

                        if (!/http/.test(lineage_links[i].substring(0, 4))) {

                            let url = new URL("/" + lineage_links[i], domain)

                            c.push(url)

                            continue

                        }
                    } else {

                        try {

                            let link_to_test = new URL(lineage_links[i])
                            let temp_url = new URL(domain)
                            let host_domain = temp_url.host.split(".")
                            let host_tester = host_domain[host_domain.length - 2] + host_domain[host_domain.length - 1]
                            let compare_domain = link_to_test.host.split(".")
                            let compare_tester = compare_domain[compare_domain.length - 2] + compare_domain[compare_domain.length - 1]

                            if (host_tester !== compare_tester) {

                                continue

                            }


                            c.push(link_to_test)

                        } catch (error) {

                            console.error(error)

                        }

                        continue

                    }

                }

            }

        }

        return c

    }catch(err){

        console.error(err)

    }

}

This file uses deno_dom (https://github.com/b-fuze/deno-dom) to catalogue and find links.

It handles the parsing of the different links to be able to make them usable.

next make your entry.ts file look like this


import {get_html_text} from "./src/http_client.ts";
import {conduct_link_harvest} from "./src/conductor.ts";

// @ts-ignore
//let html_text = await get_html_text(smoke[0])
//console.log(html_text)

//smoke[0] is the link
//the other two are set checks, 
//grab 100 links or crawl 20 pages, whichever comes first
let links = await conduct_link_harvest(smoke[0] ,100, 20)
console.log(links)

again, run the following command (may take a little time)
I am using funnyjunk as a test, feel free to use a site you want granted it doesn't have to have js load

deno run --allow-net ./entry.ts https://funnyjunk.com

if this runs, we are ready to move on to the last part of this build.

8 . create operator.ts file

create operator.ts file

./smooth_crawl/src/operator.ts

copy paste the following code

import {
    conduct_link_harvest,
} from "./conductor.ts"

import {HttpRecord} from "./models/HttpRecord.ts";

import {conduct_basic_archive} from "./conductor.ts";





export async function operate_crawl(url:string,link_limit:number){

    try{

        let crawl_links = await conduct_link_harvest(url,link_limit,50)
        let http_records = new Array<HttpRecord>()

        for(let i = 0; i < crawl_links.length;i++){

            let record = await conduct_basic_archive(crawl_links[i])

            http_records.push(record)

        }

        return http_records

    }catch(error){

        console.error(error)

    }

}

this code is to use the conductor to do many things in a more consolidated way.

now go back to the entry.ts files, delete it's contents and paste in the following

import {operate_crawl} from "./src/operator.ts";


let smoke = Deno.args

try{


    let limit_int = parseInt(smoke[1])
    let url = new URL(smoke[0])

    // @ts-ignore
    let crawled_pages : Array<HttpRecord> = await operate_crawl(url.href, limit_int)


    crawled_pages.forEach(element =>{

        console.log(element.url.href, element.response.status)

    })

} catch (err) {

    console.error(err)

}

now run the command:
(again I use funnyjunk, use what you please)
NOTE: 5 is the amount of pages to crawl

deno run --allow-net ./entry.ts https://funnyjunk.com 5

That is the crawler. The operate_crawl is a blind crawler for any site to go to a custom depth. This returns the crawled object in the form of a HTTP record (which we created first).

This is a slow way to do it, it crawls blindly unique links, returns a list of objects and is naturally rate limited.

But we can go faster in the same process to get a better archival tool

Below we implement the bookie. The bookie serves as a way to keep track of archival records, like our HttpRecord. This has a function "book_http_record" with could conduct a basic archive and then potentially save it to a database.

9 . Lets create our new bookie

create file

./smooth_crawl/bookie.ts

copy past the following in

import EventEmitter from "https://deno.land/x/events/mod.ts"
import {conduct_basic_archive} from "./conductor.ts";


export let bookie_emitter = new EventEmitter()

export function book_http_record(unparsed_url : string){
    (async () =>{
        try {

            let parsed_url = new URL(unparsed_url)
            let record =  await conduct_basic_archive(parsed_url.href)

            console.log(record.url.href,"recorded with status",record.response.status)



        }catch (error) {
            let funk = "book_http_record"

            console.error(funk)
            console.error(error)

        }
    })()
}

bookie_emitter.on("book_http_archive", await book_http_record)

this is a deno event emitter

so now we go back to the operator.ts file

add the method

export async function operate_harvest(url:string,link_limit:number,page_limit:number){

    try{

        // let publisher = new PublisherFactory("havest_basic_archive")
        let crawl_links = await conduct_link_harvest(url,link_limit,page_limit)


        for(let i = 0; i < crawl_links.length;i++){

            await new Promise(resolve => setTimeout(resolve, 80))

            bookie_emitter.emit("book_http_archive", crawl_links[i])

        }

        // publisher.close_publisher()

        return crawl_links

    }catch(error){

        console.error(error)

    }

}

That promise acts as a rate limiter in js, a blocking function in non-blocking io, I know. But, you'd get some people mad at you if you go zooming over their site.

now head back to the entry file
and change its contents to this


import {operate_crawl, operate_harvest} from "./src/operator.ts";


let smoke = Deno.args

try{


    let limit_int = parseInt(smoke[2])
    let url = new URL(smoke[1])
    let option = smoke[0].split(/-/)[1]

    switch(option){

        case "sc":

            // @ts-ignore
            let crawled_pages : Array<HttpRecord> = await operate_crawl(url.href, limit_int)


            crawled_pages.forEach(element =>{

                console.log("crawled", element.url.href, "with status", element.response.status)

            })

            break;

        case "eh":

            let harvested_links = await operate_harvest(url.href,limit_int,10)

            // @ts-ignore
            console.log("harvested", harvested_links.length, "links")
            console.log(harvested_links)

            break;

        default:

            console.error("not a valid option")

            break;

    }


} catch (err) {

    console.error(err)

}

Notice we kept the other method. "sc" stands for slow craw and "eh" stands for event harvest.

now run the command:
(again, I use funnyjunk as an example, pick your own)

$ deno run -A --unstable ./entry.ts -sc https://funnyjunk.com 5

if that doesn't error out we are finished.

Now, I wouldn't recommend anyone do thousands of links on a single site with either. One because you may blow your buffer and the over because you may make some server admin super mad.

Deno DNS Resolver

moose — Fri, 01 Oct 2021 14:22:04 +0000

I make a DNS resolver in Deno using type script with Deno's built in Deno.resolveDns functionality. The following snippit grabs an array of record types and builds a return object with them. Deno is still new so there is some "novel" usage here, but I believe that more functionality will come for sure.

create dns_resolver.ts file

Copypaste the following:


let records = ["MX", "AAAA", "A", "SRV","ANAME"]
export async function dns_call(domain:string, records_types:Array<string>):Promise<Object>{
    let promise:Promise<any> = new Promise(function(resolve,reject) {

        let ret_object = {}
        for (let i = 0; i < records_types.length;i++){
            // @ts-ignore
            Deno.resolveDns(domain, records_types[i]).then(function (dns_resolution) {
                // @ts-ignore
                ret_object[records_types[i]] = dns_resolution
                if(i === records_types.length -1){
                    resolve(ret_object)
                }
            }).catch(function () {
                // @ts-ignore
                ret_object[records_types[i]] = "not present"
                if(i === records_types.length -1){
                    resolve(ret_object)
                }
            })
        }
    })
    return promise
}
let smoke = await dns_call("example.com", records)
console.log(smoke)

run

$ > deno run --allow-net --unstable dns_resolver.ts

Play around with it.

Reactive Paradigm: RedLight/GreenLight

moose — Sat, 05 Jun 2021 08:35:55 +0000

The reactive paradigm. Theres a bunch of words around the concept. Many big words that mean a lot of things. So let me break this down into the idea of state and the "reactive" program.

First we should get a basic understanding, you and I should be on the same page.
Think of traffic Lights at intersections.

We have a green light and a red light:

The circles that are green are in a "green light" state which allows them to cross the divide of the two sides. The redlight circles and in a "redlight" state.
Since the circles do not want to wait at the light all day,
and there would be likely more circles to cause a back up,
they have to take turns.

When a reasonable amount of time passes: the two sides change.
Green lights changes to red
and
the red light needs to react to the green light changing to red to allow all the circles to freely move across the divide.

Then, again, after a reasonable amount of time the red turn green and the green to red.

What does that look like in code?

Since they are incorporated in the example, the following has these facts:

This is a Kotlin enumeration class
In this context they serve as a tool to illustrate the idea of state in a reactive process.

import kotlin.properties.Delegates //delegate import

    //Class declarations enum class Type<*>(Any<*>):Any<*>   
   enum class StopGo {
        //Establish the enumerated value
        REDLIGHT {
           //establish abstract overdrive
            override fun go() {
               //show the light type
               println("red light")
            }
        },
        GREENLIGHT {
            override fun go() {
                println("green light")
            }
        };
        //Empty container to be used
        abstract fun go()
    }//End of enum

Note: I am using an abstract function here and if you aren't
familiar, think of it as a function only GREENLIGHT and
REDLIGHT have. So, in keeping the ideas of abstraction,
separation of concerns you cannot initiate an enum and access
it without assigning it to an enumerated value.


    //main function 
    fun main() {
        //light is what is going to react to stop light
        var light = StopGo.GREENLIGHT
        var stopLight: StopGo by 
        Delegates.observable(StopGo.GREENLIGHT){
            property,oldValue,newValue ->
light = if(light == StopGo.GREENLIGHT) StopGo.REDLIGHT else 
StopGo.GREENLIGHT
        }
  /*

stopLight is a delegate

for now I want you to think of a delegate as a function within  
a variable much like how in javascript you can do:

var a = () => { //do work } 

so..

a() <- "does work"

you can think of Lambdas as that as well,
given they can also operate as code blocks. We basically say 
"If A Changes, so does B."
But if B changes, a does not, and that is an important 
distinction. A linear progression of reaction would look more 
like this if plotted:
A  ->  B  ->  A -> <--- declarative 
B  ΔA  A  ΔB  B Δ  <--- reactive
*/


light.go()  /* light = REDLIGHT   */

stopLight = StopGo.REDLIGHT /*GREENLIGHT-(Δ switch)->REDLIGHT*/             

light.go()  /* light = GREENLIGHT */

stopLight = StopGo.REDLIGHT /*REDLIGHT-(Δ switch)->GREENLIGHT*/

light.go()  /* light = REDLIGHT   */

    }

So, light was reacting to stopLight changing.

Now, this conceptualization is just a very basic example of a reactive program so you can build on it and then tell me what a noob I am because I would be overjoyed to hear it.

Note
This code compiles and works in Intellij, using Kotlin. You should be able to insert it.

The function(){of 'JS strings'}

moose — Tue, 01 Jun 2021 14:23:00 +0000

This post is going to go over dynamic function creation and it will take a general look at how JS looks at strings, and different ways they can be used.

These methodologies should be avoided as they tend to deliver unpredictable results that often off little reasoning as to why the evaluated the way they did. There are also concerns around accidental invocations. Unless you do the due diligence to take it to task I wouldn't incorporate it. But, we've all been in the spot where we just need to make it work even if its a quick and dirty way of doing the deed. (technical debt.. cough ough..)

I think this is a cool characteristic of the language and one I believe developers should be aware of, as it is an important facet in which the language itself operates.

Example: (take a moment to picture how this is going to run)

/*
JS Regex - Vanilla bby
*/
var b = "" + (f = () => {
    (a = () => {
        console.log("hello, in a")
        setTimeout(() => {
            console.log("in time out :(")
        }, 1);
    }).call(a())
    console.log("out of timeout, in f")
}).call(f()) + ""

var str = "";
console.log("comparison check b, before timeout call");
console.log(typeof b === "string");
console.log(typeof b);

var res = undefined;

setTimeout(() => {
console.log(" first comparison check b ");
console.log(typeof b === "string");
console.log(typeof b);

console.log("second comparison check b");
console.log(typeof b === "string");
console.log(typeof b);

res = str.split(str, b);
console.log("first comparison check res");
console.log(typeof res === "string");
console.log(typeof res);
}, 1);
console.log(res);
console.log("third comparison check b");
console.log(typeof b === "string");
console.log(typeof b);

What we have is a function within a string, or a string concatenated with a function. When run we will check the type of the value, and of a value and do a strict comparison check as to what each variable is. This is going to be run through the v8 engine, which really shouldn’t matter (let me know if you have found different)

Lets take a look now at what my output from the this code looks like:

Did you expect that output? I did not. In fact for the most part I cannot explain the repetition either outside of the variable initializing itself for comparison checks.

But I want you to pay attention to the fact that though the system saw it was a string, told me it was in fact a string, and did so with a strict comparison check multiple times: the b variable was not a string as it carried out the routine embedded inside of it.

What does this say?
Well it says there is some sort of duality in the way JS views strings (I've often thought of it as one big regex engine). That JS saw it was a string, but still had the direction to carry out the embedded instructions instead of print out what the instruction would be. Cant strings in fact be functions? Yes, in face strings can indeed be evaluated at runtime as functions. The next few examples were pulled from answered questions as I believe there is some great discussions on the topic.

Example 2

[https://stackoverflow.com/questions/43726544/how-to-call-a-function-using-a-dynamic-name-in-js]

//variables
var ta = 3213;
var da = 44;
var s = [];

//Create string representation of function
s[1] = function test0(){ alert(" + da + ");}; //this["test"+0]
s[0] = function test1(){ alert(" + ta + ");}; //this["test"+1]

s.forEach((fun) => { this[fun.name] = fun;});

// calling the function
this["test"+1](); //calls test1
this["test"+0](); //calls test0

This is using an invocation to access the the two functions. This is a dynamic invocation which is a way a good amount of libraries are built (sans the dynamic way to invoke the function.

Example 3

[https://stackoverflow.com/questions/7650071/is-there-a-way-to-create-a-function-from-a-string-with-javascript]

var name = "foo";
// Implement it
eval("function " + name + "() { alert('Foo'); };");
// Test it
foo();
// Next is TRUE
foo.name === 'foo'

This is using the "eval()" function to evaluate the string as a function instead of printing it. instead of printing it. Outside of a whonky DB return value, there really isn't a whole lot to use this with.

Example 4

[https://stackoverflow.com/questions/20129236/creating-functions-dynamically-in-js]

var f = new Function('name', 'return alert("hello, " + name + "!");');
f('erick');

This is constructing the body and contents of the function and using the built the function feature to store a body that evaluates as a function and storing it in variable f.
I personally think this is a cleaner and more approachable way to do this task as it mitigates the concerns around accidental invocation. In the example, the 'name' param is the value being passed in for the function when called and the second param contains the contents of the function itself.

String can be evaluated as functions in JS, a characteristic of the language that not many others share. While fun to toy with to see what spits out the console, I think it best not to incorporate it en masse, but to understand better how the HUGE ecosystem surrounding the language itself is used.

Kotlin Pure Regex Routing Logic

moose — Mon, 31 May 2021 09:18:09 +0000

This article is just going to be a step my step of my turn sketch of utilizing pure regex patterns to develop complex and robust routing logic that is extremely lightweight, compact and configurable. In this example we will take a look at whats going on in the code. Let's do it.

Regex.kt

These are regex checkers that do protocol match checks

//     ft or ftps
var ftpReg = Regex("(ftp|ftps://)")

//    https or https
var httpReg = Regex("(?:(http|https)"

//. http://www.example.com
//takes to root directorty of any site
var httpReg = Regex("(?:(http|https)://\\w{1,})(\\.)")

//Handles site queries
//this is also dynamically expandable and contractable
var httpReqReg = Regex("(?:(http|https)://\\w+)(\\.)(\\w+\\2\\w+)/(?:\\w+[/]?){0,}(?:\\?)?(\\w+=\\\"[a-z]+\\\"&?)+")

These are the email links

/*
These are the URLs being matched and routed through proxy
Sorts protocol, site+(n(directories))+(n(query/params))
*/
var ftpReg = Regex("(ftp|ftps://)")
var https  = "https://wwww.google.com"
var http = "http://wwww.nigel.com"
var ftp = "ftp://modog"
var hReqURL = "https://www.google.com/asdasd/asdasd/asdas?someVar=\"y\"&someVar=\"y\"&someVar=\"y\""

Here is the execution


    var urlArray = arrayOf(http, https, ftp, hReqURL)
    for(url in urlArray) {
        when {
            httpReg.containsMatchIn(url) -> {
                println(url)
            }
            ftpReg.containsMatchIn(url) -> {
                println(url)
            }
            httpReqReg.containsMatchIn(url) -> {
                println(url)
            }
        }
    }

Here is the whole file now together:


fun main(){
    var httpReg = Regex("(?:(http|https)://\\w{1,})(\\.)") //just the sit
    var httpReqReg = Regex("(?:(http|https)://\\w+)(\\.)(\\w+\\2\\w+)/(?:\\w+[/]?){0,}(?:\\?)?(\\w+=\\\"[a-z]+\\\"&?)+")
    var ftpReg = Regex("(ftp|ftps://)")
    var https  = "https://wwww.google.com"
    var http = "http://wwww.nigel.com"
    var ftp = "ftp://modog"
    var hReqURL = "https://www.google.com/asdasd/asdasd/asdas?someVar=\"y\"&someVar=\"y\"&someVar=\"y\""
    var urlArray = arrayOf(http, https, ftp, hReqURL)
    for(url in urlArray) {
        when {
            httpReg.containsMatchIn(url) -> {
                println(url)
            }
            ftpReg.containsMatchIn(url) -> {
                println(url)
            }
            httpReqReg.containsMatchIn(url) -> {
                println(url)
            }
        }
    }
}