DEV Community: Huxley

picoCTF: where are the robots

Huxley — Sun, 15 Nov 2020 08:33:49 +0000

Challenge

Name: where are the robots

Points: 100

Description:

Can you find the robots?

Link: https://jupiter.challenges.picoctf.org/problem/56830/

Hints

What part of the website could tell you where the creator doesn't want you to look?

Solution

Go to /robots.txt

Look for the Disallow: /1bb4c.html

Go to that page

And done

Protect Your Apps From This

don't save secrets on the client 😉

picoCTF: Insp3ct0r

Huxley — Sat, 14 Nov 2020 04:16:43 +0000

Challenge

Name: Insp3ct0r

Points: 50

Description:

Kishor Balan tipped us off that the following code may need inspection

Link: https://jupiter.challenges.picoctf.org/problem/9670/

Hints

How do you inspect web code on a browser?
There's 3 parts

Solution

View source

PC: CTRL + U
Mac: OPTION + COMMAND + U
Both: Right click view source

The first flag is at the bottom of the index.html file

<!-- Html is neat. Anyways have 1/3 of the flag: *** -->

The second flag is at the bottom of the mycss.css file

/* You need CSS to make pretty pages. Here's part 2/3 of the flag: *** */

The third flag is at the bottom of the myjs.js file

/* Javascript sure is neat. Anyways part 3/3 of the flag: *** */

Python Selenium Infinite Scrolling

Huxley — Thu, 12 Dec 2019 03:36:28 +0000

Scraping web pages with infinite scrolling using python, bs4 and selenium

Scroll function
This function takes two arguments. The driver that is being used and a timeout. The driver is used to scroll and the timeout is used to wait for the page to load.

def scroll(driver, timeout):
    scroll_pause_time = timeout

    # Get scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # Wait to load page
        time.sleep(scroll_pause_time)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            # If heights are the same it will exit the function
            break
        last_height = new_height

Here is an example using the function

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup

# Your options may be different
options = Options()
options.set_preference('permissions.default.image', 2)
options.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', False)


def all_links(url):
    # Setup the driver. This one uses firefox with some options and a path to the geckodriver
    driver = webdriver.Firefox(options=options ,executable_path='./geckodriver')
    # implicitly_wait tells the driver to wait before throwing an exception
    driver.implicitly_wait(30)
    # driver.get(url) opens the page
    driver.get(url)
    # This starts the scrolling by passing the driver and a timeout
    scroll(driver, 5)
    # Once scroll returns bs4 parsers the page_source
    soup_a = BeautifulSoup(driver.page_source, 'lxml')
    # Them we close the driver as soup_a is storing the page source
    driver.close()

    # Empty array to store the links
    links = []

    # Looping through all the a elements in the page source
    for link in soup_a.find_all('a'):
        # link.get('href') gets the href/url out of the a element
        links.append(link.get('href'))

    return links

And that's how you scrap a page with infinite scrolling