DEV Community: CCONLEY-FI

Building and running javascriptlets for advanced data acquisition.

CCONLEY-FI — Sat, 17 Feb 2024 01:11:49 +0000

In an effort to utilize the full potential of an Android device, I decided to make a short scriplet for web scraping. Particularly for finding specific file types like PDFs, EPUBs, or JPGs, the combination of Javascriptlets, Termux, and enhanced browser functionalities offers a compelling solution. This detailed guide walks through setting up the necessary tools and crafting scripts to automate the search for these file types directly from an Android device, illustrating the process with practical examples.

Initial Setup with Termux

Termux is the backbone of this operation, providing a powerful Linux environment on Android. After installing Termux from the Google Play Store, or F-droid if needed, the following commands will prepare the environment for scripting:

pkg update && pkg upgrade
pkg install python
pkg install git

These steps ensure that the Termux environment is ready for advanced operations, including web scraping tasks.

Enhancing Capabilities with Browser Extensions

To augment the web scraping process, installing browser extensions on a compatible browser like Kiwi or fenix(firefox) can significantly streamline operations. Adding an extension like Tampermonkey or Mobile Dev Tools enables the user to manage and execute Javascriptlets with ease, facilitating the automation of web tasks directly from the browser.

Crafting Javascriptlets for File Search

Javascriptlets can be designed to initiate searches for specific file types across the web. Here’s a concise script aimed at finding PDFs usong googles search logic:

javascript:(function() {
    var query = encodeURIComponent('filetype:pdf');
    var url = `https://www.google.com/search?q=${query}`;
    window.open(url);
})();

Adapting this script to search for EPUBs or JPGs is as straightforward as changing filetype:pdf to filetype:epub or filetype:jpg in the script.

Advanced Web Scraping with Termux

For more nuanced scraping tasks, such as parsing search results to extract specific URLs or directly downloading files, Python scripts executed within Termux are exceptionally useful. Tools such as Beautiful Soup can parse HTML content to find and list downloadable links. Here's an example script that searches for downloadable PDF links on a webpage:

import requests
from bs4 import BeautifulSoup

def find_downloads(url):
    page = requests.get(url)
    soup = BeautifulSoup(page.content, 'html.parser')
    links = soup.find_all('a', href=True)

    for link in links:
        if link['href'].endswith('.pdf'):
            print(link['href'])

if __name__ == "__main__":
    target_url = 'https://example.com'
    find_downloads(target_url)

This script could be easily modified to search for .epub or .jpg files by replacing .endswith('.pdf') with the desired file extension in the script.

Automating and Scheduling with Termux

To automate the execution of scripts for repeating data collection, Termux supports scheduling through cron jobs. This functionality allows scripts to run at specified intervals, ensuring continuous data collection without manual intervention:

echo "0 * * * * python /path/to/find_downloads.py" | crontab -

This command sets the find_downloads.py script to run hourly, demonstrating Termux’s capability to automate web scraping tasks.

Conclusion

Leveraging the capabilities of Javascriptlets for initiating web searches, coupled with the power of Termux for advanced scripting and scheduling, users can effectively automate the search and collection of specific file types like PDFs, EPUBs, and JPGs on their Android devices. This approach not only makes targeted data collection more accessible but also significantly expands the scope of projects that can be undertaken directly from a mobile device, showcasing the practical and versatile applications of these tools for sophisticated web scraping tasks. Use your own creativity to develop other use cases. Keep in mind that you should always consider a sites usage rules and legal processes.

Post-Bootcamp Studies as a Software Engineer

CCONLEY-FI — Mon, 29 Jan 2024 18:31:01 +0000

Introduction

As a current Bootcamp student and aspiring Full-stack Developer, the question on my mind, as well as the minds of fellow cohorts, is quite simple. When we're done here, how do we land the ideal career? With that in mind, I've started to take a look at the field of AI integration and Development. What follows are a few overarching core concepts for breaking into the industry at an entry level, compiled from the perspective of a nascent software engineer.

Core Languages One Should Know (or at least be familiar with)
JavaScript is indispensable for web development, enabling both client-side and server-side applications through frameworks like React and Node.js. As it is considered a foundational language among most Development teams, it is not uncommon for technical interviews to involve a demonstration of proficiency in JS, if not the frameworks.

Python excels in simplicity and versatility, making it ideal for various applications, from web development to data science. Python has one of the most user-friendly data structures out there; it can be found in industries as varied as gaming to medical databasing, and everything in between.

An example of Python code used to fine-tune AI modeling:

import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.datasets import mnist

# Load dataset (for example, MNIST handwritten digits dataset)
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

# Normalize the images
train_images = train_images / 255.0
test_images = test_images / 255.0

# Build the model
model = Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(train_images, train_labels, epochs=5)

# Evaluate the model
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)

print('\nTest accuracy:', test_acc)

Java and C# have and without a doubt will remain critical for enterprise and mobile app development. Particularly for devs working with legacy code found in the majority of long-standing code found in large companies.

An example of Spring Boot, a popular web building framework for Java:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class ExampleApplication {

    public static void main(String[] args) {
        SpringApplication.run(ExampleApplication.class, args);
    }

    @GetMapping("/hello")
    public String hello(@RequestParam(value = "name", defaultValue = "World") String name) {
        return String.format("Hello %s!", name);
    }
}

Directed Study Resources

Focused learning platforms such as Pluralsight and LinkedIn Learning are often quoted as prime learning resources with a broad, if somewhat general, scope. EdX and Coursera provide access to university courses that deepen foundational knowledge, giving you a bit more depth to your learning. As a resident of Colorado, I've also recently learned that Udemy is provided to anyone possessing a library card. Keep an eye on local resources and benefits!

Project-based learning is crucial to any developer. One of the first pieces of advice given to me was "don't stop coding." I've learned almost as much from making side projects which interested me.

Familiarity with agile methodologies and DevOps practices, including tools like Jenkins and Docker, is becoming increasingly important in modern development environments.

Essential AI Readings

For those aiming to specialize in AI, reading and understanding the work already done by others is a must. A few books which are widely recommended are:

"Artificial Intelligence: A Modern Approach" by Stuart Russell and Peter Norvig
"Pattern Recognition and Machine Learning" by Christopher M. Bishop
"Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
For a broader perspective, especially on the implications and implementations of AI, "Weapons of Math Destruction" by Cathy O'Neil is a must-read. It takes on the challenge of highlighting the impact of widely used algorithms and the effect they have on society as a whole.

A beginner's experience working with JavaScript: The Superhero Showdown

CCONLEY-FI — Fri, 05 Jan 2024 17:41:26 +0000

By Chris Conley

As part of my early experiences in the Flatiron School's coding program, I undertook a project that served as my introduction to the fundamentals of web development—specifically, event listeners, DRY programming, and API/JSON fetching. The project, named "Superhero Showdown", was a web application that allowed users to engage with a short list of superheroes in a simulated battle environment. This endeavor provided a hands-on approach to learning JavaScript, HTML, and CSS. Here, I share the process and insights gained from this project.

Project Overview

The end goal of the Superhero Showdown project was to create an interactive web app where users could select superheroes, view their statistics, and compare them in a very basic way. The project required the integration of front-end development using JavaScript for interactive functionality, HTML for the web page's structure, and CSS for its visual design. I also dipped my toes in backend dev when fetching from the API and interacting with our .json.

Data Management and Dynamic Rendering

The first iteration of the project was going to utilize access to a free API "SuperHeroAPI" for fetching data. After wrestling with fetch and some unforeseen challenges around importing an apikey, I determined that it would simply be more efficient to create a .json from which to fetch our heroes' data.

Interactive Features

Incorporating interactive elements, ie: selecting/creating a hero and applying visual effects(mouseover,mouseout) to meet deliverable requirements, represented a notable learning curve. This part of the project helped me understand the importance of scope when working with iterative programming and event listeners in JavaScript, creating dynamic user experiences.

HTML and CSS: Structure and Design

The project also served as a practical introduction to HTML and CSS, helping me become more comfortable with structuring web content and applying styles.

HTML Structure

I furthered my understanding of organizing a webpage using basic HTML elements. While I had some basic and conceptual knowledge of HTML, I explicitly learned about the challenge of creating vs modifying HTML; learning how to expand my HTML code in tandem with my JS/CSS as needed.

CSS Styling

Applying CSS was the next step. I learned about simple properties, flex, filter, grid, etc. for layout design and eventListener interactions. Major credit for the CSS refactoring goes to my collaborator in this project, Nick Gallegos.

Challenges and Key Learnings

One of the key challenges I faced was managing the state of the application, particularly in resetting selections after each showdown. Furthermore, I faced a handful of fretful refactors and one distressing wipe/fresh start to our project in order to properly implement our core "renderHero" function. While I hesitated in the reconstruction process, there's no doubt that I was able to learn a great deal from my mistakes.

Conclusion

The Superhero Showdown project was a foundational experience in my coding journey, being the first time I was able to create something from nothing. It emphasized the importance of JavaScript in adding interactivity, the role of HTML in content structuring, and the impact of CSS in design. This project was instrumental in building my understanding of web application development from the ground up.

Keep an eye on my journey!

Nick Gallegos SuperHero-Showdown collaborator and aspiring full stack wizard!

SuperHero-Showdown