DEV Community: kaelscion

VIDEO 📺 📽️ Practical Web Scraping | Auto-Upload to DEV | Part 2 - Pushing a Blog Post

kaelscion — Fri, 02 Aug 2019 22:11:22 +0000

What we've learned

We've covered the basics of logging in to DEV with a script, and you've learned the basic skills necessary to reproduce these efforts on any site you wish!

So, what's next?

Boy, am I glad you asked! In the video above, we're going to learn how to use environment variables and .env files (so as not to store passwords in the code itself), use selenium webdriver and cleverly placed input() functions to defeat oAuth device confirmations, and upload and puslih our post on DEV!

Anything else?

Of course! At the end, you'll learn where this project is going next. And throughout the video, you can see my quintissential typos, some music, a creepy turn-to-the-camera, and see what happens when I leave the editing of the video to the in-house riffer!! I really hope you enjoy it!

(by the way, this post was uploaded and published to dev using the script we'll be writing in this video!)

I Passed 3,000 Follows on Dev! 🎉🙌🎊🙌🏿🎆🙌🏾

kaelscion — Sun, 28 Jul 2019 00:24:19 +0000

Everyone on DEV is lovely, even if you don't think you are. Thank you all so much for your support, comments, reactions, and reads! I have never in my entire development career, met such a wonderful and supportive group of creatives, craftspeople, and innovators! I have tried many many times to get into blogging before and always quite because I felt that nobody cared about the content (yes, I am that insecure 😝). But with DEV, I was continually encouraged and motivated to, not only improve my writing, but my craft as a developer. Thank you all so much to this incredible community, the creators of the platform, maintainers and contributors to its OSS code, moderators who keep trolling to the lowest levels I've ever seen on a similar platform, and everyone else who makes a difference, whether you get paid to or do it for free! I encourage everyone who reads this to please leave a comment talking about their blogging journey, on DEV or off, and whether you are brand new or a seasoned veteran so that we can share our journeys together!

VIDEO 📺📹 Practical Web Scraping | Auto-Upload to DEV | Part 1 - Logging In

kaelscion — Sat, 20 Jul 2019 21:21:46 +0000

Web automation has a lot of uses. But we know that: Rotten Tomatoes, travel sites, Scraping Hub, Hootsuite, etc all use 'em. And another relatively unknown company, Google does too. So what? What can a web scraper do that would help you, in any way, personally? How about auto-uploading blog posts? If you're interested in learning how to auto-upload blog posts to DEV (the example we're using in this video), watch the video below and follow along in the series that it begins!

Best Answers to "Loaded" Interview Questions?

kaelscion — Fri, 19 Jul 2019 00:31:53 +0000

Hey all! What are some of the best stories you have about great/clever/funny answers to "loaded" interview questions? The stories don't have to be related to tech or even about you. But let's laugh and tell some stories you've heard (or been apart of on either side of the table) about our favorite answers to near-enough pointless questions like: "Would you consider this your dream job?", "Do you see yourself retiring here?", and other gems that even interviewers don't really get the point of asking. To be fair, I'll start with my favorite.

My father worked for FedEx for 42 years and just retired last year (SO PROUD OF YOU DAD!!!). In the early days of this company that has since received "utility" status, the company struggled mightily to keep from going under. For that reason, they sought to hire the absolute best salespeople they could find. One of these candidates was a good friend of my dad's.

In walks this 20-something candidate to interview with the top brass at a young Federal Express. They grilled him on sales for well over an hour and liked what they heard. Near the end, the second-in-command of the company, who had flown from Memphis to oversee interviews pointedly asked: "Would you say that this would be your dream job should you be hired?".

Without a moment's hesitation he replied, "Heaven's, no! Short-reliever for the Red Sox." (For those who don't know, a short-relief pitcher works about 2-3 hours/wk during the baseball season and is one of the highest-paid players on the roster)

The table of executives smirked, conferred with each-other, and hired him on the spot. He retired from the company a Senior VP 30 years later.

VIDEO 🎉📺 |Bye Bye 403 | Building a Filter Resistant Web Crawler in Python | Part IV - Ethics and Throttling

kaelscion — Wed, 17 Jul 2019 13:51:08 +0000

What's up everybody?! Part 4 is going to be a bit different than previous posts. How you ask? Well, it's a video!! Mainly because the topic of ethics can be really dry and boring. However, especially in a software specialty that is maligned by many, we must keep ethics and proper throttling technique involved in all of our projects.

But fear not! The following video tries to keep it entertaining. There's baby animals, a ghost hand, and some funky music so give it a watch and we'll have a good time!

Attention Web Scrapers and Pen Testers: Slither is now a PyPI package! 🎉

kaelscion — Fri, 12 Jul 2019 23:10:15 +0000

Hey data science, web automation, web scraping, and data aggregation folks. Are you tired of needing to purchase proxy IP addresses that get blocked on your goal web asset within a couple of days at most? Do you not yet have your own solution for cycling IP addresses and/or user agents? Do you like super salesy pitches like this one and tend to buy things from QVC after being asked stupid rhetorical questions!?! Well then have I got great news for you!

All kidding aside, I've finally gotten around to uploading my proxy IP and user-agent cycling library Slither to PyPI! To check out the GitHub repo, go here, for the PyPI page, head here

Only python 3 is supported and no support for python 2 is planned. This is my small way of doing my part to encourage Python 3 use over Python 2. To install it in your next project in a Python-3-only environment:

pip install slitherlib

for a multi-distro environment:

pip3 install slitherlib

To actually use the library in your scraping projects:


from slitherlib.slither import Snake
from random import choice

import requests

s = Snake()
ip_address = choice(s.ips)
user-agent= choice(s.uas)

headers = {
    "User-Agent": user-agents
}

r = requests.get('https://www.google.com', 
                 proxies={'https': ip_address, 
                          'http': ip_address},
                 headers=headers})

At this time, Slither pulls IP addresses and User-Agents from free sources around the web and dump them into two variables, ips and uas. We add new proxy ip:port sources as we can find them and verify, to the best of our ability, that they are not run by hackers looking to steal IP address information.

As this project grows, we hope to build it into a full web-scraping suite that easily supports concurrency and multi-processing, ROBOTS.txt support, webdriver browser automation, dynamic mouse-moves, and other goodies that will keep the data-collection enthusiast collecting data more and fighting 403 and 404 codes less!

If you like it, please give us a star on GitHub! I welcome bug reports, feature requests, and any comments or concerns you have so that I can make this library the best it can be! And, as always, I LOVE to collaborate so feel free to open a PR if you have improvements or ideas!

I Guess I Kinda Get it...: The Prime Generator Story

kaelscion — Thu, 13 Jun 2019 21:08:41 +0000

Today, we're going to talk about Primes. If you play Horizon: Zero Dawn and think we're talking about the Alpha Prime, Elizabet Sobek, we're not. Nor are we referring to any leader, past or present, of the Autobots. Instead, we're talking about prime numbers.

More specifically, we're going to talk about the why behind an exercise that many of us did in CS 101 or in whiteboard interviews or, at the very least, have had to explain on occasion to an interviewer: prime number generators.

Now, a prime number generator is known mostly for its use in cryptography systems, mathematical applications, and even data science. But, for most of us, its just an excercise that we do when covering the Num Theory part of university CS degrees.

So why do I decide to talk about them? Well, its because if you follow my content, you know that I like to dig into certain topics and to understand the why, not just the what. And, despite my best efforts, prime numbers have come up in my programming more often than I would like to admit. Plus, as engineers, developers and coders, it is our job to understand as much as we can about the machines we work with every day, and our duty to try and understand, as best we can, what our colleagues do every day. I, personally, believe that even little tidbits like the topic of today's blog post will help us to be better developers, even if we never write them in production ourselves. I also believe that it will help us to empathize with team members, other teams, or even friends on teams at other companies, which will make the world of software engineering better for everyone!

So, getting into it, today we're just going to cover a small, yet meaningful, tidbit on how we write efficient prime generators. I'll be using Python, because I love it and its my go-to language, but you can follow along in whatever language you like.

As we know, a Prime Number is a number that is only divisible by itself and 1. A function that generates all prime numbers between 1 and 1000 would, most often, look like this:

def prime_gen()
    for num in range(1, 1000):
        if num > 1:
            for i in range(2, num):
                if (num % i) == 0:
                    break
                else:
                    yield(num)

On my Macbook pro, the run time to print all elements in this generator is 0.564s. But what if I told you we could decrease that bigtime with only a few keystrokes, and not change the structure of the generator at all?

How? Well, you see this block of code here?

for i in range(2, num):
        if (num % i) == 0:
            break

This block tests to see if any of the numbers between 2 (because 1 is not a prime number and is skipped) and the number we are testing from prime-ness (num) are factors of num. If we find some that are, then num is not a prime number (because Primes are only divisible by themselves and 1), and therefore we can stop testing for prime-ness.

But this block actually tests twice as many numbers as it needs to, slowing the testing process down. Why is that?

To put it plainly, if you haven't found a factor of a number (aka, a number that divides evenly into it) by the time you hit that number's square root, you ain't gonna find one at all.

This is because a number's square root is a smaller number that equals the original number when multiplied by itself. This then means, that half of the factors of a number will be below the square root, and half will be above it. In turn, this means that if we're testing a number for factors and haven't found any that are less than its square root, we won't find any that are above it's square root and can safely conclude that the number is prime!

Now, armed with such knowledge, we can say with a straight face, why we only test for a prime's factors up to the square root of the number in question. It also means, that we can add the following code to our original generator:

for i in  range(2, ceil(sqrt(num)  +  1)):
    if (num % i) ==  0:
        break
    else:
        yield(num)

As a side note, in Python, the ceil() and sqrt() functions are part of the math class, so be sure to import them before trying to use them. Your completed generator, with the new code, should look like this:

from math import ceil, sqrt

def prime_gen():
    for num in  range(2, 1000):
        if num >  1:
            for i in range(2, ceil(sqrt(num)  +  1)):
                if (num % i) ==  0:
                    break
                else:
                    yield(num)

And, on the same equipment as our first test, the run time to print the elements for our revised generator is 0.023s. That is only 4% of the run time of our first generator!

So, I hope this has helped some of those out there to get a better grip on a CS principle that most of us (myself included) had an "I kinda get it..." opinion of at first.

As always, please keep coding! No matter what you do in this industry. No matter where you come from, what your social status is, what your parents did, what neighborhood/country/district you're from, or anything else that people try to convince you means you're not cut out for this gig, code on. The world needs those with every possible approach to any problem facing it. Which means they need the diversity that you can provide. Otherwise, we'll always solve the same problems the same way and get the results we've always gotten! Happy coding everybody!!!

Dynamically Filling in a CSS Grid with JavaScript

kaelscion — Fri, 17 May 2019 17:59:12 +0000

Why hello there! Today, I finally submitted. As many of you know, I am a Python developer. However, I have resolved that from now on, I will not outsource my front-end development and am forcing myself to learn JavaScript.

HOWEVER, I refuse to start by learning a "new hotness" framework or tooling system (React folks, stop typing. I'm just not digging into it yet, I'm sorry. One miracle at a time 😛) and will only be using Javascript to the ES6 standard. Please note, I just started actually writing JS code with any sort of real use today so there will plenty of "Hey, you can do that better!" opportunities in the code samples. But, I sincerely ask that you follow up those cherry-picks with how it can be better so that I can learn the right way and I thank this wonderful community in advance for any constructive criticism they might have! With that in mind, let's dive in!

So, what little project have I decided to try out first? Well, many options came to mind. But in the end, I decided to try and find a practical use for the idiotic matrix and grid traversal algorithms all of us are forced to do in the whiteboard challenge round of the dev interview process. The tiny project that I'm about to share simply creates a CSS Grid with a specified number of equal rows and columns. A separate script then places square blocks of a pseudo-random (but limited) size into the grid at a pseudo-random x and y location. This creates a fun randomized section of a page that changes on every page load. I've had great fun with it and hope you do too! The personal website for my wife and I's development studio will soon feature a version of this on our portfolio page that displays the blocks as thumbnails of our past projects.

This project will also be available on GitHub and free for anybody to modify for their own projects. If you do end up using it, please show us what you've created as we get so much more joy out of seeing what you guys build than what we come up with on our own!

So to start, let's lay out our HTML page which will serve as nothing more than a vessel for our javascript

<html>
    <head>
        <script type="text/javascript" src="gridTraversal.js"></script>
    </head>
    <body>
    </body>
</html>

Pretty basic. Bare-bones even. With that done, we can get on to the fun stuff. As I am a huge advocate of compartmentalization of code, we are going to write three functions, one to create the div that will hold our grid, one that will lay out the rows and columns of that grid, and a final one to place the blocks. Our grid creation div is created with the following JS code

const createGrid = () => {
    const markup = `
    <div class="container" id="container" style="display: grid;
                                                 border: 1px black solid; 
                                                 height:100%; 
                                                 width: 100%">
    </div> `
    document.body.innerHTML += markup
}

As you can see, this is very ES6 what with the arrow functions and template strings. But we are not importing any libraries or frameworks in this entire project other than the JS core.

I've found the easiest way, for me personally, to create HTML in JS without having to involve JSX is to simply write the markup code as a variable (in this case const markup) and append it to the body's DOM via document.body.innerHTML += markup.

All this script does is create a div that takes up the whole viewport, that has a grid display with a black border around it.

To lay out the grid columns and rows, we involve this guy

const layoutGrid = (height, width) => {
    const container = document.getElementById('container')
    container.style.gridTemplateColumns = `repeat(${width}, 1fr)`
    container.style.gridTemplateRows = `repeat(${height}, 1fr)`
}

Those of you familiar with JS should be able to see what this does pretty clearly right off. For those that aren't familiar with JS, this script selects our grid container from the DOM, and assigns two styles to it, gridTemplateColumns and gridTemplateRows, and uses the CSS repeat function to assign equally sized columns and rows using the fr css unit. The number of columns and rows are determind by the height and width arguments passed to the layoutGrid function.

So now we have a grid div that contains a grid. What about the blocks? The following script addresses that:

const fillGrid = (x, y, blockSize, numOfBlocks, color) => {
    const container = document.getElementById('container')
    const test = Array(num_of_blocks).keys()
            for (var i of test){
                let markup = `<div id="card ${i}" 
                                   style="grid-column: ${Math.floor(Math.random() * y) + 1} / span ${blockSize}; 
                                          grid-row: ${Math.floor(Math.random() * x) + 1} / span ${blockSize};
                                          background-color: ${color};
                                          border: 1px black solid;">${i}</div>`
                container.innerHTML += markup
            };
}

To pro JS devs: please be gentle. To new JS devs: I'm sorry. I am not sure what it is, but I feel like this script is very ugly. Putting that aside, let me at least explain what it does.

First, we define a function named fillGrid that accepts the arguments x, y, blockSize, numOfBlocks, and color. x and y define the starting axes of each block we're going to place. blockSize determines how many spaces in the grid each block will occupy. numOfBlocks is how many blocks we're going to be tossing into this grid and finally, color is tells the browser what the background color of these blocks will be.

Our first line selects the DOM element that contains our grid. We then generate an Array that has numOfBlocks items in it and iterate over it using a for...of loop that creates our blocks.

Each block has its size and placement defined by the Math.floor() function (which generates pseudo-random whole numbers) that are between 0 and the maximum block size established by the blockSize argument.

Each block then receives a 1px black solid border and the specified background color, and is appended to the Grid's markup.

Now, let's include our scripts in our HTML document like so

<html>
    <head>
        <script type="text/javascript" src="gridTraversal.js"></script>
    </head>
    <body>
        <script>createGrid();</script>
        <script>layoutGrid(5, 5)</script>
        <script>fillGrid(5, 5, 4, 2, "green")</script>
    </body>
</html>

Run the script and watch the fun! Every page load (or refresh) will rearrange the blocks in the grid. Play around with the size of the blocks, the number of blocks to place, and the number of columns and rows in the grid! Many times the blocks overlap and create really unique-looking grid layouts. Give it a try and let me know what you get!

If you like this kind of content, remember to give this post a like, comment, of bookmark it for reading so that I know what you all want to see next! Thanks so much for reading and I look forward to providing more content for all you Devvers in the future!

Help Testing "Sentinel": A Real-Time Proxy Bot Filter for the Web

kaelscion — Mon, 01 Apr 2019 20:58:02 +0000

Hi Devvers! So I would just like to test the waters on the community's willingness to help test a product I am developing called "Sentinel". As many of you probably know, I did a #showdev on Sentinel's sister package "Slither" insanely recently and the response has been incredible (25 stars and 4 forks since yesterday night 😱😱😱), so I thought I would toss this one out there too.

So what is Sentinel? Well, if you read my profile, you will see that I enjoy double dipping in the web crawler industry. This means that I build both sophisticated crawler bots, and the services that keep them out. Seeing as Slither is a crawler bot proxy framework, I'm sure you can guess what Sentinel does...

Over the years, I've written web crawlers that do all sorts of different things and have gained valuable insight into both how filters identify bots, and how web scrapers circumvent these filters. Being on both sides of the equation has given me a pretty unique perspective that I thoroughly enjoy!

Sentinel is a live-update crawler bot detection system that is meant to keep malicious web crawlers from getting to your resources in real time. The best part: It can be applied to specific resource pages on your website. Don't mind crawlers getting the company support phone number but don't want them scraping eCommerce data? Just tell Sentinel to only monitor your listing pages or galleries. This is done by maintaining a list, updated every 15 minutes, of proxy ips, known spammer subnets, and malicious bots and referencing every incoming connection against this list in real time! Think of it as a robots.txt that is NOT optional.

The product, as it stands, is in its infancy, but I really want to get an MVP out into the wild as soon as possible and need folks like the dev community to help me out! Let me know in the comments or via DM if you're interested in helping to test the functionality out on your websites! Thanks so much to you all for your support and interest!

Introducing Slither: Adding anonymous proxy IPs and randomized User Agents to Scraping Projects

kaelscion — Sun, 31 Mar 2019 20:21:52 +0000

Hey there Devvers! Long time, no see! I have been on vacation for the past while and thought I would return with my first open sourced project, Slither. Slither is a basic anonymizing framework for adding elite, https, anonymous proxy IPs and pseudo-random User Agents to your web scraping and/or pen-testing projects! Finally up and running and based heavily on the Bye Bye 403 series that I've been writing, Slither is my first foray into OSS and something I deeply care about and hope folks find useful!

The GitHub repo can be found here and I really hope you all enjoy it! A lot of requests and questions have come from around the web about this topic and how to make it easier to avoid the dreaded 403 (or worse, 503) when you're trying to scrape and/or aggregate data! The framework is dead simple and supports concurrent scraping as well as parallel processes.

Whenever an instance of the Slither class is declared, a list of IPs and User-Agents are pulled from proxy sites around the web and assigned to the Slither().ipand Slither().ua variables. Simple plug those two variables into your project's headers and your off and running!

I really hope this helps some newcomers to web scraping and the emerging field of collecting data for data scientists and ML engineers to use! Please give it a try and leave a comment here or on the repo. Be gentle as this is my first OSS project despite years of working in software 😝. Enjoy and happy scraping!

Bye Bye 403 - Building a Filter Resistant Web Crawler Part III: User Agents

kaelscion — Thu, 07 Mar 2019 05:08:16 +0000

So we’ve covered what web scraping is. We’ve also covered how to cycle proxy information to disguise our IP Address. But there is another step in masking the scraper that we now must consider. To illustrate, think of it this way:

Imagine that somebody is bugging you on any social media platform and you’ve blocked them. Let’s say you’re an influencer so you get hundreds of follows every day and it is really hard to do a “profile check” of every follower to make sure that they are not the person who you’ve blocked. What would be the easiest way for you to make a preliminary guess? Obviously, checking the name of the new followers would be your first line of defense. Well, what if somebody with the same name and profile picture started following you? Would the fact that their profile snippet listed them as being from a different country than before stop you from blocking them right away? Not likely. In all likelihood, it would make you more suspicious of this person because they are being deceitful.

Think of a web server bot filter as the “influencer with an annoying follower”. By cycling our IP address, we change our respective location ID, and “profile pic”, but we are still listed as a connection from the same type of laptop, with the same OS version, from the same browser, etc. It's pretty easy to peg and filter. So what is a budding web scraper to do? Cycle your User-Agent.

A User-Agent is essentially a field in your HTTP/S headers that tells the web server the browser version, host OS version, host machine form factor, and some other small tidbits about every request that comes through. This serves two purposes: diagnostics of the web content (to see if a certain bug is only reported on certain OSes, Browsers, or some combination of the two) and security (to see who’s connected and what are they connecting from to track possible offenders during and/or after a security incident). So in lamen’s terms, a User-Agent tells the web server about the connecting machine, whereas an IP address tells it more about where you’re coming from. Make sense? Good!

To begin cycling User-Agents, head here for an up-to-date collection of desktop User-Agents:

Pro Tip: We can also scrape the mobile versions of websites by changing the User Agent. That is actually how the “show desktop site” option on Mobile Browsers works. The browser will spoof your User Agent as if it came from a desktop device despite the fact that you are on mobile, thus showing you the “desktop version” of any content. The same works in reverse for scraping mobile-only content

Now, we could just copy/paste all of these user-agents into a file, but this is a web scraping post series after all so instead, create a new project file and call it user_agent.py


from bs4 import BeautifulSoup

import random
import requests

class UserAgent:

    ua_source_url='https://deviceatlas.com/blog/list-of-user-agent-strings#desktop'

    def __init__(self):
        self.new_ua = random.choice(self.get_ua_list())

So first we’re making the UserAgent class. If you’ve made it this far into the series you can probably put together that I am a big fan of encapsulation. This is largely due to coming from the Network Engineering and IT Technical worlds before I started writing code where, it is assumed, that an incident WILL wake you up EVERY TIME you are on call at no earlier than 4 AM. Because of this, I am a huge fan of creating things that are easily debugged. Getting paged at 4, getting to the office by 4:30, and leaving by 4:45 to actually have some semblance of a morning with my family is preferred to taking 3 hours to fix a simple issue that was buried under layers of junk. Because if it takes me 3 hours, then I have solved my problem by 7:30. You know, just in time to start “normal” work for the day…

That being said, the UserAgent class is going to handle all the user-agenty things for us! The class variable we’ve created ua_source_url is outside of the __init__ function, as you might have noticed. Why is that? Because everything inside __init__ only “exists” after an instance of the class is called (aka instantiated) in your code. Class level variables can be referenced directly and more importantly, are shared among all instances of that class. Meaning when we multithread this bad boy, we won't have to store that particular string in memory more than once.

Within our __init__ function is the new_ua property which we want to be different for each instance of this class but not be pulled into existence until the class is created.

Side Note: For a further understanding of __init__, what it does, and how to use it, stick around. There is a post coming on that very soon!

The new_ua property picks a random user agent from a list of user agents. Where will we get that list you ask? From the code your about to type:


def get_ua_list(self, source=ua_source_url):
    r = requests.get(source)
    soup = BeautifulSoup(r.content, "html.parser")
    tables = soup.find_all('table')
    return [table.find('td').text for table in tables]

So what have we done here? Well, we have scraped the page above (as we do) for the user-agent information listed on that page. As with all of our little scraping scripts, it started with a Requests get() function, fed the response content to our HTML parser (BeautifulSoup), found the table markup for each user-agent, and extracted the text of the data of each table. Now that this function is complete, does the __init__ make a bit more sense? If not, allow me to explain:

As soon as we create an instance of the UserAgent class ANYWHERE in our code, it will run this function without us needing to touch it. When the function completes, it will have assigned the list of potential UserAgents to the property new_ua. How is this relevant? Take a look at the terminal screen capture below which shows the output of the following code:

i=1
while i <= 4:
    print(UserAgent().new_ua)
    i+=1

As you can see, we do not need to write the user agents to a file, which involves opening the file, reading it, then closing it again, relying on disk memory. Instead, we read this list into RAM (the faster, easier-to-clean storage) and retrieve it only when we need it, getting a different UserAgent each time through the random.choice() function

Pro Tip: For those of you who are unfamiliar with random.choice(), it basically does the same thing when selecting from a list or iterable as random.randint(0, len(sample_list)). Simply put, it selects a pseudo-random item from any iterable data structure you pass to it. In instances like this, random.choice() is my preference because both approaches do roughly the same thing and .choice() is fewer keystrokes.

Alright, Mavericks (Season of the Drifter anyone 😉) you've got the tools to fool 99% of bot filters. But do you know how to use them? Head out into the Wild and find out, then report back. I'd love to hear your stories of the datasets your gathering and what you plan to do with them! Successes and failures are equally welcome because this completes the first set of countermeasures we're going to discover. These are referred to (by me at least) as the "hard skills" of web scraping. The skills we need to develop our knowledge. But how are your instincts? Find out next week in part four when we break into the intermediate stuff: Timing and Throttling!

Life Can Be Tough, But We Are DAMN Good at What We Do!

kaelscion — Fri, 08 Feb 2019 15:54:25 +0000

Devvers, lend me your ears...eyes...ears? Whatever, just listen. The past six months have been rough for me personally. Health issues always suck, but they're a real pain in the ass when they almost kill you. Something like that can change the way you look at the world. That being said, it can be a surreal experience when you are suddenly forced to acknowledge that you do indeed matter, even in a small way, and that you can only ignore your body and mind's signals for so long before stuff hits the fan. I just got out of a critical care unit Monday and need to share the good with those who may need it just like I do. Because what is a community for if not support, encourage, and unconditional love (some of ya'll like to be prickly enough to only get love unconditionally sometimes. You know who you are, and I dig you anyway 😋)?

So, here's what we're gonna do: Anybody who reads this post that inherently thinks what they do, either personally or professionally is too small to matter. For all of us out there that think "Junior Developer" translates to "coffee-getter with no talent or skills". For anybody that is having a harder and harder time refuting other's assertions that DevOps/QA/Front End/DBA work is just the crap that is put in place to annoy the "real geniuses" so that "the company can feel good about including non-critical staff in important work". For any person in tech that is female, brown or black, wears a hijab, dashiki, or yukata to work, or is just otherwise looked down on because of external crap or having the gall to value your cultural or religious traditions over the extreme views or ignorant opinions of others, this thread is for you. Whether you have an invisible illness that people think you are faking or a visible difference to your appearance that people think you should change, we're gonna show the "boss" in all of us.

Here's my contribution: Right now I am currently building a system for a client of mine that has one of the most low-level and accurate testing suites for GMOs in plants. They are tasking me with giving them a "market snapshot" of who the players are in the Corn Ethanol biofuel market globally so that they will better know how to make decisions with their products and services based on where their target customers are moving as an industry. To do this, I am putting my Data Junkie skillset to use by finding the company information (location, employee count, gross income, stock price, etc) of all the players in the US. Below is a screenshot of the runtime of two scripts that do this by extracting that particular data from the websites of the 204 players in the Continental US. The top runtime is a solution I designed and created for this purpose. The second is of a "copy and paste" solution I found online that is lightly modified to perform the same task as my custom solution.

"Hey wait, Jake. Are you grandstanding here?" How dare you ask that entirely true, accurate, and appropriate question! But seriously, that what this thread is about. I am a lone-wolf, back-end, guy-who-builds-stuff-that-is-never-seen-and-never-acknowledged kind of guy. And I have been for many years. Tooting my own horn is something I'm not usually comfortable with. But here, in this thread, BRAG YOUR BLOODY HEADS OFF. Because every now and then, we need to shout to anybody who will listen how damn smart, creative, and clever we are. Nobody is going to do it for us, so go nuts! Post screenshots or tell stories of things you've done that you think are really cool, innovative, clever, or elegant. Talk to me, and the community about your pet projects or nifty solutions that you're too nervous are not "cool" enough to get positive attention from your colleagues. They are not only cool enough, but they are too cool for those guys. So tell us, and don't be modest about it 😁 😁 😁! I very much look forward to your submissions!