DEV Community: Kyle Knapp

The Social Architecture of Engineering

Kyle Knapp — Fri, 01 Mar 2024 04:58:25 +0000

Intro

People skills – the often overlooked superhero of an engineer's toolkit. Picture this: you study computer science in university, thinking your days will be a solo codeathon, a language spoken only between you and your screen.

But here's the plot twist: reality hits different. Engineering isn't a one-person show; it's a full-on ensemble. It thrives on collaboration, synchronized efforts, teamwork, and a culture of mutual support.

Guilty as charged – I fell into the "I'll just code all day" trap too. Time to unravel the truth. Let's dive into the intricacies of engineering, where good collaboration isn't just a bonus, it's the main feature. Ready for the real story? Let's break it down.

Turns out there is more to this job than just coding...

Before my start into the world of programming, I held a simple belief: engineers spent their days immersed in code, with little need for interaction once the coding tasks were done. No customer talks, project discussions, or client meetings – just pure coding, or so I thought.

Reality, as it turns out, had a different script. Upon embarking on my programming journey, I discovered a profound satisfaction in constructing things, in the perpetual process of learning and improvement. The idea of spending an entire day engrossed in coding without the need for conversation seemed like a dream job.

Upon entering my first role as an engineer, a paradigm shift reshaped my perspective. It dawned on me that I lacked the needed skills and a comprehensive understanding of the business requirements crucial for the accurate implementation of the coding project. What became unmistakably evident was the imperative for collaboration and seamless communication.

Engineering is one of the most collaborative fields out there

There is so much need for collaboration and alignment to effectively deliver solutions that can scale long-term. Just 1 person working in isolation and making decisions will cause a lot more harm than help.

The reason is that for every piece of code we create, someone needs to maintain it. If 1 person is just making isolated decisions → nobody else will know what those decisions were. All of the future changes would depend just on that person.

The goal is to create software that is easy to maintain and adjust based on future needs. And you can only do that with great minds that are working together.

Altering a couple of words in your overall mindset can create a significant impact.

Shifting your mindset from "How can I deliver this solution" to "How can WE collectively deliver this solution" will yield a profound difference.

Final Thoughts

In moments of uncertainty about the right course of action, consistently prioritize what will benefit the team and the organization as a whole. You can never go astray with that guiding principle!

Mastering Web Automation: Building a Selenium Crawler Bot

Kyle Knapp — Mon, 29 Jan 2024 04:52:43 +0000

Before we get started let's talk about robots, not the ones from the movies like WALL-E or Terminator I'm talking about the ones that live on the internet known as bots. Here is a quick reference guide before we get started:

According to cybersecurity firm Imperva, an impressive 47.4 percent of all internet traffic in 2022 originated from bots. These digital entities assume a variety of roles, each pivotal to the seamless functioning of the online realm. From enhancing user experiences to streamlining tasks and fostering communication, bots have become indispensable. Search engine bots meticulously traverse web pages, ensuring content is efficiently indexed for delivering relevant search results. Social media bots, on the other hand, deftly automate posts, likes, and interactions, thereby optimizing content visibility. In real-time, chatbots engage users by providing immediate support or guiding them through websites. It's worth noting, however, that malicious bots lurk in the shadows, engaging in activities such as spamming, phishing, or launching cyber attacks. Beyond this, web scraping bots adeptly extract data from websites for in-depth analysis which we will discuss this type of bot in deeper detail with a project I worked on earlier in the month.

The Goal

Before I discuss my project I worked on lets first discuss what I was trying to accomplish. To keep things simple I'll explain the scenario - over christmas time there was a gift I was looking to buy for a family member the problem was that this item was only available online, limited quantity and would be released at a random time during the day. A recipe for disaster for the average person, luckily for me I had the tools on my side to have a shot as getting this high ticket item. That tool is called Selenium.

What's Selenium?

So what is selenium? Selenium is an open-source framework primarily used for automating web applications. It provides a suite of tools for web browser automation, allowing developers to interact with web pages programmatically. Selenium supports various programming languages, including Python, Java, C#, and more, making it versatile for different development environments. It enables the simulation of user actions such as clicking buttons, filling forms, and navigating through web pages, facilitating the testing of web applications. Selenium WebDriver, a key component, directly communicates with browsers, enabling seamless automation and testing across different browsers and platforms. Overall, Selenium is a powerful tool for automating repetitive tasks, conducting testing, and ensuring the robustness and reliability of web applications.

The Project

Now that we have the goal and tool understood lets talk about implementation. To kickstart this project I needed to choose a programming language along with a solid IDE to pair this with to utilize selenium. I ended up choosing Python as my language and VSCode as my dev environment. Now that the stage is set I began coding away. I kept the code relatively simple, it contains basically just two core functions:

Main()

To summarize the code in Main() it first reads the file named pings.txt to get a list of URLs that will be scraped. It then reads a pickled file named previous.pickle to get the previous HTML states of the tracked websites. After, we iterate over the URLs, retrieves the current HTML, compares it with the previous state, and logs any changes. Finally it then writes the change report to a file named updates.log.If changes are detected, and the script is running on a Windows system, it shows a toast notification using the win10toast library.

parse_pings()
Parse_pings does just two things, both very important though. It first parses ping_file with all information about the data it scraped, including URLs, actions and responses. This data will then be packed and returned into a list of tuples for evaluation.

The Code

import argparse
import datetime
import pickle
import os
import sys
from pathlib import Path
import platform
from collections import defaultdict


from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import WebDriverException
from tqdm import tqdm
from win10toast import ToastNotifier


CLICK_MACROS = {
     '#EBAY_KL': ['//*[@id="edit-birthmonth"]']
}

TARGET_MACROS = {
}



class Spider:
    def __init__(self, headless=True):
        chrome_options = Options()
        if headless:
            chrome_options.add_argument("--disable-gpu")
            chrome_options.add_argument("--headless")
            chrome_options.add_argument("--silent")
            chrome_options.add_argument("--log-level=3")

        try:
            driver = webdriver.Chrome(options=chrome_options)
        except WebDriverException:
            decision = input('add bundled chromedriver to path? y/[n]')
            if decision not in ['y', 'n'] or decision == 'n':
                sys.exit(1)
            else:
                dir_path = Path(os.path.dirname(os.path.realpath(__file__)))
                sys.path.insert(0, dir_path / 'chromedriver.exe')
                print('Warning: supplied chromedriver is intended for Windows and Chrome version 83.')

        self.driver = driver

    def get_current_html(self, url, clicks, target, html_or_content= 'content'):
        print(url)
        self.driver.get(url)

        # perform actions
        for click in clicks:
            element = WebDriverWait(self.driver, 10).until(
                EC.presence_of_element_located((By.XPATH, click))
            )
            element.click()

        # get html
        element = WebDriverWait(self.driver, 10).until(
            EC.presence_of_element_located((By.XPATH, target))
        )

        if html_or_content == 'html':
            # more accurate, but problematic due to js content etc.
            html = element.get_attribute('innerHTML')
        else:
            # more reliable:
            html = element.text

        return html

    def __del__(self):
        self.driver.close()


def parse_pings(ping_file):

    pings = []

    with open(ping_file)as inf:
        for line in inf.readlines():
            if line.strip() == '' or line.startswith('#'):
                continue
            url, actions, target, triggers = [x.strip() for x in line.split('|')]
            triggers = triggers.split(';')
            triggers = [x.strip() for x in triggers]

            if actions.startswith('#'):
                print('Reading click macros')
                actions = CLICK_MACROS[actions]
            elif actions == '':
                actions = []
            else:
                # split actions
                actions = actions.split(';')
            if target.startswith('#'):
                target = TARGET_MACROS[target]
            elif target == '':
                target = '//body'

            ping = (url, actions, target, triggers)
            pings.append(ping)

    return pings


def main():
    # parse args
    parser = argparse.ArgumentParser(description='Crawl a number of sites and compare them to a previous known state.')
    parser.add_argument('--show_driver', action='store_true', default=False, help='Disables headless mode for webdriver.')
    parser.add_argument('--no_notify', action='store_true', default=False, help='Disable toast notifications.')
    parser.add_argument('--html_or_content', choices=['html', 'content'], default='content', help='Whether to compare html or content.')
    args = parser.parse_args()

    # read pings
    pings = parse_pings('./pings.txt')

    previous_htmls = defaultdict(lambda: {
        'time': None,
        'html': ''
    })

    # read previous htmls
    if os.path.isfile('./previous.pickle'):
        with open('./previous.pickle', 'rb') as inf:
            previous_htmls.update(pickle.load(inf))

    # store new htmls separately in order to discard no longer tracked sites
    new_previous_htmls = defaultdict(lambda: {
        'time': None,
        'html': ''
    })

    spider = Spider(headless=(not args.show_driver))

    stuff_changed = False
    changes = []
    for url, clicks, target, triggers in tqdm(pings):
        try:
            current_content = spider.get_current_html(url, clicks, target, args.html_or_content)
        except Exception:
            tqdm.write(f'Retrieving html failed for: {url}')

        # do nothing if not of the given triggers are fired
        if len(triggers) > 0 and not any([x in current_content for x in triggers]):
            continue

        # store "new previous"
        new_previous_htmls[url]['html'] = current_content
        new_previous_htmls[url]['time'] = datetime.datetime.now()

        if previous_htmls[url]['html'] is not None \
            and current_content is not None \
            and current_content != previous_htmls[url]['html']:
            tqdm.write(f'Changed: {url}')

            stuff_changed = True

            # save differences
            changes.append({
                'url': url,
                'previous_time': previous_htmls[url]['time'],
                'previous_html': ' '.join(previous_htmls[url]['html'].split()),
                'current_time': datetime.datetime.now(),
                'current_html': ' '.join(current_content.split())
            })

    # write change report
    with open('./updates.log', 'a+', encoding='utf-8') as out:
        for change in changes:
            out.write(f'{change["current_time"]}, URL {change["url"]}\n')
            out.write(f'\tBefore:\t{change["previous_html"]}\n')
            out.write(f'\tAfter:\t {change["current_html"]}\n')
            out.write('----------------------------------------------\n')

    if not args.no_notify and stuff_changed and platform.system() == 'Windows':
        # create an object to ToastNotifier class
        n = ToastNotifier()
        n.show_toast("ProjectPing", f"{len(changes)} change(s) detected. Check log for details.", duration=5,
                     icon_path="./assets/spider.ico")

    # write current htmls as new previous
    with open('./previous.pickle', 'wb') as out:
        pickle.dump(dict(new_previous_htmls), out)


if __name__ == '__main__':
    # execute only if run as the entry point into the program
    main()

The Results

The script efficiently parsed the file of URLs, retrieved previous HTML states, and dynamically compared changes, recording a detailed change report in the updates.log file. When the state page detected a change I was able to trigger a notification which allowed me to complete the transaction successfully. This project not only showcased the efficacy and reliability of Selenium in automating repetitive tasks but also highlighted its versatility beyond testing, serving as a valuable tool in the dynamic landscape of online commerce.

Mastering the Modern Maze: Why Being a Software Engineer is Tougher Than Ever

Kyle Knapp — Sun, 31 Dec 2023 19:32:31 +0000

In my four-year journey through the software development industry, I've witnessed a rapid transformation in the field. With the emergence of AI, economic shifts, and a highly competitive job market, the entry barriers have noticeably raised. In this article, I'll dive into some specific observations I've made during this time, offering insights shaped by my own experiences. From the changing role of AI to the impact of economic trends, I'll provide a detailed perspective on the evolving dynamics of the software development landscape.

Lets start with a picture

Take a look here at "Your Perfect Development Team". Are you also looking at this picture and comparing it to your team and wondering why you might be missing some of these roles? Well it turns out the industry has started to condense these roles into one single position and title it "Full-Stack Software Developer" just with this title alone you can eliminate UI designers, software architects and maybe even a QA engineer just by changing the title. With this happening the bar for being a software developer just raised. Instead of just knowing how to code you must now learn front-end frameworks, testing strategies and even how to architect the code you are deploying. This has now increased the learning curve, the barrier for entry is now higher because more skills are required and on top of all that this removed positions from the job listings so there are even less jobs for the average developer to apply to.

Ever-growing markets and increased competition

The ever-changing landscape of the tech market is a well-known reality. Throughout the years, we've witnessed remarkable triumphs, ranging from WhatsApp and Uber to Airbnb and TikTok. While these instances may be considered outliers, they often serve as aspirational benchmarks for individuals, setting a standard to reach for. This mindset, however, adds an additional layer of pressure on engineers. Internally, they grapple with self-imposed stress and a loop of comparison, questioning whether they are on the right track. Externally, the scrutiny intensifies as others compare their company to numerous competitors, amplifying the challenges faced by these professionals.

Thriving in this environment demands companies to adhere to the famous Facebook motto, "move fast and break things." In the current landscape, the imperative is even more pronounced, with nearly every company transitioning into an "AI company." The widespread integration of ChatGPT, often without a clear strategy or genuine purpose, is becoming increasingly prevalent.

When companies deviate from the fundamental question of how to enhance value for customers and instead focus on outpacing competitors in the hype-driven race, engineers find themselves constructing functionalities destined for failure.

Swift Pace, Big Demands

In the pursuit of new challenges, striking a balance between constructive stress and avoiding its negative counterpart can be challenging. Positive stress arises when we view a demanding situation as an opportunity with a favorable outcome, whereas negative stress can have adverse effects on both physical and mental well-being, as illustrated below. Consistently delivering results within tight timelines can be stress-inducing, and developing features without sound reasons tends to lean towards the detrimental side of stress.

As both a mentor and a dedicated team member, I've witnessed how stress can lead to burnouts. Invariably, individuals experiencing burnout exhibit lower productivity compared to those who address the issue by reducing workload, taking breaks, and optimizing for the long term. This is why we emphasize the importance of transparent communication in our approach, making it easy to raise red flags through periodic team updates, one-on-one discussions, and fostering a nurturing culture.

Despite our efforts, challenges may still arise. In such instances, we proactively adjust our approach by cutting scope, seeking assistance from relevant parties, or communicating with clients to postpone a launch.

The current landscape brings heightened pressure from both internal and external sources. It's crucial to maintain focus and surround yourself with a supportive team dedicated to long-term optimization.

What can we do?

We all experience challenging moments, grappling with feelings of inadequacy as engineers, mentors, or colleagues. Perfection and ease are not realistic expectations, nor should they be. Mistakes and difficult times are essential for learning and growth. However, there are certain practices I've discovered that can enhance efficiency.

On a individual level, stay aligned with what truly matters. Technologies may come and go, but the enduring impact you make on the world is what truly counts. Authenticity in your commitment to hard work and accumulated experience is something that's challenging to replicate.

Within the company framework, commence decision-making by understanding the "why." This approach is the most effective means to develop functionalities that genuinely contribute value. Additionally, ensure that employees comprehend the reasoning behind decisions. In my experience, this understanding empowers them to make more informed choices, provide valuable feedback, and ultimately fosters a greater sense of satisfaction.

Culturally, institute processes that provide sustained support for employees. Collaborate with colleagues to establish a culture rooted in trust, support, and empathy. This collective ethos ensures that each member of the team brings out the best in one another.

Embracing Delayed Gratification for Long-Term Success

Kyle Knapp — Sun, 26 Nov 2023 04:20:23 +0000

In our modern society, we find ourselves immersed in a culture that places a premium on the allure of immediate gratification. Whether it's the tempting prospect of striking it rich overnight through a lottery ticket, embarking on a rapid and extreme diet for quick weight loss, or simply succumbing to the allure of impulse buying at the store – we've all been there. But why is this tendency so prevalent? The tendency to choose immediate rewards over delayed ones is deeply ingrained in our biology and psychology. Across the course of evolution, our ancestors prioritized immediate survival needs, laying the foundation for our predisposition toward swift gratification. The release of dopamine in our brains - linked to pleasure, further reinforces this behavior when we experience instant rewards. Factors such as genetic differences, financial circumstances, and individual brain function contribute to impulsivity, creating a challenge when resisting immediate temptations. Additionally, cultural values that champion consumerism and the pervasive influence of technology, with its instantaneous feedback loops, intensify our craving for quick satisfaction.

It's time we all take a breather and shift our perspective beyond the immediate horizon, and explore the countless benefits of delayed gratification. Let's delve into the advantages of building for the future – after all, as the saying goes, "Remember Rome wasn't built in a day".

It's All Around Us

In our modern world, the allure of instant gratification surrounds us at every turn. From the flickering glow of fast-food signs on street corners to the ever-present dings of notifications on our devices, the culture of immediacy weaves seamlessly into the fabric of our daily lives. The temptation is everywhere, and it's not your fault. Society, with its abundance of shortcuts and quick fixes, has cultivated an environment where the appeal of immediate rewards is hard to resist. The ease of online shopping, the swift satisfaction of a streaming service binge, and the instant validation of social media likes — these are all part of a landscape designed to cater to our impulsive tendencies.

Take Charge

Recognizing and observing these pervasive influences is the first step in navigating a path toward delayed gratification benefits. By acknowledging the subtle yet powerful impact of instant rewards in our lives, we empower ourselves to take charge of our decision-making. It's about reclaiming control over the choices we make, understanding that the allure of immediacy is a product of both biological instincts and societal conditioning. Armed with this awareness, we can intentionally pause, evaluate, and opt for actions that align with our long-term goals. It's a conscious effort to resist the gravitational pull of impulsive decisions and, instead, choose the path that leads to lasting fulfillment. Taking charge means embracing the notion that delayed gratification is not a sacrifice but an investment in a more rewarding future.

Reap the Benefits

Picture yourself aspiring to run a marathon. Instead of seeking quick fixes or shortcuts, the emphasis is on shaping a lifestyle conducive to sustained success. Choosing a nutritious, wholesome meal over the allure of fast food from a fast food joint becomes a deliberate choice. Opting not to skip a workout, even on days when fatigue lingers, becomes a commitment to the larger goal. These small but intentional decisions are the building blocks of success, illustrating the transformative power of delayed gratification. It's about sowing seeds today to reap a harvest of achievements tomorrow. By making choices that align with our aspirations, we not only set ourselves up for triumph in the long run but also cultivate a sense of discipline, resilience, and lasting well-being. In essence, it's the journey of recognizing that the path to accomplishment is often more rewarding than the destination itself.

A Quick Philosophical Story

Last week I listened to one of Alan Watts college lectures from the 1960s. He brought up a story that really resonated with me based around this subject of instant gratification. It goes something like this:

Once upon a time, in a quaint village, there lived a farmer known for his impatience. Eagerly anticipating the harvest season, he found himself growing increasingly restless as he watched his crops slowly emerge from the earth. Frustrated by the seemingly sluggish pace of growth, he hatched a plan to expedite the process.

One day, unable to contain his impatience any longer, the farmer decided to take matters into his own hands. Believing he could accelerate the growth of his crops, he began pulling on the tender stalks, hoping to stretch them and coax them into maturing more quickly.

However, his well-intentioned efforts proved to be misguided. The plants, unable to withstand the forceful intervention, began to wither and wilt. The more the farmer pulled, the more damage he inflicted upon the very crops he sought to hasten.

It wasn't long before the village wise man, observing the farmer's actions, approached him with a gentle smile. "My friend," he said, "you cannot force the hands of nature. Just as a flower needs time to bloom and a tree requires years to grow, so do our endeavors in life demand patience and the nurturing hands of time.

The tale of the impatient farmer serves as a timeless reminder that, much like the crops in the field, aspects of life, personal growth, and understanding cannot be rushed. It encourages us to embrace the natural unfolding of events, appreciating the beauty that comes with allowing things to develop organically, in their own time. In reflecting on the farmer's impulsive act of pulling on the plant, I couldn't help but see the stalk as a metaphor for short-term impulses. The more frequently these impulses are indulged, akin to the farmer's repeated pulling, the quicker the plant—symbolizing our long-term goals—wilted and perished. This analogy underscores a profound truth: consistent yielding to short-term impulses can undermine and eventually dismantle our aspirations for long-term success

Conclusion

Take a step back and figure out what you want to make a long term goal, write down a measurable timeline to achieve this objective and ACT ON IT. Remember why you started the journey and be open to where it might take you. It could be financial goal, a professional goal or even a personal goal, there is no wrong answer - it's just meant to improve your overall well being long-term. Your future self will remember the sacrifices you made to get where you are and a new level of discipline will be awarded. Look beyond the daily noise you encounter and keep your head down and get to work!

Anchored for Success - The Unspoken Perks of Staying

Kyle Knapp — Fri, 21 Jul 2023 01:43:03 +0000

In the dynamic world of software development, job hopping was once a common practice – a way to seek better pay, learning opportunities, or a way to climb the corporate ladder. The allure of exploring diverse work environments and acquiring new skills in quick succession appealed to many ambitious developers. However, recent shifts in the tech industry have prompted a reevaluation of career trajectories, with factors like FAANG layoffs, the COVID-19 pandemic, fluctuating interest rates, and inflation influencing developers' decision-making processes. While these challenges are prevalent and impactful, the focus of our discussion today is centered around the benefits of staying committed to a company long term and the invaluable takeaways this approach can bring to your career.

Reflecting on my own professional journey, I've come to a significant realization – I've been with my current company since the day I graduated college, marking a continuous tenure of three years and six months. From my beginnings as a junior developer to my current role as a mid-level professional, this steadfast commitment has taken me on a transformative journey of growth and self-reflection. Over the years, I've found satisfaction in the projects I've contributed to, the features I've meticulously coded, the enhancements I've seamlessly added, and the tenacious bugs I've diligently resolved. This accumulation of experiences has not only solidified my expertise but has also allowed me to establish a strong foundation within my corner of the company.

However, this journey of professional growth has not been without it's share of challenges. The passage of time has granted me the opportunity to witness the consequences of my technical choices – some projects failed to scale as envisioned, and certain data storage methods gradually became obsolete. While initially daunting, these experiences served as invaluable lessons, illuminating the significance of considering technical debt and emphasizing the need for meticulous project planning. The ability to confront and rectify problems that emerged from my very own decisions proved to be a catalyst for profound technical growth, shaping me into a more skilled and aware software engineer.

Contrasting this concept with the trajectory of a developer who opts for job hopping every two years, it becomes apparent that such an approach might hinder one's exposure to the long-term consequences of their actions. Repeatedly accumulating mid-level experiences without fully immersing in the journey towards seniority may lead to a superficial understanding of the challenges and responsibilities that come with a position. The significance of dedicating substantial time to a role cannot be underestimated, as it provides the invaluable opportunity to evolve into a seasoned professional capable of navigating complexities and making strategic decisions in pursuit of excellence.

Undoubtedly, the allure of job hopping still resonates with many, particularly in an era where career mobility is celebrated. However, my commitment to staying at my company has offered me the unique vantage point of witnessing the long-term impact of my contributions, especially this early in my career which is priceless to understand a concept like this. The depth of understanding and profound insights gained through this journey of stability have been instrumental in shaping my growth as a software developer. It's important to note that there is a balancing act between maximizing technical growth and maximizing your total compensation so I'm not implying that it's always a good idea to stay at a company - rather right now, during this economic slump it might be a good idea to stay at you current company a bit longer and see if you can notice some of the long term choices you've made and the long term impact its had on the company. Thinking this way will provide you an interesting perspective on your future career endeavors.

Exploring API Protocols: From Conceptualization to Implementation

Kyle Knapp — Thu, 18 May 2023 02:44:42 +0000

For the month of April I decided to do something different from my regular tech articles and embark on a project that I could work on over the course of a few days. After some consideration, I chose to create my own web API, a project in which is long over due for myself. Working with third party APIs daily at work its about time I create my own. In this article, I'll explain my journey of creating my first web API, the challenges I faced, and the lessons I learned along the way.

Understanding REST APIs:

Before we dive into my project, let's take a moment to understand what web REST APIs are and why they are crucial in modern web development. REST APIs, or Representational State Transfer APIs, act as intermediaries between different software systems, enabling them to communicate and exchange data. They are the backbone of countless web applications, facilitating seamless data retrieval, updates, and interaction between clients and servers. They have become a cornerstone in modern web development, enabling seamless communication between systems. They serve a multitude of use cases, such as fetching data, creating new records, and updating existing resources. Developers leverage the power of HTTP methods like GET, POST, and PUT to perform these operations efficiently. The beauty of REST APIs lies in their simplicity and scalability, making them an ideal choice for tech enthusiasts of all skill levels.

The Project

To kick this project, I recognized the need to address several key aspects before diving into coding. First and foremost, I determined the purpose of the API, which I concluded would be a movie API. This API would provide a list of movie titles along with their respective genres and indicate whether or not I had watched them. Additionally, I developed a basic HTML/JavaScript website to showcase the API calls in an interactive and meaningful manner (I will present the website later in this article).

Once the purpose was clearly defined, my next step was to select a programming language and framework to assist in building the API. After careful consideration, I opted for C# as my language of choice, with ASP.NET serving as my supportive framework. With the development environment in place, I commenced coding the logic for the API endpoints, enabling access to the underlying movie database.

During the endpoint creation process, I crafted several types of endpoints to fulfill different functionalities. I created POST endpoints to allow users to add movies to the list, PUT endpoints to update movie data, DELETE endpoints to remove entries, and a variety of GET endpoints to facilitate finding movie entries using different criteria such as title, ID, and watched status. Here's a brief code snippet to provide an idea of the concepts:

[HttpPut("/movies/{id}")]
public async Task<IActionResult> UpdateMovie(int id, Movie movie)
{
    var existingMovie = await _dbContext.Movies.FindAsync(id);
    if (existingMovie == null)
        return NotFound();

    existingMovie.Title = movie.Title;
    existingMovie.Watched = movie.Watched;

    await _dbContext.SaveChangesAsync();

    return NoContent();
}

[HttpGet("/movies/complete")]
public async Task<ActionResult<List<Movie>>> GetCompleteMovies()
{
    var movies = await _dbContext.Movies.Where(m => m.Watched).ToListAsync();
    return movies;
}

[HttpGet("/movies")]
public async Task<ActionResult<List<Movie>>> GetMovies()
{
    var movies = await _dbContext.Movies.ToListAsync();
    return movies;
}`

After completing the code implementation, I proceeded to test the various endpoints using Postman. Through extensive testing and fine-tuning, I successfully ensured that all the API calls were functioning as intended.

Using Postman, I simulated different scenarios and input combinations to verify the correctness of the API responses. I tested the POST endpoints to add movies to the list, PUT endpoints to update movie data, DELETE endpoints to remove entries, and the GET endpoints to retrieve movie information based on different criteria.

Throughout the testing process, I made necessary adjustments and refinements to the code, addressing any issues or bugs that arose. By conducting thorough testing and refining the endpoints, I achieved a robust and reliable API that delivered the expected outcomes.

The Fun Part

With the endpoints tested and validated, it was now time to create a website to let the user manipulate the API to their liking. So I created a webpage using HTML/CSS and some JavaScript to make this website a bit lively. On page load the code calls the GET movies API to create a tile for each instance of a movie found, it then dynamically create a genre button for each different type of movie found that will later be used for sorting on the page. I thought about coding a search bar but I figured that I already put enough time in to this thing. The functionality is very basic in its current state but in the backend I made it super simple to build upon this page and add more endpoints and UI features to further increase its functionality! Below are some screenshots how it turned out!

When "Horror" is clicked only that genre of movies is shown

Closing Thoughts

Surprisingly, creating a web API turned out to be a smoother journey than I had anticipated. With the assistance of ASP.NET as my framework and a helpful YouTube tutorial, I was able to navigate through the process quite independently once I grasped the syntax. It became apparent that certain challenges, like scalability, backwards compatibility, and security, were certain concerns I luckily didn't have to worry about for this project.

However, this experience has opened my eyes to the potential complexities that can arise when dealing with larger-scale API implementations. I now appreciate the importance of thorough planning and consideration for these aspects in future endeavors. It's evident that the world of API development holds countless opportunities for growth and further exploration.

In conclusion, creating my own web API has been enlightening and empowering. It has not only expanded my knowledge and skills but also instilled in me a sense of excitement for future API-related projects. I encourage fellow developers to embark on their own API creation journeys, as the rewards in terms of personal growth and the ability to bring innovative ideas to life are truly worth it.

HERE IS THE CODE FOR ANYONE INTERESTED

The Pragmatic Programmer

Kyle Knapp — Sat, 18 Mar 2023 01:55:54 +0000

As a software developer, you know that the tech landscape is constantly evolving, and staying on top of the latest tools and techniques is crucial to your success. But in the midst of all the buzzwords and trends like AI, machine learning, and so much more, it can be easy to lose sight of the fundamental principles that support good programming. That's where The Pragmatic Programmer By Andrew Hunt and David Thomas comes in. This classic book offers a refreshing perspective on software development that emphasizes practicality, flexibility, and common sense. Whether you're a seasoned pro or just starting out, the insights and advice in this book will help you become a more effective, efficient, and pragmatic programmer! Over the last three weeks I have been reading and taking some notes on my thoughts about this book so let's dive in and explore some of my main takeaways.

The Broken Window Theory

When I first started reading the book I made it just FIVE pages until I already had to grab a pen and paper to write my thoughts about the concept of "The Broken Window Theory". The idea behind the Broken Window Theory is that if a broken window in a building is left unrepaired, it can create an environment that encourages further vandalism and deterioration. This is because the visible broken window sends a message that no one cares about the property, and therefore there will be no consequences for the further damage.

This concept originated in criminology but somehow these authors flawlessly applied it to software development, so let me explain their analogy. In software development, the Broken Window Theory applies to the concept of technical debt. Technical debt is the accumulation of unfinished work, such as code that is poorly written or not properly documented, that must be addressed at some point in the future. According to the Broken Window Theory, if technical debt is not addressed promptly, it can create a culture that tolerates subpar work and low standards. Just like the broken window, the presence of technical debt sends a message that it's acceptable to take shortcuts and not pay attention to details. This can lead to more technical debt and a cycle of declining quality.

So in conclusion, just like a broken window, it is important to address technical debt as soon as possible. This is because the longer technical debt is allowed to accumulate, the more difficult and expensive it becomes to fix. By addressing technical debt promptly, developers can maintain high standards and prevent the cycle of declining quality from taking hold.

The Evils of Duplication

Let's talk about the D.R.Y. Principle or also known as the "Don't Repeat Yourself Principle". A key concept discussed in Chapter Two that really stuck with me. The idea behind the D.R.Y. principle is that if you have multiple instances of the same code or logic throughout your codebase, it can lead to maintenance issues, bugs, and wasted time. Instead, you should strive to have a single reference point for each piece of code, so that you can make changes in one place and have those changes reflected throughout your codebase.

You may think this concept is very closely related to cohesion and coupling but it's actually a different aspect of software development that took me some time to wrap my head around. Let me try my best to put this idea into words: Coupling measures the interdependence between modules or components in a system, cohesion measures the degree to which the elements within a module or component are related, and the D.R.Y. principle emphasizes the importance of avoiding duplication in that module or component. While reducing duplication can help to improve cohesion and reduce coupling, it is only one factor in creating well-designed, maintainable software.

To follow the D.R.Y. principle, you can start by identifying areas of your code where there is duplication. This can include repeated code blocks, similar functions with slightly different implementations, or redundant data structures. Once you've identified these areas, you can start to refactor your code to remove the duplication. This principle, which sounds so obvious to some, I can't say I have consciously made the effort to condense my code through this method. Come Monday, I look forward to going through my code base and seeing where I can apply this D.R.Y. Principle!

The Craft of Coding - What Makes a Good Developer?

Sprinkled all throughout this book are tips and tricks on how to become the best version of yourself as a software developer. In the past, these authors have worked with many software developers all the way from junior to principal engineers and they found a similar trend among the ones that performed strongest regardless of their job level. Here are some of my main takeaways that I wrote down during my reading:

Continuously Learning: Great developers are always seeking to learn and improve their skills. They are curious, stay up-to-date with the latest technologies, and are always looking for ways to improve their craft. The author noted that many of these developers do this organically, they live and breath the thrill of finding new tech that others haven't worked with before.
Analytical Skills: Software development is all about solving problems, so great developers excel at breaking down complex problems into manageable pieces and coming up with creative solutions and then turning them into clean and concise code for their projects.
Soft Skills: Developers don't work in a vacuum, so being able to work well with others and communicate effectively is critical. Great developers are able to collaborate with other developers, project managers, and stakeholders to ensure that everyone is on the same page and that the project is moving forward smoothly.
Attention to Detail: Good developers pay close attention to details, whether it's writing clean, readable code, debugging effectively, or ensuring that their software works correctly in all situations. They are able to see the big picture of the project and adjust quickly based on priorities.
Passion: Lastly, one of the most important traits according to the author. Great developers have a genuine passion for software development. They enjoy the challenges and opportunities for creativity that it provides. He noted that the personality behind the passionate ones genuinely view software development as a calling or a vocation, rather than just a job, and are motivated by a deep sense of enthusiasm and interest in the work they do.

Be a Pragmatic Programmer

So what does it mean to be a pragmatic programmer? Be someone who is focused on delivering high-quality software that meet the needs of stakeholders and your end-users. It means being open to new ideas and approaches, but also having the experience and knowledge to make informed decisions and avoid common pitfalls. It means being disciplined and organized, but also flexible and adaptable in the face of changing requirements or unexpected challenges. The pragmatic programmer is someone who is dedicated to the craft of software development, and is always striving to improve their skills, knowledge, and effectiveness as a developer.

Ultimately, the core concept I got after reading this book was simply - Emphasizing the importance of being practical, adaptable, and continuously learning over everything else.

Technology Adoption Lifecycle: The Chasm

Kyle Knapp — Tue, 07 Feb 2023 05:36:57 +0000

The Technology Adoption Lifecycle is a widely recognized concept that sheds light on the journey of a new technology from its introduction to widespread adoption. From early adopters to laggards, the model describes the stages of acceptance and usage of technology by individuals and organizations. Understanding the lifecycle is crucial for businesses and organizations looking to adopt new technologies, as it provides valuable insights into the behavior of potential customers and the market as a whole. In this article, let's explore the stages of this lifecycle and then lets talk about a few projects currently trying to breach "The Chasm" that I find interesting.

First, Let's Discuss the Model

As probably noticed above, the adoption lifecycle has a bell curve. The divisions are approximately equivalent to where standard deviations would fall. Based on demographic and psychological characteristics, customers fall into one of the five adopter groups:

Innovators (2.5% of the total population)
Early Adopters (13.5% of the total population)
Early Majority (34% of the total population)
Late Majority (34% of the total population)
Laggards (16% of the total population)

Statistically speaking, I can say most of us land somewhere between Pragmatists and Conservatives for most tech that comes mainstream. We all have the occasional "I was early" (Visionaries) story in our lives. For example, maybe you joined a social media site like Facebook or Myspace during the very early stages or you had an electric car before Tesla became well known. All examples like these turn out to be fun stories to share because you were there before "The chasm" happened which I will explain later in the article.

Let's Dive deeper Into the Adopter Groups

Innovators - The Tech Enthusiasts

The Innovators are the first adopter group in the technology adoption lifecycle. They are individuals or organizations who are the first to adopt new technology. They are willing to take risks and embrace new and untested technology. They are often early adopters of new products and services, and are motivated by factors such as prestige, being on the cutting edge, or solving a specific problem. They are willing to pay a premium price for new technology and can tolerate early versions of products with bugs or shortcomings. They are also willing to provide feedback to the manufacturer, which can help to improve the technology for the benefit of later adopters.

Early Adopters - The Visionaries

Early Adopters are the second adopter group in the technology adoption lifecycle. Unlike the Innovators, they are not the first to adopt new technology and are more cautious in their adoption decisions. Early Adopters are opinion leaders and are respected within their social and professional circles. They are often early adopters of new products and services, and are motivated by factors such as the desire to be on the cutting edge, to improve their personal or professional life, or to solve a specific problem. They are often willing to pay a premium price for new technology, but are less tolerant of bugs and shortcomings compared to Innovators.

Early Majority - Pragmatists

The Early Majority is the third adopter group in the technology adoption lifecycle. Unlike the Innovators and Early Adopters, the Early Majority is more cautious in their adoption decisions and waits for the technology to mature and for others to validate it before adopting. The Early Majority are often influenced by the experiences and opinions of the Early Adopters, and they play a key role in the diffusion of new technology by providing validation and credibility to the technology. They also tend to be more cost-conscious than the Early Adopters, and they are likely to adopt a technology only after its price has come down and its functionality and reliability have improved. Their adoption helps to establish the technology as a mainstream solution and encourages the Late Majority to adopt. They help to build critical mass and drive the widespread adoption of new technology.

Late Majority - Conservatives

The Late Majority is the fourth adopter group in the technology adoption lifecycle. Unlike the Innovators, Early Adopters, and Early Majority, the Late Majority is typically more skeptical and cautious in their adoption decisions, and they tend to wait until a technology has become established and proven before they adopt. The Late Majority are often influenced by the experiences and opinions of the Early Majority, and they adopt new technology only after it has been widely adopted and its benefits and drawbacks are well understood. They are motivated by the desire to avoid being left behind but also to minimize risk and uncertainty. The Late Majority help to solidify the position of a technology in the market and provide the stability necessary for the technology to continue to grow and evolve.

Laggards - Skeptics

The Laggards are the final adopter group in the technology adoption lifecycle. Unlike the other adopter groups, the Laggards are typically the last to adopt new technology and are often skeptical of change. They are motivated by the desire to maintain the status quo and to avoid the risk and uncertainty associated with new technology. The Laggards tend to be resistant to change and may not see the need to adopt new technology. They are often influenced by their past experiences and may have a strong attachment to traditional ways of doing things. They are also more likely to be older and may not have the skills or resources necessary to adopt new technology. The Laggards play a limited role in the technology adoption lifecycle as they tend to adopt new technology only after it has become established and its benefits are widely understood. However, their adoption can provide a sense of validation and help to further establish the technology in the market.

Now that each of the adoption groups have been evaluated. Lets talk about the most fascinating part, the Chasm.

The "Chasm"

The "chasm" is a term used to describe a gap in the technology adoption lifecycle that occurs between the Early Adopters and the Early Majority. It mostly refers to the difficult challenge of getting new technology to cross over from the early adopting, tech-savvy market segment to the more mainstream, conservative market segment.

Studies have shown that only a small fraction of new technology products and services are able to successfully cross the chasm and achieve widespread adoption. This can be due to a number of factors such as insufficient market demand, inadequate marketing and distribution, poor product quality, and intense competition from established players.

In order to successfully cross the chasm, technology projects must be able to effectively address the needs and expectations of both the Early Adopters and the Early Majority. This often requires making changes to the technology, its marketing and positioning, and its sales and distribution channels to better appeal to the Early Majority.

Overall, while there is no set number of technology projects that successfully cross the chasm, it is believed that less than a fraction of a single percent are able to achieve widespread adoption.

Here Are Some Popular Examples Currently Testing The Chasm

Virtual and Augmented Reality: Virtual and Augmented Reality (VR/AR) technologies have been around for several years, but they are still in the process of trying to cross the chasm and achieve widespread adoption.
Blockchain Technology: Blockchain technology has the potential to disrupt many industries, but it is still in the early stages of adoption and is facing challenges in trying to cross the chasm and reach widespread adoption.
Artificial Intelligence (AI) and Machine Learning (ML): AI and ML are being adopted in a number of industries, including finance, healthcare, and retail, but their widespread adoption across many industries is still in the early stages.

My Personal Favorite Project that Will Test the Chasm in the Near Future

Let's talk about Protocol: Gemini, probably the most ambitious Web3/AR/MR/XR project to date. If you haven't heard of it yet, take a few minutes and go down the rabbit hole by clicking the link above. From their website here is the description of the project:

Protocol: Gemini is a utility-based app that utilizes current GPS data, allowing a dynamic and unique user curated 3-Dimensional experience.

With our unique rewards based contribution model, users of the app will not only be able to create something awesome, but get paid to do so using our minted ERC-20 utility token called:

GEMz = Geo-Encrypted-Mappoints - (on the Z axis)

We are re-defining the term "Pay-to-Play"; by paying YOU to help build the network/community. By creating a usable platform for people to get info that is USEFUL and EFFECTIVE as well as voted on by a true democratic model (our users tell us what’s real or cool) we will be providing a unique system for interacting. That’s not where the utility of the app is going to end though...

Basically, at a high level, you can tie NFTs to GPS coordinates. Others have given uses like PokémonGo or NFT treasure hunts or even possibly leaving a passkey to a door as an NFT instead of hiding a spare key.

I'll end the article with this, a screen cap from a recent Q/A session they hosted:

Code Optimization - Read Over Speed?

Kyle Knapp — Sat, 14 Jan 2023 03:09:27 +0000

Why Do We Optimize Code?

Probably an obvious question for most of us as software developers in the industry. I'm guessing you've all heard some of these phrases while on the job - "Why is it taking so long?", "Can you speed this up somehow?", "This website is loading slower than a snail on crutches." and so on...

There are several reasons why we optimize code:

Performance: Optimizing code can improve the performance of a program, making it run faster and more efficiently. This can be especially important for applications that need to handle large amounts of data or perform complex calculations.

Scalability: Optimizing code can make it more scalable, meaning that it can handle an increasing number of users or data without a significant decrease in performance.

Resource Utilization: Optimizing code can reduce the amount of memory and other resources that a program uses, which can be important for systems with limited resources or for mobile devices with limited battery life.

Maintainability: Optimizing code can make it easier to maintain, debug and update. This can reduce the development time and cost.

Security: Optimizing code can also help to make it more secure, by reducing the risk of vulnerabilities or exploits that could be used to compromise the system.

How Come Developers don't write optimized code the first time around?

A silly question, but I've heard it before. Let's explore the reasons why we can't always magically write perfect code on the job.

One of the main reasons is due to time constraints and pressure to deliver. When faced with tight deadlines and high expectations, developers may prioritize getting the code to work over making it readable, maintainable, and efficient. This can lead to code that is difficult to understand and maintain, and may have performance issues. Additionally, lack of proper planning and design also can lead to poor code.

Another reason why developers may not write good code all the time is lack of experience or knowledge in certain areas. For example, a developer who is new to a programming language or framework may not have the same level of expertise as someone who has been working with it for many years. This can lead to poor code quality, as the developer may not know the best practices or idioms of the language or framework.

Optimize for Readability First

When optimizing, developers often focus on execution speed and memory consumption, but fail to consider the importance of code readability. Us developers spend a majority of our time reading code, whether it's for debugging, checking commits done by others, or learning new libraries. Therefore, it is crucial for code to be easy to read, follow, and understand in order to reduce development time and cost.

A use case of this situation I recently encountered was when a Junior Developer was tasked to optimize some code written by a Senior Developer. The junior came to me and simply said the code was "too complicated" for his skill level. After doing a quick code review, he wasn't wrong, the code was very tricky. The code was dense, lacked comments and did very many tasks and database calls within just a few lines. This code did not take readability into account. If this senior developer took time to implement readability into their code first we never would've been in this situation. Hence the reason why I wrote this article.

Further Thoughts

Optimization for readability to make reading and debugging code easier, thus reducing development time and cost in the future.
Avoid unnecessary one-liners. Especially if you are doing it just cause you can. This is sacrificing readability.
Add some comment to your code, especially on those scary complex algorithms that nobody wants to touch with a ten-foot pole.

TL:DR - Developers often prioritize execution speed and memory consumption over readability, but that readability is crucial for reducing development time and cost.