DEV Community: Joerg Rech

Exploring Salary Trends with JavaScript: A Spring Break Project 🕵️‍♂️📊

Joerg Rech — Mon, 03 Mar 2025 11:14:24 +0000

Use your Spring Break to explore a fascinating project that combines coding skills with real-world data analysis. By working with a Job Postings API, you'll learn to fetch and analyze salary data for software developers using JavaScript.

This tutorial involves:

Setup: Get started with Node.js and npm. Obtain the API key from RapidAPI.
API Interaction: Use node-fetch to retrieve job postings with salary details. Filter and process the data to extract relevant insights.
Statistical Analysis: Calculate average, high, and low salaries. Consider factors influencing these figures, such as location and experience.
Portfolio Development: Document your process and results, and add this project to your portfolio to demonstrate your analytical skills.

As a tutorial it helps you to develop practical coding skills while gaining insights into the tech job market. It's a great way to enhance your portfolio and prepare for future career opportunities.

Link: Hope Springs Eternal - As Do Salaries? A Spring Break Project

The Developer’s Guide to Automating Remote Job Alerts

Joerg Rech — Tue, 25 Feb 2025 14:22:15 +0000

Introduction

Looking for a remote job as a developer can feel overwhelming. There are so many job boards, and checking them regularly takes time we don’t always have. What if we could automate this process?

In this tutorial, we’ll use a Job Posting API on RapidAPI and Bash commands to fetch remote job postings, format them into a clean table, and set up a weekly email alert. By the end, we’ll have a simple yet powerful tool to keep us informed about new opportunities without the hassle of manual searches.

Prerequisites

Let’s start by setting up our environment. We’ll use Ubuntu or a similar Linux distribution for this tutorial. If you’re on Windows, you can use WSL or a virtual machine to follow along.

Creating a RapidAPI Account and Subscribing to the API

First we need a RapidAPI account to access the job postings API - don’t worry, it’s free to sign up, and we’ll grab an API key once we’re logged in.

Create a RapidAPI Account
- Go to RapidAPI and click on the Sign Up button in the top-right corner.
- You can sign up using your email, Google account, or GitHub account. Follow the prompts to complete the registration process.
Subscribe to the Job Postings API
- Once logged in, you can visit the API’s page using this link: Daily International Job Postings API.
- On the API’s page, you can read up on the API features and go to the Pricing section.
- Select the Basic plan, which offers 25 free requests per month, with each request returning up to 10 job postings.
- Click the Subscribe button and confirm your selection.
Get Your API Key
- After subscribing, navigate to the Endpoints section on the API’s page.
- Look for the Headers section in any endpoint example. You’ll see two required headers:
  - x-rapidapi-key: This is your unique API key.
  - x-rapidapi-host: This should be set to daily-international-job-postings.p.rapidapi.com.
- Copy your x-rapidapi-key and keep it secure - we’ll use it in our script to authenticate API requests.

Now that we’ve set up our RapidAPI account and subscribed to the Basic plan, we’re ready to start building our script to fetch and process remote job postings. Let’s move on to the next step!

Installing Required Bash Commands

Next, we’ll install some command-line tools to make our lives easier. We’ll use curl to fetch data from the API, jq to process the JSON responses, and ssmtp or mailutils to send emails. If you don’t already have these installed, you'll need to run some quick command to get them set up. Additionally, we’ll configure ssmtp with our email credentials so we can send the job postings directly to our inbox.

Check if curl is Installed
curl is a command-line tool for making HTTP requests. It’s commonly pre-installed on most Linux distributions, but let’s verify by running curl --version. If you see version information, curl is installed. If not, install it with:
```
  sudo apt update
  sudo apt install curl
```
Check if jq is Installed
jq is a lightweight JSON processor that we’ll use to parse and format the API responses. Check if jq is installed by running jq --version. If it’s not installed, install it with:
```
  sudo apt update
  sudo apt install jq
```
Check if ssmtp or mailutils is Installed
We’ll use either ssmtp or mailutils to send emails from the command line. Check if ssmtp is installed by running ssmtp --version. If ssmtp not installed, install it with:
```
  sudo apt update
  sudo apt install ssmtp
```
and if you want to use mail check if mailutils is installed by running mail --version. If it’s not installed, install it with:
```
  sudo apt update
  sudo apt install mailutils
```
Configure Email Sending
If you’re using mailutils, no additional configuration is required for basic usage. If you’re using ssmtp, you’ll need to configure it with your email credentials. Open the configuration file in a text editor by running sudo nano /etc/ssmtp/ssmtp.conf and add the following lines, replacing placeholders with your email provider’s SMTP settings (e.g., Gmail):
```
  root=your_email@gmail.com
  mailhub=smtp.gmail.com:587
  AuthUser=your_email@gmail.com
  AuthPass=your_email_password
```
Save and exit the file (Ctrl+O, Enter, Ctrl+X).

Finally, we’ll need a basic understanding of Bash scripting. If you’ve ever written or run a shell script, you’re good to go. If not, don’t worry - we’ll keep the code simple and explain each step as we go. By the end, we'll have a script that fetches remote job postings, formats them, and sends them to our email account at the end of the month. Let’s get everything set up so we can start building!

Overview of the Job Posting API

The Job Posting API, as documented at https://api.techmap.io/, provides comprehensive access to job postings aggregated from a wide range of sources, including career pages, job boards, job aggregators, and employment offices. The API allows users to search and filter job postings based on various parameters such as job title, skills, company, location, industry, and more. It supports advanced querying capabilities, including Boolean searches, keyphrase searches, and geographic filtering, enabling precise and flexible job searches. The API returns results in JSON format, making it easy to integrate into applications for backfilling job boards, detailed analysis, company monitoring, or automated job agents.

One of the key features of the API is its ability to handle complex queries with multiple parameters. For example, users can search for remote jobs in specific time zones, jobs in particular industries, or jobs requiring specific skills. The API also supports pagination, allowing users to retrieve job postings in manageable chunks, with each request returning up to 10 job postings per page. Additionally, the API provides metadata endpoints in higher subscription plans that offer statistics and distinct values for specific fields, such as workPlace or industry, which can be useful for analyzing trends or building filters in job search applications.

The API is designed to be developer-friendly, with clear documentation and examples for each endpoint. It also includes structured data in JSON-LD format, which can be directly integrated into webpages to enhance visibility and indexing by search engines. Overall, the Job Posting API is a powerful tool for developers looking to build or enhance job search functionalities, offering a rich set of features and flexible querying options to meet various use cases.

For our goal, we can use the /api/v2/jobs/search endpoint with the following query parameters:

workPlace for filtering jobs posted based on the place of work (e.g., on-site, hybrid, or remote),
title for selecting job for a specific position (e.g., developer) or technology (e.e., Javascript),
language to constrain the search to job postings written in English, and
page to paginate through larger lists of jobs.

Fetching Job Postings from the API

To fetch remote job postings for Java developers using the Job Posting API, we’ll construct a query that filters jobs based on specific criteria. The API allows us to search for jobs using parameters like workPlace and title. For this tutorial, we’ll focus on remote jobs (workPlace=remote) that mention Java (title=Java).

Using curl with the URL parameters, we’ll construct a GET request that includes the API endpoint, query parameters directly in the URL, and the required headers for authentication. For example, to fetch remote Java developer jobs in February 2025, we’ll use the following curl command:

curl --request GET \
    --url 'https://daily-international-job-postings.p.rapidapi.com/api/v2/jobs/search?dateCreated=2025-02&workPlace=remote&title=Java&page=1' \
    --header 'x-rapidapi-host: daily-international-job-postings.p.rapidapi.com' \
    --header 'x-rapidapi-key: YOUR_API_KEY'

This command fetches the first page of remote Java developer jobs in JSON format. The results will also return the amount of all jobs (in the totalCount field) for this query that can be used for pagination (divide by 10). To retrieve more results, we can loop through additional pages by incrementing the page parameter (e.g., page=2, page=3, etc.). The response can then be processed further to extract and format the job postings.

Creating the full Script

Finally, we’ll create the script that connects to the Job Postings API, queries for remote job postings, and stores the results in a file. To stay within the free tier of the subscription plan, the script will loop through up to 25 pages, fetching a maximum of 250 job postings. These results are then combined into a single JSON file, which will later be reformatted and sent via email to keep you updated on new job opportunities.

The script leverages curl to make HTTP requests and jq to parse and manipulate the JSON responses. It iterates through multiple pages, retrieving up to 10 job postings per page and appending them to a combined JSON array. Once all pages are processed, the aggregated job postings are saved to a file named jobs.json. This ensures you have a structured, up-to-date list of remote job opportunities ready for further processing.

After fetching and formatting the job postings, the next step is to send this information to your email in a clean, readable table. Using jq, we’ll extract key fields such as the date posted, job title, company, and URL. These details are then formatted into an HTML table for improved readability and sent to the specified email address.

To get started, create a script named fetch_jobs.sh and insert the code provided below. Before running the script, ensure you set your API key in the RAPIDAPI_KEY variable and your email address in EMAIL_RECIPIENT. Keep in mind that the free tier allows only 25 requests per month, so adjust your query parameters carefully to avoid exceeding this limit.

#!/bin/bash

# Define general values
EMAIL_RECIPIENT='your_email@gmail.com'

# Define the API values
RAPIDAPI_KEY=My-RapidAPI-API-Key # Can be defined outside the script but needs special handling for the cronjob
API_URL="https://daily-international-job-postings.p.rapidapi.com/api/v2/jobs/search"
HEADERS=(
  -H "X-RapidAPI-Key: $RAPIDAPI_KEY"
  -H "X-RapidAPI-Host: daily-international-job-postings.p.rapidapi.com"
)
# REQUEST_LIMIT=25 # if calling it monthly (free requests)
REQUEST_LIMIT=1 # for testing purposes
CURRENT_YEAR_MONTH=$(date +"%Y-%m")

# Initialize an empty array to store all job postings
ALL_JOBS=()
MAX_PAGES=1
PAGE=1

while (( PAGE <= MAX_PAGES )); do
  printf "Fetching page $PAGE for $CURRENT_YEAR_MONTH ... "
  RAW_JSON=$(curl -s -G "$API_URL" \
    --data-urlencode "dateCreated=$CURRENT_YEAR_MONTH" \
    --data-urlencode "workPlace=remote" \
    --data-urlencode "title=Java" \
    --data-urlencode "language=en" \
    --data-urlencode "page=$PAGE" \
    "${HEADERS[@]}" | jq -c .)

  TOTAL_COUNT=$(printf "%s" "$RAW_JSON" | jq -r '.totalCount // 0')
  MAX_PAGES=$(( (TOTAL_COUNT + 9) / 10 )) # calculate max pages and round up
  ((MAX_PAGES > $REQUEST_LIMIT)) && MAX_PAGES=$REQUEST_LIMIT
  printf "found $TOTAL_COUNT jobs resulting in $MAX_PAGES max pages we load for free\n"

  JOBS=$(printf "%s" "$RAW_JSON" | jq -c '.result') || {
    echo "  Error processing JSON on page $PAGE.";
    break;
  }
  ALL_JOBS+=("$JOBS")
  ((PAGE++))
done

# Combine all job postings into a single JSON array
COMBINED_JOBS=$(printf "%s" "${ALL_JOBS[@]}" | jq -s -c 'add')

# Save the combined job postings to a file
printf "%s" "$COMBINED_JOBS" > jobs.json

echo "Job postings fetched and saved to jobs.json"

# Extract job data, create HTML table and send email
HTML_TABLE=$(jq -r 'to_entries
  | map("    <tr> <td>\(.value.dateCreated | split("T") | .[0])</td> <td>\(.value.title)</td> <td>\(.value.company)</td> <td>\(.value.jsonLD.url)</td> </tr>")
  | join("\n")
' jobs.json)

HTML_TABLE="<table border='1'>
    <tr> <th>Date Posted</th> <th>Title</th> <th>Company</th> <th>URL</th> </tr>
$HTML_TABLE
</table>"

echo -e "Subject: New Remote Job Postings\nContent-Type: text/html\n\n$HTML_TABLE" | sendmail "$EMAIL_RECIPIENT"

echo "Job postings send to $EMAIL_RECIPIENT"

Automating the Script with Cron

To fully automate our job-fetching script, we can leverage Cron - the built-in scheduler in Unix-based systems like Linux and macOS - to run the script on a regular basis without manual intervention. Cron enables us to schedule commands or scripts to execute at specified intervals, ensuring that our remote job postings are fetched automatically. By simply adding an entry to the crontab, we can have the script run monthly on the 25th day, so we always receive the latest remote job postings.

Before setting up the cron job, ensure that key parameters in the script are configured, such as REQUEST_LIMIT (e.g., 25 if run monthly or 1 if run every workday) and your RAPIDAPI_KEY. To schedule the script, open your crontab editor by running:

crontab -e

Then, append the following line to the file:

0 0 25 * * /path/to/your/fetch_jobs.sh >> /tmp/fetch_jobs.log 2>&1

This line instructs Cron to run the script at midnight on the 25th of every month, with all output (including errors) directed to fetch_jobs.log for easy debugging. Finally, ensure that the script has execute permissions by running chmod +x fetch_jobs.sh. With this setup in place, our personalized job alert system will automatically fetch and deliver fresh remote job postings each month, allowing you to potentially quit your old and apply to the new job.

Conclusion

By completing this tutorial, we’ve not only automated a job search but also gained valuable skills in Bash scripting, API integration, and task automation. This project is a testament to how a little bit of coding can make a big difference in your daily life.

To take it a step further, try customizing the script to suit your specific needs - whether that’s adding more precise filters, tweaking the email format, or scheduling it to run more frequently. If you encounter any issues, refer to the API documentation for more advanced features. Don’t forget to share your success with others or contribute to the project by improving the script. Happy automating!

Weekend Project: Create a Personalized Job Posting Agent

Joerg Rech — Sat, 14 Dec 2024 09:46:57 +0000

Introduction

Job hunting can feel like a full-time job in itself. Tailoring searches and scanning multiple job boards every day can be time-consuming and overwhelming. A personalized job posting agent automates this process, saving you valuable time and effort—it’s like having your own digital assistant scouring the web for jobs that fit you.

This tutorial helps you build a personalized job posting agent that queries job postings you're interested in from an API, formats the data, and emails it to you automatically. Whether you’re actively looking for a new role or just want to keep an eye on the market, this project is a practical and fun way to make technology work for you.

Prerequisites

Before diving into the implementation, it’s crucial to prepare your system and set up the tools and services required for this tutorial. This setup will enable you to run the Python script, configure the cron job, and send email notifications.

First, you need to create an account on RapidAPI, a platform that provides access to numerous APIs, including the job postings API we’ll use. Once your account is set up, go to the Daily International Job Postings API, subscribe to the free Basic plan, and obtain your API key. The Basic plan gives you access to 25 free requests with 10 job postings each per month.

Second, you’ll need access to an SMTP-based email provider, like Gmail. SMTP (Simple Mail Transfer Protocol) is the standard protocol for sending emails, and it allows your Python script to communicate with the email server to deliver messages. You'll need the SMTP server address and your credentials to send emails. Alternatively, you can start with printing the job postings into a file and create an calendar reminder to look at it.

Next, install the necessary Python tools and libraries for your operating system. You can use pip to install essential packages, including requests for making API calls, smtplib for sending emails, and any other dependencies specified in the code. Once all prerequisites are met, verify that your Python environment is functional by running python3 --version.

NOTE: Using programming languages other than Python isn't too hard - RapidAPI shows the core API access for other languages in their Code Snippets selector tab.

Finally, on Linux or macOS, ensure that cron is installed and operational; Windows users can achieve similar functionality using Task Scheduler. These tools will enable the automation of your Python script at specified intervals.

Understanding the Job Postings API

Techmap's Daily International Job Postings API provides comprehensive access to global job data from the last 3 months. It returns structured job postings and supports queries with various filtering parameters to narrow search results such as date of posting, location, occupation, and required skills.

You can find the documentation for constructing API requests on RapidAPI's Playground for the Paginated Search or the API's OpenAPI documentation page. We will use the /api/v2/jobs/search endpoint with the following query parameters:

dateCreated for filtering jobs posted on a specific date (day or month),
countryCode for selecting the country (using the ISO3166 codes),
city for specifying a particular city,
occupation for job type, (using many synonyms)
skills for required expertise, and
page to paginate through larger lists of jobs.

These parameters help us in building a personalized job posting agent for a specific type of occupation in a specific area. The API returns data in Schema.org Microdata format using JSON, which is ideal for parsing, further processing, or direct use in webpages to improve SEO.

Building a Query with Parameters

To query job postings in Python using the API, you need to structure your request with the appropriate parameters. Consider the following example parameter configuration:

parameters = {
  "dateCreated": "2024-12",    # Filter jobs posted for December 2024
  "countryCode": "de",         # Country code for Germany
  "city":        "Berlin",     # Specific city in Germany
  "occupation":  "Programmer", # Job type
  "skills":      "Java"        # Required skill
}

These parameters allow you to filter job postings precisely for a programmer role in Berlin, Germany, looking for expertise in Java. If more than 10 results are found we'll be adding the pagination (page) parameter to fetch additional results.

These parameters will be used to construct the following query:

/api/v2/jobs/search?dateCreated=2024-12&countryCode=de&city=Berlin&occupation=Programmer&skills=Java

Alternatively, you can use other parameters described in the API documentation such as title, workPlace (e.g. remote or hybrid), company, geoPoints, timezones, etc.

// Remote jobs in core US Timezones (UTC-8 to UTC-5):
/api/v2/jobs/search?dateCreated=2023-11-01&page=1&workPlace=remote&timezoneMin=-8&timezoneMax=-5

// Jobs in the Healthcare industry of the United Kingdom:
/api/v2/jobs/search?dateCreated=2023-11-01&page=1&countryCode=uk&industry=healthcare

// Jobs for Java Developers in San Francisco:
/api/v2/jobs/search?dateCreated=2023-11-01&page=1&skills=Java&occupation=java,programmer&geoPointLat=37.757&geoPointLng=-122.449&geoDistance=100mi

// Jobs for JavaScript Developers in New York, USA:
/api/v2/jobs/search?dateCreated=2023-11-01&page=1&skills=JavaScript&countryCode=us&city=New%20York

// Jobs in English language located in Germany:
/api/v2/jobs/search?dateCreated=2023-11-01&page=1&countryCode=de&language=en

NOTE: Please remember that the free Basic Account only gives 25 free requests per month - it's probably best to start with our example parameters and then tailor it to you needs. Nevertheless, the API is cheap and 100 requests over the free ones only cost 5$.

Using curl to Call the API

The curl command is an easy way to interact with the API via the terminal and test different queries and parameters. Here’s how you can query the API using curl with the above parameters:

curl --request GET \
  --url 'https://daily-international-job-postings.p.rapidapi.com/api/v2/jobs/search?dateCreated=2024-12&countryCode=de&city=Berlin&occupation=Programmer&skills=Java&page=1' \
  --header 'x-rapidapi-host: daily-international-job-postings.p.rapidapi.com' \
  --header 'x-rapidapi-key: <YOUR_API_KEY>'

After replacing <YOUR_API_KEY> with your unique API key obtained from RapidAPI, this command sends a GET request to the API endpoint, including the query parameters and authentication headers. The response, in JSON format, will contain the job postings matching your criteria including a description of the query that the server understood.

Calling the API with Python

Using Python to call the API offers greater flexibility for processing and integrating the data into your application. Here’s an example code snippet that achieves this:

import requests

# API endpoint and headers
url = "https://daily-international-job-postings.p.rapidapi.com/api/v2/jobs/search"
headers = {
  "x-rapidapi-host": "daily-international-job-postings.p.rapidapi.com",
  "x-rapidapi-key": "<YOUR_API_KEY>"
}

# Query parameters
parameters = {
  "dateCreated": "2024-12",
  "countryCode": "de",
  "city":        "Berlin",
  "occupation":  "Programmer",
  "skills":      "Java"
}

# Make the GET request
response = requests.get(url, headers=headers, params=parameters)

# Check for successful response
if response.status_code == 200:
  job_data = response.json() # Parse JSON response
  print(f"Found {job_data.get('totalCount', 0)} jobs.")
else:
  print(f"Error: {response.status_code} - {response.text}")

The script constructs the request, sends it to the API, and processes the JSON response to extract job postings. By using Python, you can further manipulate and analyze the data, save it to a file, store it in a database, or send notifications.

The complete picture

After we covered the basics, we will walk through the full implementation of the personalized job posting agent python script. This script fetches job postings from an API based on specified parameters, processes the data to ensure relevance, and sends a customized email notification to the user. The main parts of the script are:

Importing Required Libraries: The script begins by importing the necessary Python libraries for handling API requests, email communication, file manipulation, and data processing. These libraries provide the core functionality required to fetch, process, and email job postings.
Setting Up Variables and Configuration: To personalize the job search, the script defines variables for the API query parameters, email configuration, and file paths. These variables can be easily adjusted to suit the user’s requirements.
Fetching Job Postings with Pagination: The script queries the API in a loop to handle pagination, ensuring all relevant job postings are fetched. Results are written to a temporary file for further processing.
Deduplicate and sort Database File: After fetching the job postings, the script processes the data to remove duplicates and sorts the entries by the dateCreated field in descending order. The cleaned data is saved to a final JSON file for record-keeping.
Sending the Email: Finally, the script sends an email containing the job postings to the specified recipient. The email content is read from the prepared file, and the SMTP library is used to send the message.

To sum it up, the main script for the personalized job posting agent in detail looks as follows, combining all the steps above. The script is modular and can be customized easily by changing the query parameters or email configurations.

NOTE: to run this script you need to edit the configuration and replace the placeholders for EMAIL_TO, SMTP_* info, API_KEY, city, occupation, and skills.

File: fetch_jobs_and_email.sh

################################
# Importing Required Libraries #
################################

import os
import json
import requests
import smtplib
from email.mime.text import MIMEText
from datetime import datetime
from time import sleep
from operator import itemgetter

##########################################
# Setting Up Variables and Configuration #
##########################################

# Current date in "yyyy-MM" format - "yyyy-MM-dd" is possible
DATE_CREATED = datetime.now().strftime("%Y-%m")

# Email Configuration
EMAIL_TO = "recipient@example.com"
SMTP_SERVER = "smtp.gmail.com"
SMTP_FROM = "<SMTP_FROM_ACCOUNT_EMAIL>"
SMTP_PASS = "<SMTP_PASSWORD>"
EMAIL_SUBJECT = "My Daily Job Postings"
EMAIL_FILE = f"techmap_jobs_email.txt"

# Prepare the email file
with open(EMAIL_FILE, "w") as email_file:
  email_file.write(f"Job Postings in {DATE_CREATED} for you:\n")
  email_file.write("======================================\n\n")

JOB_FILE_TEMP = f"techmap_jobs_{DATE_CREATED}_all.json"
JOB_FILE = f"techmap_jobs_{DATE_CREATED}.json"

# API details
MAX_REQUESTS_PER_MONTH = 25
MAX_REQUESTS_PER_CALL = 2 # use 2 for tests and 5 for weekly execution
API_ENDPOINT = "https://daily-international-job-postings.p.rapidapi.com/api/v2/jobs/search"
API_HOST = "daily-international-job-postings.p.rapidapi.com"
API_KEY = "<YOUR_API_KEY>"

# Query parameters
parameters = {
  "dateCreated": DATE_CREATED,
  "countryCode": "de",
  "city": "Berlin",
  "occupation": "Programmer",
  "skills": "Java"
}

#########################################
# Fetching Job Postings with Pagination #
#########################################

print(f"Fetching job postings for date: {DATE_CREATED}")

PAGE = 1
TOTAL_COUNT = 0
i = 0
while i++ < MAX_REQUESTS_PER_CALL:
  # Add pagination to parameters
  parameters["page"] = PAGE
  print(f"Querying: {API_ENDPOINT} with parameters {parameters}")

  try:
    response = requests.get(API_ENDPOINT, headers={
      "x-rapidapi-host": API_HOST,
      "x-rapidapi-key": API_KEY
    }, params = parameters)
    response.raise_for_status()
  except requests.RequestException as e:
    print(f"ERROR: Failed to fetch jobs (Maybe the API_KEY): {e}")
    break

  data = response.json()
  TOTAL_COUNT = data.get("totalCount", 0)
  page_size = data.get("pageSize", 0)
  results = data.get("result", [])
  print(f" Found {len(results)} jobs of {TOTAL_COUNT} on page: {PAGE}")

  if not results:
    print("No more jobs found. Stopping.")
    break

  # Save jobs to temporary file
  with open(JOB_FILE_TEMP, "a") as temp_file:
    for job in results:
      temp_file.write(json.dumps(job) + "\n")

  # Write email content to Database File
  with open(EMAIL_FILE, "a") as email_file:
    for job in sorted(results, key=lambda x: x.get("dateCreated", ""), reverse = True):
      email_file.write(
        f"Title: {job.get('title', 'N/A')}\n"
        f"Company: {job.get('company', 'N/A')}\n"
        f"City: {job.get('city', 'N/A')}\n"
        f"Posted On: {job.get('dateCreated', 'N/A')}\n"
        f"URL: {job.get('jsonLD', {}).get('url', 'N/A')}\n\n"
      )

  # Stop if there are no more results
  if len(results) < page_size:
    break

  PAGE += 1
  if PAGE > MAX_REQUESTS_PER_CALL:
    print("Max requests reached. Exiting.")
    break
  # Sleep due to throttled access to API
  sleep(1)

######################################
# Deduplicate and sort Database File #
######################################
try:
  with open(JOB_FILE_TEMP, "r") as temp_file:
    job_list = [json.loads(line) for line in temp_file]
  # Deduplicate
  deduplicated_jobs = {json.dumps(job, sort_keys=True): job for job in job_list}.values()
  # Sort by dateCreated in reverse order
  sorted_jobs = sorted(deduplicated_jobs, key=itemgetter("dateCreated"), reverse = True)
  # Save the sorted, deduplicated results to the final job file
  with open(JOB_FILE, "w") as final_file:
    for job in sorted_jobs:
      final_file.write(json.dumps(job) + "\n")
  print(f"Successfully deduplicated and sorted: {JOB_FILE}")
except (FileNotFoundError, json.JSONDecodeError) as e:
  print(f"ERROR: Error processing job files: {e}")

#####################
# Sending the Email #
#####################
print(f"Sending email to {EMAIL_TO}")
try:
  with open(EMAIL_FILE, "r") as email_content:
    message = MIMEText(email_content.read())
    message["Subject"] = EMAIL_SUBJECT
    message["From"] = SMTP_FROM
    message["To"] = EMAIL_TO
  with smtplib.SMTP(SMTP_SERVER, 587) as server:
    server.starttls()
    server.login(SMTP_FROM, SMTP_PASS) # Update with your credentials
    server.send_message(message)
  print("Email sent successfully.")
except Exception as e:
  print(f"ERROR: Failed to send email (maybe wrong SMTP info): {e}")

print("Done fetching job postings.")

Setting Up a Weekly Schedule

To automate the Python script to run on a weekly schedule, you’ll need to configure a task scheduler specific to your operating system. Here’s how to do it on MacOS, Windows, and Linux.

Setting up the Cron Jobs on MacOS or Linux

On MacOS and Linux, you can use cron, a built-in task scheduler, to automate your script. Follow these steps:

Open the Terminal: On MacOS press Command + Space to open the Spotlight search bar, type Terminal, and hit Enter. On Linux start a shell.
Edit the Crontab: Type crontab -e to open the cron configuration file. If prompted, choose a text editor such as nano or vim.
Add the Cron Job: To schedule the Python script to run weekly, add the line below. This runs the script every Saturday at 6:00 AM. Replace /usr/local/bin/python3 with the path to your Python interpreter and /path/to/your_script.py with the full path to your Python script.

0 6 * * 6 /usr/local/bin/python3 /path/to/your_script.py

Save and Exit: Save the file (Ctrl + O in nano; Esc and ':x!' in vim) and exit (Ctrl + X in nano).
Finally, you can check if the cron job is active with crontab -l.

Setting up the Cron Jobs on Windows

Windows uses the Task Scheduler for automation. Here’s how to set up a weekly task:

Open Task Scheduler: Search for Task Scheduler in the Start menu and open it.
Create a New Task: Click on Create Basic Task in the right-hand menu, name your task (e.g., “Weekly Job Agent Python Script”) and click Next.
Set the Schedule: Choose Weekly and specify the day of the week and time you want the script to run.
Configure the Action: Choose Start a Program., and enter the path to your Python executable (e.g., C:\Python39\python.exe) in the Program/Script field. In the Add Arguments field, enter the full path to your script (e.g., C:\path\to\your_script.py).
Finish the Setup: Confirm and save the task. Your script will now run automatically as scheduled.

Conclusion

In this tutorial, we developed a fully functional and personalized job posting agent in Python to automate the retrieval of job postings using an API, customized the results for specific criteria, and sent the data as a formatted email notification. The script demonstrates how to query different API parameters such as location, occupation, and skills. With just a few adjustments, you can fetch data tailored to a wide range of use cases, from remote jobs in specific time zones to opportunities in specific industries or geographic regions.

Beyond sending notifications, the collected job data opens doors to more advanced applications. For instance, storing the retrieved job postings in a database enables the creation of a niche job board. You could enhance this board by integrating search and filtering features, allowing users to find tailored job opportunities. Additionally, analyzing this data for trends—such as demand for specific skills, regional salary variations, or emerging industries—could provide valuable insights for businesses and job seekers alike.

I hope you found value in this tutorial and are inspired to try it out as a personal project over a weekend or holidays. It’s an excellent opportunity to enhance your Python skills, experiment with API integration, and explore the world of automation.

How you can extract company signals from job postings

Joerg Rech — Thu, 25 Jul 2024 13:59:37 +0000

Identifying New Office Locations in Job Postings

As software developers, we know the importance of staying ahead in the tech industry, whether it's through learning new programming languages or understanding market trends. By analyzing job postings from companies, we can uncover new office locations, reveal growing tech hubs, or even identifying potential business partners for freelancing.

Imagine knowing that a major tech company is setting up a new office before everyone else does—that's the kind of edge we’re talking about. Let's dive into how we can do this using practical steps and tools.

The Basics: Job Postings and Company Signals

Job postings are typically targeted at potential new employees and used to create awareness that a company exists, is hiring for a specific role, why they are hiring, what skills an applicant should have, what the company works on, and provide insights into what companies does, plan, want, use, build, etc.

But job posting also contain many signals what a company is up to and are indicators that provide insights into a company's activities, strategies, and future plans. They help to identify sales opportunities, understand market trends, and develop targeted sales strategies. These signals can be derived from various sources, including job postings, press releases, financial reports, social media activity, and more.

In the remaining of this post we will use the free Luxembourg job posting data feed from Techmap on AWS Data Exchange (ADX) which gives access to all our historical data since January 2020. More specifically, we use the data from June 2024 with 14.7k job postings where the main sources are Linkedin, CareerJet, Indeed, Eures, and Smartrecruiters. For Luxembourg the compressed data files, typically, have a size between 100KB and 1.5MB per day.

Identifying New Office Locations in Job Postings

As Luxembourg has multiple official languages, we can identify location signals in English, French, and German. Furthermore, the regular expressions can identify location signals by using synonyms used by the hiring companies, such as “bureau” or “premise” instead of “office”.

(?i)\b(
  (new (\w+ ){0,2}(office|premise|location|facilit(y|ie)|site)s?)
  |(nouveau (\w+ ){0,2}(bureau|locaux|facilité|site)s?)
  |(neue(n|m|s)? (\w+ ){0,2}(Büro|Office|Räum(en|lichkeiten)|Standorte?|Einrichtung(en)?))
  |(we (are )?(expand(ing)?) to)|(nous (\w+ )?(étendons|développons?) (à|en))
  |(wir (\w+ )?(expandieren) nach)
)\b

Extracting Location Signals from Job Postings

In order to use the job postings you can follow AWS’s tutorial to subscribe to AWS Data Exchange for Amazon S3 but use our free Luxembourg data feed on ADX instead of their test product.

Then you can download the files from AWS ADX and decompress all files from June 2024 (results in 130 MB)

aws s3 sync \
    s3://<YOUR_BUCKET_ALIAS>/lu/ . \
    --request-payer requester \
    --exclude "*" \
    --include "techmap_jobs_lu_2024-06-*.jsonl.gz"

gzip -d *.gz

Now that we have the job postings in textual files with JSON Lines format, we can program the identification of location signals:

#!/bin/bash

# Define the regex as an environment variable 
export REGEX='\b((new (\w+ ){0,2}(office|premise|location|facilit(y|ie)|site)s?)|(nouveau (\w+ ){0,2}(bureau|locaux|facilité|site)s?)|(neue(n|m|s)? (\w+ ){0,2}(Büro|Office|Räum(en|lichkeiten)|Standorte?|Einrichtung(en)?))|(we (are )?(expand(ing)?) to)|(nous (\w+ )?(étendons|développons?) (à|en))|(wir (\w+ )?(expandieren) nach))\b' 

# Define and clear output file for the results 
export OUTPUT_FILE="location_signals.txt"
printf '' > "$OUTPUT_FILE"

# Loop over all files matching the pattern
for file in techmap_jobs_lu_2024-06-*.jsonl; do
    # Check if the file exists to avoid errors if no files match the pattern
    if [[ -e "$file" ]]; then
        # Decompress the file, filter JSON lines with jq, and extract fields
        cat "$file" | jq -r --arg regex ".{0,20}$REGEX.{0,20}" '
            select(
                . | to_entries[] | select(.value | type == "string" and test($regex))
            ) | {
                job_name: .name,
                job_url: .url,
                company_name: .company.name,
                location: (.location.orgAddress.addressLine // .location.orgAddress.city),
                matched_text: [
                    . | to_entries[] | select(.value | type == "string" and test($regex)) | .value | match($regex).string
                ] | unique | map("..." + . + "...") | join(", ")
            }
        ' >> "$OUTPUT_FILE"
    else
        echo "No files matching the pattern found."
    fi
done

To extract unique companies from job postings, regardless of the number of postings they placed, use the following code snippet:

cat location_signals.txt | grep company_name | sort | uniq

And get a result that looks like this:

  "company_name": "Allianz Global Investors",
  "company_name": "Amazon EU Sarl - A84",
  "company_name": "Amazon EU Sarl",
  "company_name": "Amazon",
  "company_name": "ArcelorMittal",
  "company_name": "Astel Medica",
  "company_name": "Deloitte",
  "company_name": "Euro Exim Bank",
  "company_name": "Koch Global Services",
  "company_name": "Koch Industries",
  "company_name": "Luxscan Weinig Group",
  "company_name": "MD Skin Solutions",
  "company_name": "ROTAREX",
  "company_name": "Thales",
  "company_name": "e-Consulting RH, Sourcing & Recrutement de Profils pénuriques",
  "company_name": "myGwork - LGBTQ+ Business Community",

In summary, in under a minute, we identified 144 job postings in Luxembourg mentioning new offices in June 2024. After refining the data, we pinpointed 16 unique companies with location signals, showcasing the effectiveness of our extraction method.

Conclusion

In this article, we showed how to identify 144 job postings mentioning a location change for 16 unique companies in June 2024 in Luxembourg. A similar analysis in the USA revealed 1241 signals from 918k job postings (i.e., ~1 signal per 1k job postings).

For more details, check out our full article on how to extract company signals from job postings.

If you are interested in further exploring the potential of job postings or applying machine learning / AI techniques, we encourage you to experiment with our data feed and source code provided in this article. Happy analyzing!

Harness the Power of Job Ads to Boost Your Ad Revenue

Joerg Rech — Tue, 12 Sep 2023 13:08:20 +0000

Are you looking for a way to increase your ad revenue and gain valuable insights into the job market? Look no further than job ads datasets! With access to comprehensive data on job postings, you can uncover emerging trends, identify new markets, and stay informed about your competitors. In this article, we will explore how harnessing the power of job ads can help you boost your ad revenue and drive your business growth.

Why Job Ads Datasets Matter

Job postings provide a wealth of information that can be leveraged to enhance your business intelligence capabilities. By analyzing job ads datasets, you can gain insights into hiring trends within specific industries or regions. This information is invaluable for market research, competitor analysis, and identifying potential sales leads.

Uncover Emerging Trends

One of the key benefits of analyzing job ads datasets is the ability to identify emerging trends in job roles, technologies, and markets. By monitoring job postings in a particular industry or region, you can spot new and upcoming job titles or skills that are in high demand. This information can help you align your products or services with market needs and stay ahead of the competition.

For example, let's say you run an online learning platform that offers courses on data science. By analyzing job ads datasets, you may discover a surge in demand for professionals with expertise in machine learning or natural language processing. Armed with this knowledge, you can develop new courses or update existing ones to cater to this growing market.

Stay Informed About Competitors

Job ads datasets can also provide valuable insights into your competitors' hiring strategies and growth trajectory. By monitoring the job postings of competing companies, you can gain a better understanding of their organizational structure, focus on specific departments, or expansion into new locations.

For instance, if you are a headhunter looking to place candidates in high-growth companies, analyzing job ads datasets can help you identify companies that are actively hiring. This information allows you to target your efforts towards organizations that are in need of employees and increase your chances of successful placements.

Optimize Your Hiring Strategy

Analyzing job ads datasets can also provide valuable insights into labor market trends and help optimize your own hiring strategy. By understanding changes in candidate preferences, skill requirements, or benefits offered by employers, you can tailor your recruitment efforts to attract top talent.

Let's say you are a recruiter for a tech company specializing in artificial intelligence. By analyzing job ads datasets, you may notice a shift in the desired skills for AI roles, such as an increased emphasis on deep learning or experience with specific programming languages. Armed with this knowledge, you can adjust your job descriptions and screening criteria to attract candidates who possess these sought-after skills.

How Job Ads Datasets Drive Ad Revenue

Now that we understand the value of job ads datasets, let's explore how they can specifically boost your ad revenue. By leveraging job postings, you can enhance your sales intelligence capabilities and uncover new leads for companies that may be interested in your solutions.

Identify Companies in Need of Employees

As a headhunter or recruitment agency, job ads datasets can be a goldmine for identifying companies in need of employees. By analyzing job postings from various industries and regions, you can compile a list of organizations actively hiring for specific roles.

With this information at hand, you can reach out to these companies and offer your services as a headhunter. This targeted approach increases your chances of success and allows you to place your candidates more efficiently.

Promote Your Courses or Services

If you are an educator offering courses or training programs, job ads datasets can help you promote your offerings to individuals seeking to acquire new skills or advance their careers. By analyzing the skills and qualifications required by employers in job postings, you can identify gaps in the market where your courses or services can provide value.

For example, if there is a high demand for professionals with expertise in cloud computing, you can develop and market courses that cater to this specific skill set. By aligning your offerings with market needs, you increase the likelihood of attracting students who are actively seeking to acquire those skills.

Offer Your Tools or Software

If you are a software manufacturer or provider of tools and solutions, job ads datasets can help you identify potential customers who may benefit from your products. By analyzing job postings, you can gain insights into the tools and technologies used by companies in specific industries.

For instance, if you manufacture a data analytics platform, analyzing job ads datasets may reveal a growing demand for professionals with experience in big data processing. Armed with this knowledge, you can target your marketing efforts towards organizations that are actively seeking to hire individuals with these skills.

The Power of Job Ads Datasets

In conclusion, harnessing the power of job ads datasets can significantly boost your ad revenue and provide valuable insights into the job market. Whether you are conducting market research, optimizing your hiring strategy, or promoting your products or services, job ads datasets offer unparalleled opportunities for growth and success.

Operating your own scraping infrastructure to collect job posting data can be time-consuming and costly. Instead, consider leveraging pre-existing job posting datafeeds like Techmap's. With flexible pricing options and support for various data formats, Techmap's job posting datafeeds provide easy access to comprehensive datasets that can fuel your business intelligence activities.

Don't miss out on the opportunity to enhance your data science capabilities, uncover emerging trends, and gain valuable insights into hiring practices. Contact Techmap today to learn more about their job posting datasets and how they can help drive your growth and success.

How Job Ads Data Uncovers Market Trends

Joerg Rech — Mon, 03 Jul 2023 15:33:06 +0000

Discover how analyzing job ads data can provide valuable insights into market trends, helping businesses in commercial investigation and trend identification.

More at: https://jobdatafeeds.com/blog/post/how-job-ads-data-uncovers-market-trends

Master the Art of Competitive Intelligence with Job Posting Data

Joerg Rech — Sun, 02 Jul 2023 10:58:43 +0000

Unlock valuable market insights and stay ahead of the competition by leveraging job posting data. Discover emerging trends, analyze hiring practices, and enhance your competitive intelligence capabilities.

More at: https://jobdatafeeds.com/blog/post/master-the-art-of-competitive-intelligence-with-job-posting-data

Improving Candidate Experience with Job Posting Data

Joerg Rech — Sat, 01 Jul 2023 09:09:54 +0000

Enhance candidate experience and optimize your recruitment strategy by leveraging job posting data. Gain insights into hiring trends and industry standards to make informed decisions. Learn how to use HR data and web scraping job ads effectively.