DEV Community: Leon Wei

A Django Developer's Guide to Leveraging Google's Indexing API

Leon Wei — Mon, 25 Mar 2024 17:28:39 +0000

Originally published at AISaaSTemplate

In the dynamic landscape of web development, keeping your site's content fresh and swiftly indexed by Google is crucial for maintaining visibility and relevance. For Django developers, integrating Google's Indexing API into your projects can significantly streamline this process, ensuring that your content is quickly discoverable through Google Search. This guide provides a comprehensive walkthrough on setting up and using Google's Indexing API within your Django applications, empowering you to programmatically request Google to index your website's content.

Step 1: Project Setup in Google API Console

Before diving into code, the initial step involves setting up your environment to communicate with Google's Indexing API. This setup is a one-time process that grants your application the necessary permissions to interact with Google services.

Creating a Google API Console Project

Navigate to the Google API Console: Start by accessing the Google API Console. Here, you're greeted with an intuitive interface to manage Google APIs across your projects.
Create a New Project: Click on the project drop-down menu at the top of the page and select "New Project". Provide a meaningful name that represents your client or the specific application you're working on.
Enable the Indexing API: Once your project is created, use the search bar to find the "Indexing API". Select it and click "Enable" to activate the API for your project.

Setting Up a Service Account

A service account acts as a non-human user that your application can use to authenticate with Google and make authorized API calls.

Access Service Accounts: In your Google API Console project, navigate to the "Service accounts" section.
Create a New Service Account: Click on "Create Service Account" and fill in the required details. The service account ID can be the default one or a custom ID that you specify.
Generate a JSON Key: After creating your service account, you'll need to generate a private key. Choose JSON as the format, which will download the key to your machine. Keep this key secure as it will be used to authenticate your API requests.

Step 2: Verify Site Ownership with Google Search Console

To use the Indexing API, Google requires verification of site ownership. This step ensures that only authorized users can request indexing for a given website.

Verify Your Site: If you haven't already, add your site to Google Search Console and complete the verification process. This can be done through various methods, such as HTML file upload, DNS record, or through your domain registrar.
Add Your Service Account as a Site Owner: In the Search Console, navigate to the property (site) you've verified. Under the "Settings" menu, find the "Users and permissions" section and add your service account email as an owner. This email is found in the JSON key you downloaded earlier.

Step 3: Integrating with Your Django Project

With the setup complete, it's time to integrate the Indexing API within your Django project. This involves programmatically generating OAuth tokens and making API requests to notify Google of new or updated content.

Generating OAuth Tokens

Google provides libraries in various languages to handle authentication. For Python and Django, the google-auth library can be used to generate OAuth tokens using your service account's JSON key.

Install google-auth: Use pip to install the google-auth library if you haven't already.
Generate Tokens: Utilize the library to load your service account key and generate OAuth tokens. These tokens are used to authenticate your API requests.

Making API Requests

With authentication taken care of, you can now make requests to the Indexing API to notify Google of new or updated pages.

Prepare Your API Request: Construct a POST request to the Indexing API's endpoint, including the URL of the page you wish to be indexed. Ensure your OAuth token is included in the request headers for authentication.
Handle Responses: Google's API will respond with the status of your request. Successful requests will return a status code indicating the request has been accepted for processing.

Sample code:

# Standard library imports
import os
import datetime

# Django-specific imports
import django
from django.urls import reverse
from django.conf import settings

# Third-party imports
import google.auth
from google.auth.transport.requests import Request
from google.oauth2.service_account import Credentials

# Local imports
from your_project.blog.models import Blog

# Setup Django environment
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
django.setup()

# Constants
SCOPES = ["https://www.googleapis.com/auth/indexing"]
ENDPOINT = "https://indexing.googleapis.com/v3/urlNotifications:publish"
JSON_KEY_FILE_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "your-project-12345.json")

# Initialize and authorize service account credentials
credentials = Credentials.from_service_account_file(JSON_KEY_FILE_PATH, scopes=SCOPES)
authed_session = google.auth.transport.requests.AuthorizedSession(credentials)


def get_blog_urls(num_blogs=200):
    """
    The maximum number of request you can send per day is 200 by default.
    You can adjust this number based on your daily quota or contact Google for a higher quota.
    Generates a list of blog URLs to be indexed, based on a calculated daily offset.
    """    
    blogs = Blog.objects.order_by('-id')[:num_blogs]
        
    base_url = settings.SITE_URL  # Assuming SITE_URL is defined in your Django settings

    urls = [f"{base_url}{reverse('blog_detail', args=[blog.slug])}" for blog in blogs]
    return urls


def index_urls():
    """
    Indexes URLs by making POST requests to the Google Indexing API.
    """
    url_list = get_blog_urls()
    print("Indexing URLs:", url_list)

    for url in url_list:
        body = {
            "url": url,
            "type": "URL_UPDATED"
        }
        response = authed_session.post(ENDPOINT, json=body)
        print(response.status_code, response.text)


if __name__ == "__main__":
    index_urls()

Conclusion

Integrating Google's Indexing API into your Django projects offers a proactive approach to SEO, ensuring that your content is indexed promptly by Google. By following the steps outlined in this guide, you can set up the necessary infrastructure, authenticate your applications, and begin making indexing requests programmatically. This not only enhances your site's visibility on Google Search but also gives you greater control over how and when your content is indexed.

How to Programmatically Populate Related Blogs in Django: Boosting User Experience and SEO

Leon Wei — Sat, 23 Mar 2024 16:24:57 +0000

Originally published at AISaaSTemplate

In this blog post, we will explore a practical solution for Django developers looking to programmatically populate related blogs using Python. This approach is particularly useful for enhancing the functionality of blog platforms or content management systems built with Django.

By implementing this feature, you can significantly improve the user experience on your site by suggesting relevant additional readings to your audience.

We will guide you through setting up your Django environment, computing similarities between blogs, and ultimately updating your database with related blog entries.

Creating a system that programmatically populates related blogs in a Django application is not just about enhancing the user experience; it's also a powerful strategy to improve your site's SEO.

Before we dive into the technical implementation, let's discuss why this feature is a win-win for both your audience and your site's search engine rankings.

Related blogs contribute significantly to internal linking, a crucial factor in SEO. Internal links are hyperlinks that point from one page to another within the same domain. Here's why they're important for SEO:

Navigation: They help users navigate your site more effectively, keeping them engaged longer.
Website Hierarchy: Internal links establish a hierarchy on your site, giving Google an idea of the structure of your content and which pages are most important.
Link Equity Distribution: They distribute page authority and ranking power throughout your site, improving the SEO of individual pages.

By automatically linking related blogs, you ensure that your site maintains a dynamic internal linking structure. This not only helps Google crawl and index your content more efficiently but also enhances the relevancy of your pages to specific search queries.

The end result? Improved visibility in search engine results pages (SERPs), driving more organic traffic to your site.

Setting Up Your Django Environment

Firstly, ensure your Django project is properly set up and configured. For our demonstration, we're assuming you have a Blog model defined in your Django application, typically located in models.py within your app directory (e.g., your_app_name/models.py). The Blog model should have a ManyToManyField to itself to represent related blogs, which can be defined as follows:

from django.db import models 
class Blog(models.Model): 
    # Other fields (e.g., title, content, etc.) 
    related_blogs = models.ManyToManyField('self', symmetrical=False, related_name='related_to')

It's crucial to have your Django settings properly configured, especially the database settings, since we'll be working directly with the database to populate the related blogs. The code snippet provided at the beginning of this post uses a specific database named 'production' for demonstration purposes. Make sure to adjust this to match your project's configuration.

Computing Similarities Between Blogs

To determine which blogs are related to each other, we employ the Jaccard similarity coefficient, a statistical measure used for gauging the similarity and diversity of sample sets.

We don't want to use common words in deciding the similiarities, so we can exclude them from the following list.

# Set of English stop words
stop_words = ['a',
'about',
'above',
'after',
'again',
'against',
'ain',
'all',
'am',
'an',
'and',
'any',
'are',
'aren',
"aren't",
'as',
'at',
'be',
'because',
'been',
'before',
'being',
'below',
'between',
'both',
'but',
'by',
'can',
'couldn',
"couldn't",
'd',
'did',
'didn',
"didn't",
'do',
'does',
'doesn',
"doesn't",
'doing',
'don',
"don't",
'down',
'during',
'each',
'few',
'for',
'from',
'further',
'had',
'hadn',
"hadn't",
'has',
'hasn',
"hasn't",
'have',
'haven',
"haven't",
'having',
'he',
'her',
'here',
'hers',
'herself',
'him',
'himself',
'his',
'how',
'i',
'if',
'in',
'into',
'is',
'isn',
"isn't",
'it',
"it's",
'its',
'itself',
'just',
'll',
'm',
'ma',
'me',
'mightn',
"mightn't",
'more',
'most',
'mustn',
"mustn't",
'my',
'myself',
'needn',
"needn't",
'no',
'nor',
'not',
'now',
'o',
'of',
'off',
'on',
'once',
'only',
'or',
'other',
'our',
'ours',
'ourselves',
'out',
'over',
'own',
're',
's',
'same',
'shan',
"shan't",
'she',
"she's",
'should',
"should've",
'shouldn',
"shouldn't",
'so',
'some',
'such',
't',
'than',
'that',
"that'll",
'the',
'their',
'theirs',
'them',
'themselves',
'then',
'there',
'these',
'they',
'this',
'those',
'through',
'to',
'too',
'under',
'until',
'up',
've',
'very',
'was',
'wasn',
"wasn't",
'we',
'were',
'weren',
"weren't",
'what',
'when',
'where',
'which',
'while',
'who',
'whom',
'why',
'will',
'with',
'won',
"won't",
'wouldn',
"wouldn't",
'y',
'you',
"you'd",
"you'll",
"you're",
"you've",
'your',
'yours',
'yourself',
'yourselves']

The code snippet provided offers a jaccard function to compute this similarity between two sets of words (extracted from blog titles in this case):

def jaccard(set1, set2): 
    """Compute the Jaccard similarity of two sets.""" 
    intersection = set1.intersection(set2) 
    union = set1.union(set2) 
    return len(intersection) / len(union) if union else 0

Using this function, we iterate over all combinations of blog titles to compute their pairwise Jaccard similarities. We focus on titles as a simplified example, but you could extend this to other attributes (e.g., content, tags) for a more comprehensive analysis. Here's a simplified version of how we process titles to compute similarities:

def compute_jaccard_similarity(blog_titles):
    title_sets = {
        blog_id: set(word for word in title.lower().split() if word not in stop_words)
        for blog_id, title in blog_titles
    }
    similarities = {blog_id: [] for blog_id, title in blog_titles}
    for (id1, title1), (id2, title2) in combinations(blog_titles, 2):
        score = jaccard(title_sets[id1], title_sets[id2])
        similarities[id1].append((id2, score))
        similarities[id2].append((id1, score))
    for blog_id in similarities:
        similarities[blog_id].sort(key=lambda x: x[1], reverse=True)
        similarities[blog_id] = [id for id, score in similarities[blog_id][:3]]
    return similarities

After calculating the similarities, we select the top N most similar blogs for each blog entry and update the related_blogs ManyToManyField accordingly. This is where we directly interact with the Django ORM to update the database:

def populate_related_blogs():
    blog_titles = list(Blog.objects.values_list('id', 'title'))
    print("blog_titles: ", blog_titles)

    related_blogs = compute_jaccard_similarity(blog_titles)
    print("related_blogs: ", related_blogs)

    for blog_id, related_ids in related_blogs.items():
        blog = Blog.objects.get(id=blog_id)
        if not blog.related_blogs.count() > 0:
            print("No related blogs found for blog: ", blog.title)
            blog.related_blogs.set(Blog.objects.filter(id__in=related_ids))


# Call this function to populate related blogs
populate_related_blogs()

This final step ties everything together by ensuring that each blog entry is linked to its most relevant counterparts in the database, thus enriching the user's browsing experience by suggesting related reads.

In our continuous effort to enhance our blog's user experience, we've implemented a feature that ensures you're always in the loop with content that's relevant to your interests. How do we keep things fresh and interconnected, you ask? Let me introduce you to a little behind-the-scenes magic: automated related blog population.

The Power of Automation

Using Django's robust framework, we've crafted a custom Python command that intelligently populates related blogs. This isn't just any script—it's a carefully designed function that links blogs by relevance, ensuring that you're always a click away from exploring similar topics of interest.

Here's a peek under the hood at the Python code making this possible:

from django.core.management.base import BaseCommand
from your_project_slug.blog.utils.populate_related_blogs import populate_related_blogs
from django.utils import timezone

class Command(BaseCommand):
    help = 'Populates related blogs by calling the populate_related_blogs function from blog/utils.'

    def handle(self, *args, **kwargs):
        self.stdout.write(self.style.SUCCESS(f'Starting to populate related blogs at: {timezone.now()}'))
        try:
            populate_related_blogs()
            self.stdout.write(self.style.SUCCESS(f'Successfully populated related blogs at {timezone.now()}'))
        except Exception as e:
            self.stdout.write(self.style.ERROR(f'Error during populating related blogs: {e} at {timezone.now()}'))

You can run Django command line with the following command

python manage.py populate_related_blogs

Scheduling Magic

But, how do we ensure this operation doesn't require manual intervention and the blog universe is always expanding at the right pace? Through the magic of scheduling!

By leveraging Django's management commands, we can schedule this script to run at regular intervals, ensuring that related content is always up to date and dynamically linked.

This scheduled operation runs seamlessly in the background, populating related blogs based on the latest content, without any downtime or disruption to your reading experience.

For example: in Heroku, you can create a new cron add-on and add a daily job to run it at 5am utc.

0 5 * * * /path/to/your/django/project/manage.py populate_related_blogs >> /path/to/your/logfile.log 2>&1

Why This Matters

Why go through all this trouble? Because we believe in providing a seamless, engaging, and interconnected reading experience. By automating the process of linking related blogs, we ensure that our content is not just informative but also accessible and relevant, enhancing your journey through our collection of insights, stories, and tutorials.

Wrapping Up

Implementing a system to programmatically populate related blogs can significantly enhance the functionality and user experience of your Django-based blog or content management system. By leveraging the power of the Jaccard similarity coefficient and Django's ORM, you can create dynamic, interconnected content that keeps readers engaged and encourages them to explore more of your site's offerings.

Remember, the code snippets provided here are a starting point. You might need to adjust them based on your specific project requirements and database configurations. Additionally, consider expanding the similarity computation beyond just blog titles to include content, tags, or even reader engagement metrics for more sophisticated recommendations.

We hope this guide helps you enhance your Django projects with dynamic related blog functionalities. Happy coding!

Let me know if you're ready to move on or if there's any part of this section you'd like to adjust or expand upon!

Django SaaS & AI Boilerplate

We hope you found this article helpful! If you're looking to bypass the setup complexities and dive straight into development with a production-ready, battle-tested template that includes Tailwind CSS, Django, Stripe, and much more, our Premium Django SaaS Boilerplate is the perfect solution for a swift launch.

Don't miss out on our launch special—get $100 OFF today and set your SaaS application on the fast track to success! Check out the Premium Django SaaS & AI Boilerplate now and take a significant leap forward in your development journey.

How to add Tailwind CSS to your Django Project

Leon Wei — Fri, 16 Feb 2024 02:32:39 +0000

Originally posted at
How to add Tailwind CSS to your Django Project

For Django developers eager to set their projects apart, this article offers a strategic guide on transitioning from Bootstrap to Tailwind CSS within Django Cookiecutter projects.

Tailwind CSS, a utility-first framework, is renowned for its capability to facilitate unique, custom designs efficiently. Its utility-class-based styling methodology grants developers enhanced control over their project's aesthetics, fostering a contemporary, streamlined appearance that diverges from conventional Bootstrap designs.

This guide is designed to streamline the process of integrating Tailwind CSS, thus empowering your Django projects with a visually distinct and modern frontend while maintaining the robust backend functionality Django is known for.

By following this tailored approach, developers can achieve a bespoke visual identity for their web applications, leveraging the strengths of Tailwind CSS to create engaging, high-quality user experiences.

Let's get started.

Step 1: Create an Empty Cookiecutter Django Project

1.1. Install Cookiecutter:

First, you need to install Cookiecutter. Use pip to install it globally on your machine:

pip install cookiecutter

After installing Cookiecutter, you can verify the installation by running cookiecutter --version.

1.2. Generate Your Django Project:

Use Cookiecutter with the Django template to scaffold your project or build it online with our free django cookiecutter generator. Run the following command and follow the prompts to customize your project:

cookiecutter https://github.com/pydanny/cookiecutter-django

This command creates a new Django project with a standard structure. Adjust settings as needed based on the prompts.

1.3 Optional CSS Minification:

pip install django_compressor

To compress your Tailwind CSS file, wrap your CSS link tag with {% compress css %} and {% endcompress %} in your templates.

Step 2: Setup Tailwind CSS

2.1 Initialize Tailwind CSS in Your Project:

Navigate to your Django project's root directory and initialize Tailwind CSS:

cd your_project

npm install -D tailwindcss

npx tailwindcss init

This command creates a tailwind.config.js file in your project root directory.

2.2 Configure Tailwind for Required Styles:

Modify the tailwind.config.js file to incorporate styles into your HTML files by adjusting the content option to encompass the paths to all your HTML template files.

module.exports = {
 content: [
        // Main Templates
        './your_project/templates/*.html',
        './your_project/templates/**/*.html',

        // Add other app's templates here
        //  './your_project/your_app/templates/*.html',
        // './your_project/your_app/templates/**/*.html',
    ],


  theme: {
    extend: {},
  },
  plugins: [],
};

2.3 Create Your Tailwind CSS File:

Create a CSS file (e.g., tailwind.css) in your static/css directory with the following content to include Tailwind's directives:

@tailwind base;
@tailwind components;
@tailwind utilities;

This file will be located at your project's static folder your_project/your_project/static/css/

2.4 Build Your Tailwind CSS:

Run the following command to generate your Tailwind CSS file (directly under your project root dirctory):

npx tailwindcss-cli@latest build ./your_project/static/tailwind.css -o ./your_project/static/css/style.css --watch

This command performs the following actions:

-i ./your_project/static/css/tailwind.css: Specifies the input file path (tailwind.css) located in the static/css directory of your Django SaaS application.

-o ./your_project/static/css/style.css: Defines the output file path (style.css) where the compiled CSS will be saved, also within the static/css directory.

--watch: Instructs Tailwind CSS to monitor the input file for changes. Whenever you modify and save tailwind.css, Tailwind CSS automatically recompiles it to style.css, ensuring that your changes are immediately reflected in the output file without needing to run the command again.

This setup is particularly useful during development, as it streamlines the workflow by eliminating the need for manual recompilation, allowing you to focus on styling and iterating on your designs more efficiently.

Once the style.css file is generated, you can include it at your your_project/template/base.html and style your entire project.

{% compress css %}

    <link href="{% static 'css/style.css' %}" rel="stylesheet"/>

{% endcompress %}

2.5 Removing Bootstrap Files

Within your base.html file, you will find two Bootstrap-related files: one with a .css extension and the other with a .js extension. Please remove both of these files to ensure they do not interfere with the new Tailwind CSS setup.

Step 3: Handling Safe CSS for Server-Side Rendering

This section emphasizes tailoring the Tailwind CSS setup to accommodate dynamic content and ensure styles are retained during server-side rendering. The crux of this adjustment lies in meticulously configuring tailwind.config.js to encompass your template paths and any dynamic class names your application employs.

For instance, if your server generates content using aspect- classes to maintain embedded youtube video aspect ratios, that aren't directly visible in your static HTML files monitored by the Tailwind CSS command line tool, these classes might inadvertently be omitted from the final stylesheet.

To prevent this, you should explicitly include such classes in the safelist within your tailwind.config.js file. This ensures that dynamically generated classes are preserved in the compiled CSS, maintaining the intended styling for server-rendered content.

module.exports = {
    content: [
        // Main Templates
        './your_project/templates/*.html',
        './your_project/templates/**/*.html',

        // Add other app's templates here
        //  './your_project/your_app/templates/*.html',
        // './your_project/your_app/templates/**/*.html',
    ],


    theme: {
        extend: {},
    },
    plugins: [],

    // NEW
    safelist: [
        {
            pattern: /aspect-+/
        }
        ]


};

Enjoy the Modern Look

After configuring Tailwind CSS and ensuring it's building correctly, you can start developing your Django application with a modern, responsive design.

For example: add the following tailwind styled sample code into your home.html inside the {% block content %}

<div class="conatiner flex min-h-screen flex-col justify-center overflow-hidden bg-gray-50 py-6 sm:py-12">

    <div class="absolute inset-0 bg-[url(/static/images/grid.svg)] bg-center [mask-image:linear-gradient(180deg,white,rgba(255,255,255,0))]"></div>
    <div class="relative bg-white px-6 pt-10 pb-8 shadow-xl ring-1 ring-gray-900/5 sm:mx-auto sm:max-w-lg sm:rounded-lg sm:px-10">
        <div class="mx-auto max-w-md flex">
            <div class="divide-y divide-gray-300/50 ml-5">
                <h1 class="text-3xl font-bold">Django SaaS Boilerplate + Tailwind CSS</h1>
                <div class="space-y-6 py-8 text-base leading-7 text-gray-600">
                    <p>Jumpstart your SaaS project with Django SaaS Boilerplate, including features like:</p>
                    <ul class="space-y-4">
                        <li class="flex items-center">
                            <svg class="h-6 w-6 flex-none fill-sky-100 stroke-sky-500 stroke-2"
                                 stroke-linecap="round" stroke-linejoin="round">
                                <circle cx="12" cy="12" r="11"/>
                                <path d="m8 13 2.165 2.165a1 1 0 0 0 1.521-.126L16 9" fill="none"/>
                            </svg>
                            <p class="ml-4">
                                Integrated Stripe for payments
                            </p>
                        </li>
                        <li class="flex items-center">
                            <svg class="h-6 w-6 flex-none fill-sky-100 stroke-sky-500 stroke-2"
                                 stroke-linecap="round" stroke-linejoin="round">
                                <circle cx="12" cy="12" r="11"/>
                                <path d="m8 13 2.165 2.165a1 1 0 0 0 1.521-.126L16 9" fill="none"/>
                            </svg>
                            <p class="ml-4">
                                Pre-configured Tailwind CSS for design
                            </p>
                        </li>
                        <li class="flex items-center">
                            <svg class="h-6 w-6 flex-none fill-sky-100 stroke-sky-500 stroke-2"
                                 stroke-linecap="round" stroke-linejoin="round">
                                <circle cx="12" cy="12" r="11"/>
                                <path d="m8 13 2.165 2.165a1 1 0 0 0 1.521-.126L16 9" fill="none"/>
                            </svg>
                            <p class="ml-4">User authentication and management</p>
                        </li>
                    </ul>
                    <p>Perfect for launching your SaaS application with a solid foundation and modern
                        aesthetics.</p>
                </div>
                <div class="pt-8 text-base font-semibold leading-7">
                    <p class="text-gray-900">Ready to accelerate your SaaS project?</p>
                    <p>
                        <a href="https://djangosaas.com/#pricing" class="text-sky-500 hover:text-sky-600">Get the
                            Boilerplate &rarr;</a>
                    </p>
                </div>
            </div>
        </div>
    </div>
</div>

Do you see the following? Enjoy the great modern look and feel!

Additional Resources for Tailwind CSS and Django Integration

Official Tailwind CSS Documentation: Start with the Tailwind CSS documentation for the most up-to-date information on installation, configuration, and usage. It's an invaluable resource for understanding Tailwind's core concepts and features.

Stack Overflow: For troubleshooting specific issues, Stack Overflow offers a wealth of information. Searching for "Tailwind CSS Django integration issues" can lead you to discussions and solutions that others in the Django and Tailwind community have shared.

Django and Tailwind CSS Community Forums: Engage with communities on platforms like Reddit, Discord, or specialized forums for Django and Tailwind CSS. Community forums are excellent places to ask questions, share experiences, and get advice from experienced developers.

By exploring these additional resources, you can deepen your understanding of how Tailwind CSS works within a Django environment and navigate any challenges that arise during the integration process.

Whether you're troubleshooting a specific issue or looking to enhance your project with advanced Tailwind features, the collective knowledge and experiences shared within these resources can significantly aid in your development journey.

Django SaaS & AI Boilerplate

Reddit Social Listening with Python

Leon Wei — Sun, 13 Nov 2022 00:35:01 +0000

Originally published at Reddit Social Listening with Python

All source code can be found here:
https://github.com/theleonwei/reddit_bot

Introduction:
Reddit is the second-most popular website in the United States, with more than 300 million unique visitors per month.

It's also one of the most trafficked sites on the internet and has become an important part of online marketing strategy for brands across industries.

This article will show you how to programmatically set up a keyword monitor service on Reddit using Python and the PRAW library.

You will also learn how to set up this web service using Django and run it on your local machine to monitor Reddit, save the leads into a database automatically.

All of the code and step-by-step instructions are based on the assumption that you are on a  Mac.

Table of contents:

Setting up the Django Project with the CookieCutter template
Register a Reddit application and install the PRAW library
Keyword monitoring with regular expression
Persisting data to a Postgres database
Leads report view
Schedule a cron job to check Reddit periodically

Setting up the Django Project with the CookieCutter template
Cookiecutter Django is a framework for jumpstarting production-ready Django projects quickly.

To learn more, you can visit their official website on Github:

https://github.com/cookiecutter/cookiecutter-django

Step 1: install cookiecutter.
Open up your favorite terminal app (mine is iTerm2), and install the latest cookiecutter.

pip install "cookiecutter>=1.7.0"

Step 2: start a Django project.

cookiecutter https://github.com/cookiecutter/cookiecutter-django

Follow the instruction, answer the questions, and set up the project; here are my choices:

project_name [My Awesome Project]: Reddit Bot
project_slug [reddit_bot]:
description [Behold My Awesome Project!]: My awesome Reddit Bot Project
author_name [Daniel Roy Greenfeld]: Leon W
domain_name [example.com]:
email [leon-w@example.com]:
version [0.1.0]:
Select open_source_license:
1 - MIT
2 - BSD
3 - GPLv3
4 - Apache Software License 2.0
5 - Not open source
Choose from 1, 2, 3, 4, 5 [1]: 5
timezone [UTC]: US/Pacific
windows [n]:
use_pycharm [n]: y
use_docker [n]:
Select postgresql_version:
1 - 14
2 - 13
3 - 12
4 - 11
5 - 10
Choose from 1, 2, 3, 4, 5 [1]:
Select cloud_provider:
1 - AWS
2 - GCP
3 - None
Choose from 1, 2, 3 [1]:
Select mail_service:
1 - Mailgun
2 - Amazon SES
3 - Mailjet
4 - Mandrill
5 - Postmark
6 - Sendgrid
7 - SendinBlue
8 - SparkPost
9 - Other SMTP
Choose from 1, 2, 3, 4, 5, 6, 7, 8, 9 [1]:
use_async [n]: n
use_drf [n]: n
Select frontend_pipeline:
1 - None
2 - Django Compressor
3 - Gulp
Choose from 1, 2, 3 [1]:
use_celery [n]: n
use_mailhog [n]: n
use_sentry [n]: n
use_whitenoise [n]: n
use_heroku [n]: y
Select ci_tool:
1 - None
2 - Travis
3 - Gitlab
4 - Github
Choose from 1, 2, 3, 4 [1]:
keep_local_envs_in_vcs [y]: n
debug [n]: n
 [SUCCESS]: Project initialized, keep up the good work!

Step 3: Install dependencies

cd reddit_bot; ls

Those are the files and directories that we have got so far.

Procfile  locale  reddit_bot  setup.cfg
README.md  manage.py  requirements  utility
config  merge_production_dotenvs_in_dotenv.py requirements.txt
docs  pytest.ini  runtime.txt

Step 3.1: Create a virtual environment.

reddit_bot ➤ python3 -m venv ./venv

After, you should see a newly created venv folder.

Procfile  locale reddit_bot  setup.cfg
README.md  manage.py requirements  utility
config  merge_production_dotenvs_in_dotenv.py requirements.txt  venv
docs  pytest.ini  runtime.txt

Step 3.2: Activate the virtual environment and install all dependencies.

source venv/bin/activate

Notice there is a (venv) prompt, which means you have successfully activated the virtual environment.

Note: if you have not installed a Postgres database on your  Mac, you must install it first.

Check this article on Postgres installation on Mac for more details.

Install the dependencies from the local.txt files (slightly different than production requirements as it gives you more tools for debugging and testing.)

(venv) reddit_bot ➤
(venv) reddit_bot ➤ pip install -r requirements/local.txt

Step 3.3: Create a local database reddit_bot

(venv) reddit_bot ➤ createdb reddit_bot

After that, start the Django web server for testing.

(venv) reddit_bot ➤ python manage.py runserver

You will probably see something like the following.

Watching for file changes with StatReloader
INFO 2022-09-17 11:28:04,131 autoreload 17789 4335895936 Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 28 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): account, admin, auth, contenttypes, sessions, sites, socialaccount, users.
Run 'python manage.py migrate' to apply them.
September 17, 2022 - 11:28:04
Django version 3.2.15, using settings 'config.settings.local'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[17/Sep/2022 11:28:16] "GET / HTTP/1.1" 200 13541
[17/Sep/2022 11:28:16] "GET /static/debug_toolbar/css/toolbar.css HTTP/1.1" 200 11815
[17/Sep/2022 11:28:16] "GET /static/css/project.css HTTP/1.1" 200 228
[17/Sep/2022 11:28:16] "GET /static/debug_toolbar/css/print.css HTTP/1.1" 200 43
[17/Sep/2022 11:28:16] "GET /static/debug_toolbar/js/toolbar.js HTTP/1.1" 200 12528
[17/Sep/2022 11:28:16] "GET /static/js/project.js HTTP/1.1" 200 45
[17/Sep/2022 11:28:16] "GET /static/debug_toolbar/js/utils.js HTTP/1.1" 200 4479
[17/Sep/2022 11:28:16] "GET /static/images/favicons/favicon.ico HTTP/1.1" 200 8348

Since the database is brand new, we need to initialize it with some built-in Django tables.

(venv) reddit_bot ➤ python manage.py migrate

Once the migration is completed, restart the server.

(venv) reddit_bot ➤ python manage.py migrate

open your favorite browser (I am using the latest Chrome) and visit localhost:8000 make sure you see the website succesfully.

Congratulations on finishing setting up your local Django server; next, let's set up our Reddit account and install the Reddit API library: PRAW.

Register a Reddit application and install the PRAW library
We assume you already have a Reddit account to set up a Reddit developer account. If not, simply visit https://reddit.com and create one, then come back.

You must first register an application of the appropriate type on Reddit.

Then Visit https://www.reddit.com/prefs/apps/

Note: sometimes, I am having some page redirect issues when visiting the above page; if that happens to you, try visiting the following instead:

https://old.reddit.com/prefs/apps/

Scroll down and click the create another app... button

reddit monitoring tutorial | register application

For the name of your app, anything should be fine

On the app type: since we will be building something running in the backend, choose script.

And for redirect uri, enter http://localhost:8000

Then click the create app button to finish this step.

There are two tokens that we will need for our service to run:

client id: the one beneath personal use script
client secret: the one to the right of secret

Finally, we need to install the PRAW library and try to connect with Reddit using the secret keys from the last step.

(venv) reddit_bot ➤ pip install praw

Next, we append PRAW to the dependency (otherwise, the service won't work when we deploy to production:

(venv) reddit_bot ➤ pip freeze | grep praw >> requirements/base.txt

It's a common best practice not to save sensitive information such as your app secret tokens in the git repository, so let's create a new file .env at the root of your project so we can access the app secret on the local machine.

(venv) reddit_bot ➤ vim .env

Replace the secret and the client_id with yours from the last step.

For the user agent, it's not that important. To find out exactly what your user agent is, simply go to google and type "find my user agent" and copy and paste yours into the .env file.

Next, make sure you add .env into the list of files that will not be checked into the git repository by editing the .gitignore file.

(venv) reddit_bot ➤ vim .gitignore

And add .env to the file; after that, run the following to load the environment variables.

Test your Reddit connection. In your terminal, run the following:

(venv) reddit_bot ➤ source .env

We also need to load the variables into Django's environment variables.

Open up the config/settings/local.py file (I am using Pycharm) and add the following:

# Reddit settings:
REDDIT_SECRET=env('SECRET')
REDDIT_CLIENT_ID=env('CLIENT_ID')
REDDIT_USERAGENT=env('REDDIT_USERAGENT')

In the terminal

Launch the Django console

(venv) reddit_bot ➤ python manage.py shell


In [1]: from django.conf import settings

In [2]: import praw

In [3]: reddit = praw.Reddit(
   ...:     client_id=settings.REDDIT_CLIENT_ID,
   ...:     client_secret=settings.REDDIT_SECRET,
   ...:     user_agent=settings.REDDIT_USERAGENT,
   ...: )

If you don't receive any error message, it means you have successfully created a Reddit instance through PRAW.

Next, let's run a simple task to check the connection.

In [4]: for submission in reddit.subreddit("marketing").hot(limit=10):
   ...:     print(submission.title)
   ...:
New Job Listings
Sorry if this isn't allowed, but I recently created a subreddit focused on the business side of art, and would love for people to go there and share their knowledge and experiences.
I read privacy and policies of Tiktok, IG and Other Platforms. Here’s what I learned about Social Media Platforms!
Has anyone actually worked with an impressive agency?
How to bring first customers to shop?
Facebook ad numbers don't ad up
Beginning my career in marketing, looking to go in to an Agency
I hate digital marketing - help me find a new role
Where to start for ecom store?
Making a website

If you see something similar to the above, congratulations, you've successfully connected with Reddit's official API and retrieved the top 10 hot posts from r/marketing, congratulations!

Keyword monitoring with regular expression
Assuming you run a Facebook ad agency and your target customers are new to Facebook ads, wouldn't it be nice if you could respond to someone who has questions about Facebook ads on Reddit?

Chiming in and joining a conversation on Reddit will help:

Establish your reputation as a Facebook ads expert;
Spread the word about your service to the world's largest online community and drive quality traffic to your website;
If you get enough votes, your response may become a backlink to improve your SEO

For this article, we will show you how to find any posts whose title contains the phrase 'facebook'.

Of course, you can continue optimizing this matching rule and develop your own solution.

Again, we first open up a Django console.

(venv) reddit_bot ➤ python manage.py shell

Inside of the Django console:

import praw
from django.conf import settings

import re # new, the python regular expression library

keyword =  "facebook"

reddit = praw.Reddit(
     client_id=settings.REDDIT_CLIENT_ID,
     client_secret=settings.REDDIT_SECRET,
     user_agent=settings.REDDIT_USERAGENT,
)

for submission in reddit.subreddit("marketing").hot(limit=100): # we are searching 100 hottest posts on the marketing subreddit
    if re.search(keyword, submission.title, re.IGNORECASE): # new, notice we are ignoring the case sensitivity
        print(submission.title)

Your results may differ from mine (as we ran this search on September 18th, 2022).

In [2]: for submission in reddit.subreddit("marketing").hot(limit=100):
   ...:     if re.search(keyword, submission.title, re.IGNORECASE): # new, notice we are ignoring the case sensitivity
   ...:         print(submission.title)
   ...:
Facebook ad numbers don't ad up
Facebook & Instagram Ads campaign Setup
Measuring the impact of Facebook
How many interests is too many Interest - Facebook Ads


We've found four discussions about Facebook without too much work. How awesome is it!

Let's save those results and other metadata such as URLs, post date, and content into a database so we don't lose them.

Persisting data to a Postgres database.
Let's first create a new app

# In the root dir of your project
(venv) reddit_bot ➤ django-admin startapp reddit

Note: we also need to move the newly created app to the reddit_bot subdirectory. This step is important due to the way cookie-cutter structured our project.

mv reddit ./reddit_bot

Next, let's update the apps.py config file.

The default name is "reddit", we need to update it to "reddit_bot.reddit" since we have moved it from the root directory to the subdirectory.

And don't forget to include this app in the base.py config file.

Now let's open up the models.py file and add our first class.

class Lead(models.Model):
    post_id = models.CharField(max_length=10) # Original post id
    title = models.TextField()
    content = models.TextField()
    posted_at = models.DateTimeField()
    url = models.URLField(max_length=500)

In the command line, let's install the app and model and make the migration.

(venv) reddit_bot ➤ python manage.py makemigrations
Migrations for 'reddit':
  reddit_bot/reddit/migrations/0001_initial.py
    - Create model Lead

(venv) reddit_bot ➤ python manage.py migrate
Operations to perform:
  Apply all migrations: account, admin, auth, contenttypes, reddit, sessions, sites, socialaccount, users
Running migrations:
  Applying reddit.0001_initial... OK

Next, let's create a command line script to execute the keyword matching and save the results into the Lead model.

For this script, let's call it lead_finder.py and put it under reddit/management/commands folder.

First we need to create the two folders:

# Move to the reddit app directory
(venv) reddit_bot ➤ cd reddit_bot/reddit
(venv) reddit ➤ mkdir management
(venv) reddit ➤ mkdir management/commands

# Inside reddit_bot/reddit_bot/reddit/management/commands/lead_finder.py file
import datetime as DT
import re

import praw
from django.conf import settings
from django.core.management.base import BaseCommand
from django.utils import timezone
from django.utils.timezone import make_aware

from reddit_bot.reddit.models import Lead

KEYWORD = "facebook"
SUBREDDIT = 'marketing'

reddit = praw.Reddit(
    client_id=settings.REDDIT_CLIENT_ID,
    client_secret=settings.REDDIT_SECRET,
    user_agent=settings.REDDIT_USERAGENT,
)

def convert_to_ts(unix_time):
    try:
        ts = make_aware(DT.datetime.fromtimestamp(unix_time))
        return ts
    except:
        print(f"Converting utc failed for {unix_time}")
        return None


def populate_lead(keyword, subreddit):
    for submission in reddit.subreddit(subreddit).hot(limit=100):
        if re.search(keyword, submission.title, re.IGNORECASE):
            if not Lead.objects.filter(post_id = submission.id):
                Lead.objects.create(post_id=submission.id,
                                    title=submission.title,
                                    url=submission.permalink,
                                    content=submission.selftext,
                                    posted_at=convert_to_ts(submission.created_utc))


class Command(BaseCommand):
    help = 'Populating leads'

    def handle(self, *args, **kwargs):
        try:
            current_time = timezone.now()
            self.stdout.write(f'Populating leads at {(current_time)}')
            populate_lead(KEYWORD, SUBREDDIT)
        except BaseException as e:
            current_time = timezone.now().strftime('%X')
            self.stdout.write(self.style.ERROR(f'Populating feeds failed at {current_time} because {str(e)}'))

        current_time = timezone.now()
        self.stdout.write(self.style.SUCCESS(f'Successfully populated new leads at {current_time}'))
        return

Some explanation:

This script has 3 parts, the convert_to_ts function converts a UNIX time to human-readable format. Reddit stored the timestamp when a post was first created in the format of a big integer.

The populate_lead uses the same logic in our last section and saves the new lead (if it has not already been saved in our table, remember, we enforced the post_id as the primary key in our Lead model definition)

Lastly, we created a Command class so that we can execute the populate_lead in a command line. There are other ways to execute a script on the command line, but this way is more of a Django style, in my opinion.

Finally, we can try to execute the script and populate some leads.

(venv) reddit_bot ➤ python manage.py lead_finder
Populating leads at 2022-09-18 17:50:35.045981+00:00
Successfully populated new leads at 2022-09-18 17:50:36.558052+00:00

Let's open up the Django console to verify the results are saved successfully.

from reddit_bot.reddit.models import Lead

for lead in Lead.objects.all():
        print(f'''title: {lead.title}\nposted_at:{lead.posted_at}\nurl: {lead.url}\n''')



title: Facebook ad numbers don't ad up
posted_at:2022-09-17 21:14:24+00:00
url: /r/marketing/comments/xgxv9c/facebook_ad_numbers_dont_ad_up/

title: Facebook & Instagram Ads campaign Setup
posted_at:2022-09-17 08:01:38+00:00
url: /r/marketing/comments/xgghl9/facebook_instagram_ads_campaign_setup/

title: Measuring the impact of Facebook
posted_at:2022-09-16 20:49:14+00:00
url: /r/marketing/comments/xg2kbi/measuring_the_impact_of_facebook/

title: How many interests is too many Interest - Facebook Ads
posted_at:2022-09-16 01:32:28+00:00
url: /r/marketing/comments/xfdwsg/how_many_interests_is_too_many_interest_facebook/

Here we go. All four leads persisted successfully!

Leads report view
It's cool we can see the data in the console, but it will be easier if we can view the leads in a table from a browser.

Inside the view.py file, let's create a ListView.

#inside reddit_bot/reddit_bot/reddit/views.py

from django.views.generic import ListView

from .models import Lead

# Create your views here.
class LeadView(ListView):
    model = Lead
    template_name = 'lead_list.html'

lead_view = LeadView.as_view()

We also need to create a HTML file 'lead_list.html' inside of a new directory called templates under the reddit app.

{% extends 'base.html' %}

    {% block content %}
        <table class="table table-striped">
        <thead>
        <tr>
        <th scope="col">ID</th>
        <th scope="col">Title</th>
        <th scope="col">Posted At</th>
        <th scope="col">Content</th>
        </tr>
        </thead>
        <tbody>

        {% for lead in object_list %}
            <tr>
            <th scope="row">{{ lead.post_id }}</th>
            <td><a href="https://reddit.com{{ lead.url }}"> {{ lead.title }}</a></td>
            <td>{{ lead.posted_at }}</td>
            <td>{{ lead.content }}</td>
            </tr>
        {% endfor %}

        </tbody>
        </table>

    {% endblock %}

Next, we need to add a URL path to access this view.

Create a new file: urls.py under the reddit app.

inside reddit_bot/reddit_bot/reddit/urls.py

from django.urls import path

from reddit_bot.reddit.views import lead_view

app_name = "reddit"

urlpatterns = [

    path("leads/", view=lead_view, name="leads"),

]

Finally, we must include the reddit app's URLs file on the project level.

Final step: check the page, open your browser, and visit: http://localhost:8000/reddit/leads/

And if you click on the title, you will be redirected to the Reddit post page, where you can engage with your target customers. How cool is that!

Schedule a cron job to check Reddit periodically.
We are almost done, and if you are like me, we like to automate our tasks; how about we schedule the job to be run automatically?

And that's super easy.

In the terminal, type crontab -e and enter.

Add the following line (you will need to edit the path of the reddit_bot Django project)

1 * * * * cd ~/reddit_bot; source venv/bin/activate; source .env; python manage.py lead_finder >/tmp/stdout.log 2>/tmp/stderr.log

It will run every hour at the 1 minute past that hour, for example, if now is 11:35 am, and the next time this job will run at 12:01 pm, and 13:01 pm, etc.

Of course, you can change the schedule that works best for you. You can use the following to customize your cron job.

https://crontab.guru/

Conclusion

We've gone through how to set up your own reddit keywords monitoring with python.

Building a Python/Django Startup?

Check out Django SaaS & AI Boilerplate, ship fast and start generating revenue within days, not weeks. Kickstart your success journey today.

DEV Community: Leon Wei

A Django Developer's Guide to Leveraging Google's Indexing API

Step 1: Project Setup in Google API Console

Creating a Google API Console Project

Setting Up a Service Account

Step 2: Verify Site Ownership with Google Search Console

Step 3: Integrating with Your Django Project

Generating OAuth Tokens

Making API Requests

Sample code:

Conclusion

How to Programmatically Populate Related Blogs in Django: Boosting User Experience and SEO

The SEO Benefits of Related Blogs

Setting Up Your Django Environment

Computing Similarities Between Blogs

Populating the Database with Related Blogs

One More Thing: Automate Related Blog Populations with Django Command

The Power of Automation

Scheduling Magic

Why This Matters

Wrapping Up

Django SaaS & AI Boilerplate

How to add Tailwind CSS to your Django Project

Step 1: Create an Empty Cookiecutter Django Project

1.1. Install Cookiecutter:

1.2. Generate Your Django Project:

1.3 Optional CSS Minification:

Step 2: Setup Tailwind CSS

2.1 Initialize Tailwind CSS in Your Project:

2.2 Configure Tailwind for Required Styles:

2.3 Create Your Tailwind CSS File:

2.4 Build Your Tailwind CSS:

2.5 Removing Bootstrap Files

Step 3: Handling Safe CSS for Server-Side Rendering

Enjoy the Modern Look

Additional Resources for Tailwind CSS and Django Integration

Django SaaS & AI Boilerplate

Reddit Social Listening with Python

inside reddit_bot/reddit_bot/reddit/urls.py

Conclusion

Building a Python/Django Startup?