DEV Community: Solve Computer Science

Summarization experiments with Hugging Face Transformers - part 1

Solve Computer Science — Mon, 14 Jul 2025 17:10:24 +0000

🧩 The problem

Summarization is a useful process to condense information while retaining the original message. There are several ways to perform this, but, of course, AI is the trend now. If you follow my YouTube channel you know I tend to prefer self-hosted solutions. These systems guarantee more privacy and independence, but there are some trade offs.

In this post, and subsequent ones, I'll show you some tests I made for summarization purposes using the Hugging Face Transformers Python library, locally, on CPU.

🤔 Reason

My final objective is to automatically generate YouTube video summaries to put in blog posts.

Now, I can't copy the YouTube video descriptions verbatim for SEO purposes: search engines don't like duplicate content. The obvious solution is to do some kind of summarization. First of all, I always use whisper and proofread the SRT file containing the recognized audio before uploading it to YouTube. I then copy the SRT file verbatim in an LLM chat, in chunks, and ask ChatGPT to generate a summary once I say so. This last step is tedious and you always rely on a SaaS. Some alternatives include the use local AI models via Ollama, or more specialized software that can run different tasks besides text generation.

The Python Transformers library by Hugging Face does just that and, thanks to its Pipeline API it is trivial to do this work (or it should be ?!).

🤗 Hugging face documentation

If you search the official docs for summarization, this is the first page that comes up. The TL;DR of that page is:

fine-tune a model
perform inference to generate a summary

A screenshot of the Transformers summarization page

To keep things simple, I just wanted to implement use a pre-trained model for inference.

🖥️ Meet the pipeline API

In HF, pipelines are probably the simplest notion I learned until now. If you follow this other tutorial you'll see that you can perform most tasks, including summarization, with just 3 lines of code. So this is more or less what I tried first (I added newlines to avoid scrolling)¹:

First attempt done by folowing the tutorial

The result is this²:

First output

The first two lines are warnings from the Transformers library which can be suppressed by setting a less verbose logging level.

As I said, I'm doing inference on CPU, so no surprises here. About the max_length parameter, it corresponds to the maximum number of tokens in the output. Apparently the default value for this model is too big for this input: it should be tweaked appropriately so that the summary still makes sense.

🎉 Conclusion

Can you see the potential problem here? Although the quality of the summary seems OK for this specific text, the model seems to have been trained for bigger inputs. In-fact, the length of the summary is about 82% of length of the input text.

In the next posts we'll see how to improve this kind of inference. I also tested other models to see how they measure up, so stay tuned.

Footnotes

Code


from transformers import pipeline

pipeline = pipeline(task='summarization', model='facebook/bart-large-cnn')

# Text from https://en.wikipedia.org/wiki/Artificial_intelligence
# Creative Commons Attribution-ShareAlike 4.0 License
print(pipeline('Artificial intelligence (AI) is the capability of computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making. It is a field of research in computer science that develops and studies methods and software that enable machines to perceive their environment and use learning and intelligence to take actions that maximize their chances of achieving  defined goals')[0]['summary_text'])



Device set to use cpu ↩

Your max_length is set to 142, but your input_length is only 79. Since this is a summarization task, where outputs shorter than the input are typically wanted, you might consider decreasing max_length manually, e.g. summarizer('...', max_length=39)

Artificial intelligence is the capability of computational systems to perform tasks typically associated with human intelligence. It is a field of research in computer science that develops and studies methods and software that enable machines to perceive their environment and use learning and intelligence to take actions that maximize their chances of achieving defined goals.

YouTube channel mirror on Jekyll - part 4

Solve Computer Science — Thu, 10 Jul 2025 17:00:00 +0000

🧩 The problem

In this last post of the series we'll see how to automatically create the Jekyll pages for YouTube videos to mimic a mirror-like system. We already know how to download the elements using the script we finalized in the previous post.

These pages simply contain an HTML5 video embed, useful in case YouTube goes down or something happens to the channel.

⚠️ Warning

Just like the previous posts, ⚠️⚠️ before continuing, please only mirror content you have permission to... ⚠️⚠️

✅ The solution

This time we are back in the Jekyll blog repository. We also have all the content we need served via Apache:

videos
thumbnails
titles

🗂️ Apache HTTPD listings

The directory with the long name only contains the channel avatar and needs to be ignored.

As you see, each directory name corresponds to a YouTube video ID. The standard Apache HTTP directory listing makes things really simple and linear to get the information we need. To keep things stateless and maintenance-free I can scrape this page and assume that every element we need is already present in the directories:

def filter_youtube_video_ids(tag) -> bool:
    r"""A BeautifulSoup 4 compatible callback function."""
    ok: bool = False
    if tag and tag.string and re.match(r'[a-zA-Z0-9_-]{11}/', tag.string):
        ok = True

    return ok


def get_youtube_video_ids_from_apache_auto_indexing(url: str) -> list[str]:
    r"""Scrape YouTube video ids from an Apache HTTPD "auto-indexing" page."""
    with httpx.Client() as client:
        response = client.get(url)

    soup = BeautifulSoup(response.text, 'html.parser')
    new_soup = list(set(soup.find_all('a', string=filter_youtube_video_ids)))
    return [tag.string[:-1] if tag.string[-1] == '/' else tag.string for tag in new_soup]

What does all this do? First of all we download the page content and load it as a BeautifulSoup instance. We then need to get all unique <a> HTML elements having a YouTube video ID as content. We can use a callback function with a regex to filter names for this purpose. As far as I know, YouTube video IDs are strings of length 11 containing a combination of these characters:

letters (a-zA-Z)
numbers (0-9)
dash (-)
underscore (_)

The second function returns all matching video IDs present on the Apache listings page.

🕵️‍♂️ Get local video IDs

The next step is to know which video pages are already present on the blog. Each markdown page file name corresponds to a video ID:

def get_youtube_video_ids_from_markdown_files(src_dir: str) -> list[str]:
    r"""Get all markdown files."""
    markdown_dir: pathlib.Path = pathlib.Path(src_dir)
    return [f.stem for f in markdown_dir.rglob('*') if f.is_file() and f.suffix == '.md']

🔍 Filter video IDs

We now need to filter out the video IDs already present locally. The set difference operator is really useful for this.

def filter_youtube_video_ids_markdown_files(videos_baseurl: str, pages_base_directory: str) -> list[str]:
    r"""Get all remote videos not available locally."""
    remote_videos: list[str] = get_youtube_video_ids_from_apache_auto_indexing(videos_baseurl)
    local_videos: list[str] = get_youtube_video_ids_from_markdown_files(pages_base_directory)

    return list(set(remote_videos) - set(local_videos))

🥣 Prepare the content

Our pages need a title attribute. We can simply prepare all the URLs in a data structure and scrape the title files, one by one. Remember to use an HTTP session (Client) to improve performance.

def get_remote_data(video_baseurl: str, video_ids: list[str]) -> list[dict]:
    r"""Get the data structure"""
    title_files: list[dict] = [
        {
            'id': v,
            'title_url': urljoin(urljoin(video_baseurl, v) + '/', 'title.txt'),
            'video_url': urljoin(urljoin(video_baseurl, v) + '/', v + '.webm'),
            'thumbnail_url': urljoin(urljoin(video_baseurl, v) + '/', v + '.png'),
        } for v in video_ids
    ]

    with httpx.Client() as client:
        for t in title_files:
            t['title'] = client.get(t['title_url']).text.strip()

    return title_files

📽️ Video pages

Creating a page is easy once you have the data. f-strings come in handy:

def create_markdown_page(data: dict):
    return f"""---
layout: youtube_mirror
title: "|-"
    {data["title"]}


permalink: /courses/solvecomputerscience-youtube-channel/{data["id"]}/
description: "|-"
    {data["title"]}


enable_markdown: true
lang: 'en'
backup_url: ! '{data["video_url"]}'
poster_image: ! '{data["thumbnail_url"]}'
is_youtube_mirror: true
canonical_url: ! 'https://www.youtube.com/watch?v={data["id"]}'
---
"""


def create_markdown_pages(pages_base_directory: str, data: list[dict]):
    for d in data:
        page: str = create_markdown_page(d)
        filename: str = ''.join([d['id'], '.md'])
        pathlib.Path(pages_base_directory, filename).write_text(page)

Remember from an earlier post in this series that these pages only need some YAML front matter.

👨‍💻 pre-commit hook

Finally, just like the other script, we need to set up the local pre-commit hook with the Python dependencies as requirements.

- repo: local
  hooks:
    - id: generate_youtube_mirror_pages
      name: generate_youtube_mirror_pages
      entry: python ./.scripts/youtube/generate_youtube_video_pages.py
      verbose: true
      always_run: true
      pass_filenames: false
      language: python
      types: [python]
      additional_dependencies: ['httpx>=0,<1', 'beautifulsoup4>=4.13,<4.14']

🎉 Conclusion

Click to open the full script

import re
import pathlib
import itertools
import time
import datetime
from urllib.parse import urlparse, parse_qs, urljoin

import httpx
from bs4 import BeautifulSoup


VIDEOS_BASEURL: str = 'https://your.apache.directory/listing/page'
PAGES_BASE_DIRECTORY: str = '_courses/solve-computer-science-youtube-channel'


def filter_youtube_video_ids(tag) -> bool:
    r"""A BeautifulSoup 4 compatible callback function."""
    ok: bool = False
    if tag and tag.string and re.match(r'[a-zA-Z0-9_-]{11}/', tag.string):
        ok = True

    return ok


def get_youtube_video_ids_from_apache_auto_indexing(url: str) -> list[str]:
    r"""Scrape YouTube video ids from an Apache HTTPD "auto-indexing" page."""
    with httpx.Client() as client:
        response = client.get(url)

    soup = BeautifulSoup(response.text, 'html.parser')
    new_soup = list(set(soup.find_all('a', string=filter_youtube_video_ids)))
    return [tag.string[:-1] if tag.string[-1] == '/' else tag.string for tag in new_soup]


def get_youtube_video_ids_from_markdown_files(src_dir: str) -> list[str]:
    r"""Get all markdown files."""
    markdown_dir: pathlib.Path = pathlib.Path(src_dir)
    return [f.stem for f in markdown_dir.rglob('*') if f.is_file() and f.suffix == '.md']


def filter_youtube_video_ids_markdown_files(videos_baseurl: str, pages_base_directory: str) -> list[str]:
    r"""Get all remote videos not available locally."""
    remote_videos: list[str] = get_youtube_video_ids_from_apache_auto_indexing(videos_baseurl)
    local_videos: list[str] = get_youtube_video_ids_from_markdown_files(pages_base_directory)

    return list(set(remote_videos) - set(local_videos))


def get_remote_data(video_baseurl: str, video_ids: list[str]) -> list[dict]:
    r"""Get the data structure"""
    title_files: list[dict] = [
        {
            'id': v,
            'title_url': urljoin(urljoin(video_baseurl, v) + '/', 'title.txt'),
            'video_url': urljoin(urljoin(video_baseurl, v) + '/', v + '.webm'),
            'thumbnail_url': urljoin(urljoin(video_baseurl, v) + '/', v + '.png'),
        } for v in video_ids
    ]

    with httpx.Client() as client:
        for t in title_files:
            t['title'] = client.get(t['title_url']).text.strip()

    return title_files


def create_markdown_page(data: dict):
    return f"""---
layout: youtube_mirror
title: |-
    {data["title"]}


permalink: /courses/solvecomputerscience-youtube-channel/{data["id"]}/
description: |-
    {data["title"]}


enable_markdown: true
lang: 'en'
backup_url: ! '{data["video_url"]}'
poster_image: ! '{data["thumbnail_url"]}'
is_youtube_mirror: true
canonical_url: ! 'https://www.youtube.com/watch?v={data["id"]}'
---
"""


def create_markdown_pages(pages_base_directory: str, data: list[dict]):
    for d in data:
        page: str = create_markdown_page(d)
        filename: str = ''.join([d['id'], '.md'])
        pathlib.Path(pages_base_directory, filename).write_text(page)


if __name__ == '__main__':
    video_ids: list[str] = filter_youtube_video_ids_markdown_files(VIDEOS_BASEURL, PAGES_BASE_DIRECTORY)
    data: list[dict] = get_remote_data(VIDEOS_BASEURL, video_ids)
    create_markdown_pages(PAGES_BASE_DIRECTORY, data)

Let me know if you found this series of post useful! You can comment here and check my YouTube channel.

A fast, persistent Meshtastic web app - part 2

Solve Computer Science — Mon, 07 Jul 2025 17:30:00 +0000

🤔 Intro

Let's continue our journey with Meshtastic by showing you an update of fastmeshapi, a FastAPI-based Meshtastic dashboard.

Each Meshtastic-capable board has its unique characteristics. Since their number is growing, I thought showing some important information directly on the dashboard could be beneficial.

As always, the YouTube video is at the bottom of this post.

📋 Fields

Some notable fields include the name of the board, the Meshtastic support status, hardware revisions, available radio bands, antenna type, chipset, GPIO pins, ports and interfaces.

Heltec Wireless Paper modal

🖥️ Backend

🗒️ REST API

/v0/boards endpoint

All the board data is fetched from the /v0/boards endpoint. At the moment, I only added information for a couple of common Heltec boards I own, the v3 and the Wireless Paper. Finding their PDF datasheet was quite easy and it's also very detailed.

🧮 YAML data and classes

Heltec v3 modal

All the raw board data you see in this picture is stored in a YAML file. I decided to avoid designing database tables for this scope since all the data is static and limited in size. Modifying the Pydantic classes and the frontend should only be done if there is a real necessity to show new and/or different fields.

Here's a small extract of the YAML file:

boards:
  - name: 'HELTEC_V3'
    note: "The beginner's device"
    links:
      - homepage: null
      - datasheet: 'https://resource.heltec.cn/download/WiFi_LoRa_32_V3/HTIT-WB32LA_V3.2.pdf'
      - hardware_changelog: null
    hardware_revisions:
      - 'V3.1'
      - 'V3.2'
    supported_frequencies_mhz:
      - 470.0
      - 510.0
      - 868.0
      - 915.0
      - 920.0
      - 923.0
    screen:
      type: 'oled'
      size_mm:
        # Approximate.
        base: 27
        height: 18
      size_inch: 0.95
      resolution: '128x64'
    buttons:
      - 'prg'
      - 'rst'

The data is readable and can be easily extended.

Pydantic classes store the board data so that FastAPI can serve it. This information is at first loaded from the YAML file into dictionaries. This is done using the famous PyYAML library.

Pydantic makes field validation available by default, which is very important for FastAPI as well, because we can easily double check if we need to fix how the frontend consumes this data.

Let's see a very small extract of these classes:

# Lots of other stuff here...

class BoardPowerConsumption(BaseModel):
    amount_volts: float
    lora: Optional[list[BoardPowerConsumptionLora]]
    bluetooth_amount: float
    wifi: Optional[BoardPowerConsumptionWifi]
    sleep_amount: float
    deep_sleep_amount: float

# Lots of other stuff here...

class Board(BaseModel):
    name: str
    note: Optional[str]
    links: Optional[
        list[
            dict[
                str, Optional[HttpUrl]
            ]
        ]
    ]
    hardware_revisions: list[str]
    supported_frequencies_mhz: list[float]
    power_consumption_mah: list[BoardPowerConsumption]

    # Lots of other stuff here...

    @validator('buttons', each_item=True)                                       
    def validate_buttons(cls, v):                                               
        if not re.match(r'[a-z]{3,}', v):                                       
            raise ValueError('Each board button identifier must be of at least length 3, and consist of lowercase letters only.')
        return v

Besides the nested classes we can also see a custom validator for the board button names. We can enforce the rules we want, and, in this specific case, a button name must be a 3 letter lowercase string.

As you see, at the YAML and Pydantic class level there is support for lots of fields which are not shown on the dashboard.

🏢 Frontend

As I anticipated in the first post, the frontend of this app is written in Svelte.

✨ Icons

Sveltestrap icons

Sveltestrap has a nice vector icon set. Some important icons I would have liked to use in this project are not available. Anyway I figured out alternatives which convey the same meanings. These same icons are also part of the standard Bootstrap package.

Svelte icons

🪟 Modals

An example modal

By using Sveltestrap not only you are provided by convinient icons but also by common patterns such as Modal, which are these small windows drawn on top of the rest of the content.

To display the fields in a readable manner I used a Table element.

Icon data is stored in objects: if a board feature is available, its value field will be set to true (false otherwise). The tooltip field is the text displayed on the icon during the mouse hover action. Finally, the key of each object is the exact name of the icon in the Sveltestrap icon set. So, to organize the interface data, I did like this:

let boardDetailDataInterfaces= {};
function showDetailsBoardInterfaces(interfaces) {
    boardDetailDataInterfaces = {
        // icon_name: {tooltip_text, value}
        'arrow-left-right': {
            tooltip: 'Serial',
            value: interfaces?.serial ? true : false
        },
        'wifi': {
            tooltip: 'WiFi',
            value: interfaces?.wifi ? true : false
        },
        'bluetooth': {
            tooltip: 'Bluetooth',
            value: interfaces?.bluetooth ? true : false
        },
        'map': {
            tootlip: 'GPS',
            value: interfaces?.gps ? (interfaces.gps.available ? true : false) : false
        }
    }
}

Symbol used to represent the serial interface

So, to be able to actually display the icons in the Svelte HTML section we need to unpack the object into key and value, and create a div, Icon, and Tooltip.

{#each Object.entries(boardDetailDataInterfaces) as [key, value]}
    {#if value.value}
        <div id="{key}" class="icon-tooltip"><Icon style="font-size: 2rem;" name="{key}"/><Tooltip target="{key}">{value.tooltip}</Tooltip></div>
    {/if}
{/each}

Modal icon tootlip hover action

I later used this same Tooltip pattern for other elements in the project.

🎉 Conclusion

The purpose is to build a persistent, high performance Meshtastic web app that provides a REST API as well.

Subscribe on YouTube

Let me know in the comments if you already know Meshtastic or if you'd like to try it in the future.

Ollama CPU: Model Manager Script & Inference with a Terminal UI

Solve Computer Science — Thu, 03 Jul 2025 22:00:00 +0000

DuckDuckGo AI Chat allows you to talk with AI models, such as ChatGPT, however it recently started to put some IP address limits on the number of daily interactions. This prompted me to research other local AI models to run with Ollama. The ones which work better up until now (CPU only) are the 14 billion params Qwen 2.5 coder models. I specifically tried the ones with the least quantization. Smaller models with less parameters, such as the 1.5b ones replied me in Chinese, so they are not usable.

To improve inference quality I wrote a custom Modelfile, specifically for coding assistance in Python, FastAPI and Svelte. These are by far the best Ollama models I tried.

I’ve written a custom whiptail script to simplify Ollama administration via docker-compose. I can now upgrade and list all models, add (customize) new ones, as well ad simply downloading and deleting existing ones, all via a simple keyboard interface.

Instead of relying on Alpaca, this time I used a terminal UI called tgpt. The software is similar to ShellGPT, but it seems to work better: it’s fast and reliable, plus, the multi-line input is really convenient. You can use tgpt with other AI systems besides Ollama.

All the links to the codes and resources are in the video description as usual.

Note: post auto-generated from YouTube feeds.

YouTube channel mirror on Jekyll - part 3

Solve Computer Science — Thu, 03 Jul 2025 17:00:00 +0000

🧩 The problem

As promised in the previous post let's add support for reading RSS feeds from YouTube, so we can download the most recent videos which are still not present locally. This way we'll avoid changing the ones already downloaded.

It's pretty straight forward if we follow what has been done in the first post of this series. In-fact, all the feed reading code remains the same. What changes, is that we now have to compare video IDs instead of markdown file contents.

⚠️ Warning

⚠️⚠️ Before continuing, please only mirror content you have permission to... ⚠️⚠️

✅ The solution

Before starting, let's not forget the Python requirements.txt file:

yt-dlp
feedparser>=6,<7

We only need these two dependencies for the moment.

🆔 YouTube video ID extraction

This first function can be imported verbatim from the original script. What it does is very simple: given a YouTube video URL it returns a video id. For example:

get_youtube_video_id('https://www.youtube.com/watch?v=PwEMWnuMdac')

returns

PwEMWnuMdac

Here it is:

# Import this from the generate_posts.py script.
def get_youtube_video_id(video_url: str) -> str:                                
    parsed_url = urlparse(video_url)

    # Check for the path and query parameters.
    if 'youtube.com' in parsed_url.netloc:
        # Standard YouTube URLs.
        if parsed_url.path == '/watch':
            # parse_qs gets a dict.
            query_params = parse_qs(parsed_url.query)
            return query_params.get('v', [None])[0]
        elif parsed_url.path.startswith('/shorts'):
            return pathlib.Path(parsed_url.path).stem

    return None

📶 YouTube URL feeds

The feeds function has been simplified a lot because we only need the video URLs. All the other fields will not be used.

def extract_feed_youtube_urls(feed_source: str) -> list[str]:
    data = feedparser.parse(feed_source)
    return [d['link'] for d in data.entries if 'link' in d]

Remember the power of list comprehensions!

📽️ Filter Missing Videos

As described in the introduction, we need to know exactly which videos need to be downloaded, and avoiding re-iterating the ones already available locally. The following filter function does just that. Remember the directory structure:

videos
├── -9Tp3rB-5n0
│   ├── -9Tp3rB-5n0.en.vtt
│   ├── -9Tp3rB-5n0.it.vtt
│   ├── -9Tp3rB-5n0.png
│   ├── -9Tp3rB-5n0.webm
│   ├── description.txt
│   └── title.txt
├── 02RSfvOvb3g
│   ├── 02RSfvOvb3g.en.vtt
│   ├── 02RSfvOvb3g.it.vtt
│   ├── 02RSfvOvb3g.png
│   ├── 02RSfvOvb3g.webm
│   ├── description.txt
│   └── title.txt
[...]

This means that each video is inside a directory having the video ID as its name. The set data structure comes in very handy here:

def filter_missing_videos(dst_dir: str, urls: list[str]) -> list[str]:
    # 1. Get all directories at level 0 (root) that have
    # compatible YouTube video ids (length 11).
    directory: pathlib.Path = pathlib.Path(dst_dir)
    directory_video_ids: list[str] = ([
        d.stem for d in directory.rglob('*')
            if (d.is_dir()
             and str(d.parent) == dst_dir
             and len(d.stem) == YOUTUBE_VIDEO_ID_LENGTH)])

    # 2. Get YouTube URL video IDs.
    url_video_ids: list[str] = [get_youtube_video_id(url) for url in urls]

    # 3. Get all video IDs present on YouTube and not present locally.
    ids_to_download: list[str] = (
        list(set(url_video_ids) - set(directory_video_ids)))

    # 4. Rebuild original YouTube URLs.
    return ['https://www.youtube.com/watch?v=' + u for u in ids_to_download]

🧮 Main function

Finally, we have to glue every bit and piece.

To be able to optionally download, or re-sync the whole channel, just in case, as well as to download all videos in the RSS feed and avoid the previous filtering, I used argparse and some simple logic:

if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)

    parser = argparse.ArgumentParser(
        description='Mirror YouTube channel locally.'
    )
    subparsers = parser.add_subparsers(
        dest='command',
        required=True
    )

    download_parser = subparsers.add_parser(
        'download',
        help='Download all videos, rsync style'
    )

    download_group = download_parser.add_mutually_exclusive_group()
    download_group.add_argument(
        '--ignore-existing',
        action='store_true',
        help='Download all videos listed in the RSS feed'
    )
    download_group.add_argument(
       '--whole-channel',
       action='store_true',
       help='Synchronize the whole YouTube channel. Use this to update titles and descriptions'
    )

    args = parser.parse_args()

    urls: list[str]

    if args.ignore_existing:
        urls = extract_feed_youtube_urls(FEED_URL)
    elif args.whole_channel:
        urls = [CHANNEL_URL]
    else:
        urls = extract_feed_youtube_urls(FEED_URL)
        urls = filter_missing_videos(DST_DIR, urls)

    logging.info(f'URLs to download: {urls}')

    if len(urls) > 0:
        with YoutubeDL(ydl_opts) as ydl:
            ydl.download(urls)
    else:
        logging.info('no videos to download')

🤖 Systemd unit files

In the service unit file we simply need to call the script:

[Unit]
Description=mirror-yt SolveComputerScience
Requires=network-online.target
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/home/myuser/.scripts
ExecStart=/bin/sh -c '. .venv/bin/activate && python -m mirror_yt download; deactivate'
User=myuser
Group=myuser

And of course you can run it daily, for example:

[Unit]
Description=Once every day mirror-yt SolveComputerScience

[Timer]
OnCalendar=Daily
Persistent=true

[Install]
WantedBy=timers.target

🎉 Conclusion

Click to open the full script

import argparse
import logging
import pathlib
import sys
from urllib.parse import urlparse, parse_qs

from yt_dlp import YoutubeDL
import feedparser


DST_DIR: str = '/srv/http/videos'
FEED_URL: str = 'https://www.youtube.com/feeds/videos.xml?channel_id=UC2rr0LbIuy34JHEoCndmKiA'
CHANNEL_URL: str = 'https://www.youtube.com/channel/UC2rr0LbIuy34JHEoCndmKiA'
YOUTUBE_VIDEO_ID_LENGTH: int = 11


ydl_opts: dict = {
    'verbose':             True,
    'no_overwrites':       True,
    'call_home':           False,
    'add_metadata':        True,
    'fixup':               'detect_or_warn',
    'prefer_ffmpeg':       True,
    'subtitleslangs':      ['en', 'it'],
    'writesubtitles':      True,
    'writeautomaticsub':   True,
    'prefer_free_formats': True,
    'writethumbnail':      True,
    'final_ext':           'webm',
    'outtmpl': {
        'default': str(pathlib.Path(DST_DIR,'%(id)s','%(id)s.%(ext)s'))
    },
    'postprocessors': [
        {
            # --convert-thumbnails png
            'format': 'png',
            'key': 'FFmpegThumbnailsConvertor',
            'when': 'before_dl'
        },
        {
            # --recode webm
        'key': 'FFmpegVideoConvertor',
            'preferedformat': 'webm'
        },
        {
            'exec_cmd': ["cat > " + str(pathlib.Path(DST_DIR, '%(id)s', 'title.txt')) + " << 'EOF'\n"
                         '%(title)s\n'
                         'EOF'],
            'key': 'Exec',
            'when': 'after_move'
        },
        {
            'exec_cmd': ["cat > " + str(pathlib.Path(DST_DIR, '%(id)s', 'description.txt')) + " << 'EOF'\n"
                         '%(description)s\n'
                         'EOF'],
            'key': 'Exec',
            'when': 'after_move'
        },
    ],
}


def extract_feed_youtube_urls(feed_source: str) -> list[str]:
    data = feedparser.parse(feed_source)
    return [d['link'] for d in data.entries if 'link' in d]


# Import this from the generate_posts.py script.
def get_youtube_video_id(video_url: str) -> str:                                
    parsed_url = urlparse(video_url)

    # Check for the path and query parameters.
    if 'youtube.com' in parsed_url.netloc:
        # Standard YouTube URLs.
        if parsed_url.path == '/watch':
            # parse_qs gets a dict.
            query_params = parse_qs(parsed_url.query)
            return query_params.get('v', [None])[0]
        elif parsed_url.path.startswith('/shorts'):
            return pathlib.Path(parsed_url.path).stem

    return None


def filter_missing_videos(dst_dir: str, urls: list[str]) -> list[str]:
    # 1. Get all directories at level 0 (root) that have
    # compatible YouTube video ids (length 11).
    directory: pathlib.Path = pathlib.Path(dst_dir)
    directory_video_ids: list[str] = ([
        d.stem for d in directory.rglob('*')
            if (d.is_dir()
             and str(d.parent) == dst_dir
             and len(d.stem) == YOUTUBE_VIDEO_ID_LENGTH)])

    # 2. Get YouTube URL video IDs.
    url_video_ids: list[str] = [get_youtube_video_id(url) for url in urls]

    # 3. Get all video IDs present on YouTube and not present locally.
    ids_to_download: list[str] = list(set(url_video_ids) - set(directory_video_ids))

    # 4. Rebuild original YouTube URLs.
    return ['https://www.youtube.com/watch?v=' + u for u in ids_to_download]


if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)

    parser = argparse.ArgumentParser(description='Mirror YouTube channel locally.')
    subparsers = parser.add_subparsers(dest='command', required=True)

    download_parser = subparsers.add_parser('download', help='Download all videos, rsync style')

    download_group = download_parser.add_mutually_exclusive_group()
    download_group.add_argument('--ignore-existing', action='store_true', help='Download all videos listed in the RSS feed')
    download_group.add_argument('--whole-channel', action='store_true', help='Synchronize the whole YouTube channel. Use this to update titles and descriptions')

    args = parser.parse_args()

    urls: list[str]

    if args.ignore_existing:
        urls = extract_feed_youtube_urls(FEED_URL)
    elif args.whole_channel:
        urls = [CHANNEL_URL]
    else:
        urls = extract_feed_youtube_urls(FEED_URL)
        urls = filter_missing_videos(DST_DIR, urls)

    logging.info(f'URLs to download: {urls}')

    if len(urls) > 0:
        with YoutubeDL(ydl_opts) as ydl:
            ydl.download(urls)
    else:
        logging.info('no videos to download')

In the next post we'll start to automate the Jekyll side of things.

You can comment here and check my YouTube channel.

Is GitHub a single point of failure?

Solve Computer Science — Tue, 24 Jun 2025 17:00:00 +0000

🧩 The problem

The title may seem provocative, but think about it for a moment: lots of open source projects host their Git repositories exclusively on GitHub. What happens when the website is down or experiences other issues? Here are just a few examples:

You can find older news articles such as the famous 2018 DDoS attack.

In this post I show you how to use multiple Git remotes to increase "Git hosting resilience".

The way you set up mirrors can change between services (web UI), while using remotes is platform-agnostic (Git CLI). For this reason we'll avoid Git mirrors in this discussion.

📓 Software and services

Notes:

The order of items in the tables is alphabetical
The choices I list here are just a tiny subset of the ones available

🌐 Services

Service name	Is open source	Notes
bitbucket.org	No	-
codeberg.org	Yes	powered by Forgejo
framagit.org	Yes	powered by Gitlab
gitlab.com	No (only the community edition is fully libre)	powered by Gitlab, probably the most direct competitor of GitHub
sr.ht	Yes	powered by Sourcehut

🖥️ Software

These are all open source and self-hostable.

Software name	Notes
Forgejo	Fork of Gitea, maintained by Codeberg e.V. and contributors. Powers Codeberg
Gitea	Fork of Gogs
GitLab	Use the community edition
Gogs	-
Sourcehut	Powers sr.ht

🤔 So, how do you do it

Given the distributed nature of Git you can keep GitHub as main and then add one or more of these services as non-default remotes.

First of all check the existing remotes:

git remote -v

By today's standards it's expected not to use HTTP basic auth with Git anymore, but something like SSH, possibly with unique encrypted keys for each service. For example, in my local md-toc project checkout I have these remotes:

codeberg    git@codeberg.org:frnmst/md-toc.git (fetch)
codeberg    git@codeberg.org:frnmst/md-toc.git (push)
framagit    git@framagit.org:frnmst/md-toc.git (fetch)
framagit    git@framagit.org:frnmst/md-toc.git (push)
origin  ssh://git@github.com/frnmst/md-toc.git (fetch)
origin  ssh://git@github.com/frnmst/md-toc.git (push)
forgejo ssh://git@software.franco.net.eu.org/frnmst/md-toc.git (fetch)
forgejo ssh://git@software.franco.net.eu.org/frnmst/md-toc.git (push)

To add a new remote such as Codeberg, what I did at the time was simply:

git remote add codeberg git@codeberg.org:frnmst/md-toc.git

This method, however, has the disadvantage that you need to push changes one remote at a time. Alternatively, you can create a new all label and combine multiple push remotes into one. This way there is no need to manually git push each remote one by one. You can simply git push all.

Here's how you do it:

git remote add all ssh://git@github.com/frnmst/md-toc.git
git remote set-url --add --push all ssh://git@github.com/frnmst/md-toc.git
git remote set-url --add --push all ssh://git@software.franco.net.eu.org/frnmst/md-toc.git
git remote set-url --add --push all git@codeberg.org:frnmst/md-toc.git
git remote set-url --add --push all git@framagit.org:frnmst/md-toc.git

So we now have a new virtual all remote we can push to but a single pull remote from GitHub. Having multiple Git pull remotes could cause conflicts if all these don't have the same history. If there is a change in a single remote you can pull the changes directly from it and merge when necessary.

You can check the result of the previous commands like this:

git remote -v | grep all

all ssh://git@github.com/frnmst/md-toc.git (fetch)
all     ssh://git@github.com/frnmst/md-toc.git (push)
all ssh://git@software.franco.net.eu.org/frnmst/md-toc.git (push)
all git@codeberg.org:frnmst/md-toc.git (push)
all git@framagit.org:frnmst/md-toc.git (push)

Let's test the setup. In my case I didn't have any changes so I got the same message 4 times:

git push all dev --verbose

Pushing to ssh://github.com/frnmst/md-toc.git
To ssh://github.com/frnmst/md-toc.git
 = [up to date]      dev -> dev
updating local tracking ref 'refs/remotes/all/dev'
Everything up-to-date
Pushing to ssh://software.franco.net.eu.org/frnmst/md-toc.git
To ssh://software.franco.net.eu.org/frnmst/md-toc.git
 = [up to date]      dev -> dev
updating local tracking ref 'refs/remotes/all/dev'
Everything up-to-date
Pushing to codeberg.org:frnmst/md-toc.git
To codeberg.org:frnmst/md-toc.git
 = [up to date]      dev -> dev
updating local tracking ref 'refs/remotes/all/dev'
Everything up-to-date
Pushing to framagit.org:frnmst/md-toc.git
To framagit.org:frnmst/md-toc.git
 = [up to date]      dev -> dev
updating local tracking ref 'refs/remotes/all/dev'
Everything up-to-date

🎉 Conclusion

Git has tons of options and there's an ecosystem of web services and libre software you can choose from to host your repository. Don't limit yourself to exclusively host on GitHub. The real challenge is how to sync stuff like issues, pull requests, projects, actions, etc... between all these platforms. Alternative systems like Fossil could help solve this.

Do you already use multiple remotes? If not, which ones do you intend to try? Are there easier ways to manage this use case (I'm not a Git expert)? Let me know in the comments.

A fast, persistent Meshtastic web app - part 1

Solve Computer Science — Fri, 20 Jun 2025 17:00:00 +0000

🤔 Intro

I love picking up new things and using what I learn to solve real-world problems. If you don't know Meshtastic, it is a LoRa (Long Range) radio messaging system that uses inexpensive boards (ESP32, nRF52, etc...). It's all license free so you don't have to be ham radio operator to use it. Messages are all packet-based and can travel several km (even hundreds) depending on antennas, terrain conditions, etc... all with tiny output power involved.

Note: the picture is directly taken from their docs, GPLv3 license

Like the name implies, the network created is a mesh type. Each element in the network is called a node, and what they can also do is to relay messages, i.e. re-transmit them. This extends the packet range a lot. I think this picture will give you a good idea on how it works:

Note: the picture is directly taken from their docs, GPLv3 license

There are public and private channels determined by crypto keys.

Lots of other things could be said about Meshtastic but I want to concentrate this post on what the title states.

📱 Apps

Meshtastic has an Android and iOS app, a web UI, APIs, etc. The Android app works quite well. The web UI is different: it's ephemeral, which means that if you refresh the page you are going to lose the messages which have already been downloaded from the board. The boards, in-fact, can only store a certain number of messages due to their limited memory and once the limit is reached it's overwritten. I like to think of this as a circular array.

Anyway, I have to say this app is improving lately and there are alternatives, but I wanted to learn more about Meshtastic, FastAPI, Svelte and SQLModel, so I am trying to implement one myself.

🖥️ fastmeshapi

fastmeshapi is a project that involves several components:

Dependency name	Notes
Meshtastic	has a Python API
FastAPI	it is the core component
SQLModel	a database ORM based on SQLAlchemy
Svelte	provides the JavaScript frontend
SvelteStrap	provides Bootstrap elements as convenient Svelte components

There are also other minor dependencies which I'm importing along the way but these are the most important ones.

A subset of the API endpoints. There are 43 at the moment of writing this post.

The purpose is to build a persistent, high performance Meshtastic web app that provides a REST API as well.

This first video shows you the initial version of the dashboard and some of the FastAPI endpoints.

Subscribe on YouTube

Let me know in the comments if you already know Meshtastic or if you'd like to try it in the future.

YouTube channel mirror on Jekyll - part 2

Solve Computer Science — Thu, 19 Jun 2025 17:30:00 +0000

🧩 The problem

Continuing the previous post we'll now transform the shell script into a Python one. We want an integration with Python to pull feeds from YouTube directly, so we must, first of all, transform the original instructions into Python API calls. This will enable yt-dlp to directly download the necessary videos avoiding the whole channel every time, which might result in IP throttling or temporary bans.

yt-dlp is also capable of working with text files in two ways:

to know which files have already been downloaded by saving their ids
to store the options instead of manually passing them via the shell or API

We're not going to use these two features today.

⚠️ Warning

⚠️⚠️ Before continuing, please only mirror content you have permission to... ⚠️⚠️

✅ The solution

🤖 Vibe coding

I'm lazy so I asked GPT o4-mini (via Duck.ai) how to do it. Here I redacted some output so it's more straight forward to read.

original shell command

yt-dlp "${URL}" \
    --verbose \
    --fixup detect_or_warn \
    --prefer-ffmpeg \
    --sub-langs "en,it" \
    --write-subs \
    --embed-subs \
    --write-auto-sub \
    --prefer-free-formats \
    --no-call-home \
    --no-overwrites \
    --recode webm \
    --add-metadata \
    --write-thumbnail \
    --convert-thumbnails png \
    --exec "echo \"%(title)s\" > ${DST_DIR}/%(id)s/title.txt" \
    --exec "echo \"%(description)s\" > ${DST_DIR}/%(id)s/description.txt" \
    --output "${DST_DIR}/%(id)s/%(id)s.%(ext)s"

Starting from this command, and setting URL='my youtube channel url', please transform the shell command to the Python calls

o4-mini

Here’s how you can do the same download in Python using yt-dlp’s API. You define essentially the same options in a dict, register a small progress hook to emit your “exec” behavior, and then call .download().

import os
from yt_dlp import YoutubeDL

# ========== configuration ==========

URL     = 'https://www.youtube.com/c/YourChannelNameHere'
DST_DIR = '/path/to/your/download/folder'

# Make sure the DST_DIR exists
os.makedirs(DST_DIR, exist_ok=True)

# ========== progress hook to write title & description ==========
def write_meta_hook(d):
    """
    This hook is called multiple times; we care about when download
    of a single video is finished so we can write title/description.
    """
    if d['status'] == 'finished':
        info = d.get('info_dict', {})
        video_id    = info.get('id')
        title       = info.get('title', '')
        description = info.get('description', '')

        # Create a per-video subfolder, mirror --output structure
        vid_dir = os.path.join(DST_DIR, video_id)
        os.makedirs(vid_dir, exist_ok=True)

        # write out title.txt and description.txt
        with open(os.path.join(vid_dir, 'title.txt'), 'w', encoding='utf-8') as f:
            f.write(title)
        with open(os.path.join(vid_dir, 'description.txt'), 'w', encoding='utf-8') as f:
            f.write(description)

# ========== yt-dlp options ==========
ydl_opts = {
    # general
    'verbose':             True,
    'no_overwrites':       True,
    'nocallhome':          True,
    'add_metadata':        True,
    'fixup':               'detect_or_warn',
    'prefer_ffmpeg':       True,
    'recode_video':        'webm',

    # subtitles
    'sub_langs':           ['en', 'it'],
    'writesubtitles':      True,   # --write-subs
    'writeautomaticsub':   True,   # --write-auto-sub
    'embedsubtitles':      True,

    # formats
    'prefer_free_formats': True,

    # thumbnails
    'writethumbnail':      True,
    'convert_thumbnails':  'png',

    # output template
    'outtmpl': os.path.join(DST_DIR, '%(id)s', '%(id)s.%(ext)s'),

    # post hooks
    'progress_hooks': [write_meta_hook],
}

# ========== run the download ==========
with YoutubeDL(ydl_opts) as ydl:
    ydl.download([URL])

🔎 Clues

I then go and double check if all the options in the generated code actually exist. Surprise! They do not.

By opening the yt-dlp source code, specifically the YoutubeDL class, you'll see all the supported API options.

🆘 Trying to help the bot help me

Me

i'll tell you the missing options from the yt-dlp code. you need to fix them:

'recode_video', 'embedsubtitles', 'convert_thumbnails',

o4-mini

The three fixed option names are:

recode_video: your --recode webm remux into WebM

embed_subtitles: your --embed-subs

convert_thumbnails: your --convert-thumbnails png

Turns out these fixes are wrong.

😌 Script that saved the day

There's one interesting utility in the yt-dlp repository called cli_to_api.py which is able to translate shell options into Python ones. One of the maintainers suggests to use that script for this use case. Here's how to run it:

clone the repository

  git clone https://github.com/yt-dlp/yt-dlp

install locally

  cd yt-dlp
  python3 -m venv .venv
  . .venv/bin/activate
  pip install .

run the script

  cd devscripts
  python -m cli_to_api "--verbose"

This is the output for the --verbose option

The arguments passed translate to:

[debug] Override config: ['--verbose']
{'verbose': True}

Combining these with the CLI defaults gives:

[debug] Override config: ['--verbose']
{'extract_flat': 'discard_in_playlist',
 'fragment_retries': 10,
 'ignoreerrors': 'only_download',
 'postprocessors': [{'key': 'FFmpegConcat',
                     'only_multi_video': True,
                     'when': 'playlist'}],
 'retries': 10,
 'verbose': True}

You can of course pass multiple options, each one between quotes:

python -m cli_to_api "--output" "DST_DIR/%(id)s/%(id)s.%(ext)s"

Which results in:

The arguments passed translate to:

{'outtmpl': {'default': 'DST_DIR/%(id)s/%(id)s.%(ext)s'}}

Combining these with the CLI defaults gives:

{'extract_flat': 'discard_in_playlist',
 'fragment_retries': 10,
 'ignoreerrors': 'only_download',
 'outtmpl': {'default': 'DST_DIR/%(id)s/%(id)s.%(ext)s'},
 'postprocessors': [{'key': 'FFmpegConcat',
                     'only_multi_video': True,
                     'when': 'playlist'}],
 'retries': 10}

🎯 Result

The write_meta_hook provided by GPT seems correct enough. However, to keep things simple I decided to translate the original shell exec options verbatim. I also like pathlib more than os to manage paths.

Here's the complete result:

import pathlib
import sys
from yt_dlp import YoutubeDL

DST_DIR = '/srv/http/videos'

ydl_opts: dict = {
    'verbose':             True,
    'no_overwrites':       True,
    'call_home':           False,
    'add_metadata':        True,
    'fixup':               'detect_or_warn',
    'prefer_ffmpeg':       True,
    'subtitleslangs':      ['en', 'it'],
    'writesubtitles':      True,
    'writeautomaticsub':   True,
    'prefer_free_formats': True,
    'writethumbnail':      True,
    'final_ext':           'webm',
    'outtmpl': {
        'default': str(pathlib.Path(DST_DIR,'%(id)s','%(id)s.%(ext)s'))
    },
    'postprocessors': [
        {
            # --convert-thumbnails png
            'format': 'png',
            'key': 'FFmpegThumbnailsConvertor',
            'when': 'before_dl'
        },
        {
            # --recode webm
        'key': 'FFmpegVideoConvertor',
            'preferedformat': 'webm'
        },
        {
            'exec_cmd': ["cat > " + str(pathlib.Path(DST_DIR, '%(id)s', 'title.txt')) + " << 'EOF'\n"
                         '%(title)s\n'
                         'EOF'],
            'key': 'Exec',
            'when': 'after_move'
        },
        {
            'exec_cmd': ["cat > " + str(pathlib.Path(DST_DIR, '%(id)s', 'description.txt')) + " << 'EOF'\n"
                         '%(description)s\n'
                         'EOF'],
            'key': 'Exec',
            'when': 'after_move'
        },
    ],
}

if __name__ == '__main__':
    # RSS feed code here: TODO
    # This will replace the single URL passed to the `download` method.

    url: str = sys.argv[1]

    with YoutubeDL(ydl_opts) as ydl:
        ydl.download([url])

The script works exactly the same as the one using the shell:

python -m mirror_yt "video or channel URL"

🎉 Conclusion

As you read, this is yet another evidence that vibe coding does not always work 100%.

Next time we'll integrate the YouTube RSS feeds into the script like I did in the first post of this series.

You can comment here and check my YouTube channel.

YouTube channel mirror on Jekyll

Solve Computer Science — Wed, 18 Jun 2025 18:00:00 +0000

🧩 The problem

Having a YouTube mirror can be beneficial in case something happens to your main channel. I have mirrors of my Git repositories, and self-host other stuff as well. So, why not do this for video content?

In this post I will show you how to create a simple YouTube mirror you can host through Jekyll and a standard web server.

No, I'm not going to talk about Peertube, although that can also be an alternative. My solution is still centralized and dumb, but it works. The advantage is that it runs on a static site, so no database or maintenance is involved.

⚠️ Warning

⚠️⚠️ Before continuing, please only mirror content you have permission to! ⚠️⚠️

✅ The solution

yt-dlp is of course the tool of choice for this job. It's incredibly powerful because of the hundreds of options available. This first version is a shell script that downloads the following elements:

File	Type	Notes
video	WebM	format widely used and supported by all major browsers
thumbnail	PNG	we need this as video cover
subtitles	VTT & embedded in the video	just in case
title	TXT	debug & will be useful in the next iterations
description	TXT	might be useful in the future

🖥️ Script

First of all create a Python virtual environment:

python3 -m venv .venv

This is the script:

#!/usr/bin/env bash

set -euo pipefail

URL="${1}"
DST_DIR='/srv/http/videos'

# YouTube changes frequently, we keep the program always up to date like this.
. .venv/bin/activate
pip install -U yt-dlp

pushd "${DST_DIR}"

# Reason to recode to webm:
# https://github.com/yt-dlp/yt-dlp/issues/775#issuecomment-904271715

yt-dlp "${URL}" \
    --verbose \
    --fixup detect_or_warn \
    --prefer-ffmpeg \
    --sub-langs "en,it" \
    --write-subs \
    --embed-subs \
    --write-auto-sub \
    --prefer-free-formats \
    --no-call-home \
    --no-overwrites \
    --recode webm \
    --add-metadata \
    --write-thumbnail \
    --convert-thumbnails png \
    --exec "cat > ${DST_DIR}/%(id)s/title.txt << 'EOF'
%(title)s
EOF" \
    --exec "cat > ${DST_DIR}/%(id)s/description.txt << 'EOF'
%(description)s
EOF" \
    --output "${DST_DIR}/%(id)s/%(id)s.%(ext)s"

popd

Special care must be taken in correctly quoting the title and description inputs! This is very tricky!!! Alternatively you can use "here-documents" as I did in the example.

You can pass a single video URL or the channel URL as input:

./mirror_youtube.sh "video or channel URL"

A canonical video URL on YouTube might be:

https://www.youtube.com/watch?v=aaa000

In this example the video ID is aaa000. This means the downloaded files will be placed inside /srv/http/videos/aaa000. Specifically the video, thumbnail and other elements will be named:

/srv/http/videos/aaa000/aaa000.webm
/srv/http/videos/aaa000/aaa000.png
/srv/http/videos/aaa000/aaa000.en.vtt
/srv/http/videos/aaa000/aaa000.it.vtt
/srv/http/videos/aaa000/description.txt
/srv/http/videos/aaa000/title.txt

respectively. Titles and video descriptions are saved via separate spawned commands (exec).

If we keep this naming consistent we can create Jekyll pages just using a layout and some front matter later on.

🕸️ Serving the files

For the moment I avoid tracking the YouTube files in my blog repository. I think it's unnecessary and would create a huge repo even if using Git LFS.

I simply serve these files on a separate domain using normal web server. This is an example Apache configuration:

<IfModule mod_ssl.c>                                    
<VirtualHost *:443>

    UseCanonicalName on

    Keepalive On
    RewriteEngine on

    ServerName your.video.assets.domain

    Alias /icons/ "/usr/share/apache2/icons/"

    DocumentRoot "/srv/http/videos"
    <Directory "/srv/http/videos">
        Options -ExecCGI -Includes                 
        Options +Indexes +SymlinksIfOwnerMatch
        IndexOptions NameWidth=* +SuppressDescription FancyIndexing Charset=UTF-8 VersionSort FoldersFirst

        ReadmeName footer.html
        IndexIgnore header.html footer.html   
        #                                          
        # AllowOverride controls what directives may be placed in .htaccess files.
        # It can be "All", "None", or any combination of the keywords:
        #   AllowOverride FileInfo AuthConfig Limit               
        #
        AllowOverride All

        #
        # Controls who can get stuff from this server.
        #
        Require all granted
    </Directory>

    SSLCompression      off

    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/your.video.assets.domain/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/your.video.assets.domain/privkey.pem
</VirtualHost>
</IfModule>

I prefer directory listings enabled so it acts as a sort of debug output.

📄 Jekyll

Let's see the Jekyll HTML layout now:

---
layout: page
---
{% if page.comment %}
{{ page.comment }}
{% endif %}

{% if page.embed_html %}
{{ page.embed_html }}
{% endif %}

{% if page.backup_url != '' %}
{% if page.embed_html %}
<em>Backup</em>
{% endif %}
<video
   controls
   width="560"
   height="315"
   preload="none"
   {% if page.poster_image %}poster="{{ page.poster_image }}"{% endif %}>
    <source src="{{ page.backup_url }}" type="video/webm" />
    <p>Your browser doesn't support HTML videos. Here is a [link]({{ page.backup_url }}) instead.</p>
</video>
{% endif %}

<p><a href="{{ page.canonical_url }}">Original video on YouTube</a></p>

What all this means is that we're embedding a video using plain HTML5 elements. The thumbnail is used as well, like on YouTube. This example file, called 7Rh8hWMmZeA.md, might help you understand better:

---                                                                             
layout: youtube_mirror                                                          
title: ! 'fastmeshapi: a fast, persistent Meshtastic web app (part 3 - packet table)'
permalink: /courses/solvecomputerscience-youtube-channel/7Rh8hWMmZeA/           
description: ! 'fastmeshapi: a fast, persistent Meshtastic web app (part 3 - packet table)'
enable_markdown: true                                                           
lang: 'en'                                                                      
backup_url: ! 'https://your.video.assets.domain/video/SolveComputerScience/7Rh8hWMmZeA/7Rh8hWMmZeA.webm'
poster_image: ! 'https://your.video.assets.domain/video/SolveComputerScience/7Rh8hWMmZeA/7Rh8hWMmZeA.png'
is_youtube_mirror: true                                                         
canonical_url: ! 'https://www.youtube.com/watch?v=7Rh8hWMmZeA'                  
---

The canonical URL pointing to YouTube is very important to avoid the page being marked as possible duplicate content by search engines.

🎯 Result

The whole system works, but at the moment the markdown files need to be created manually. The video download script can be managed by a Systemd timer to trigger once a day for example.

Incidentally, if you run the script multiple times and you change the title or descriptions in the meantime, these will be automatically updated.

🚀 Future updates

In the next posts I will show you how to improve automation for this use case. We'll transform the shell script into Python and use the YouTube feeds.

🎉 Conclusion

If you are interested in the source code you can find it on its repo.

You can comment here and check my YouTube channel.

Jekyll auto posts from YouTube feeds

Solve Computer Science — Tue, 17 Jun 2025 17:00:00 +0000

🧩 The problem

I wanted to automate this boring and repetitive workflow: my idea is that every time a YouTube video is published on my channel I want to have an associated post on my personal Jekyll blog.

Until very recently I handled all this by hand. This process was very tedious because, except for the video summary (more on this later), it was merely a copy paste operation of fields in the YAML front matter.

✅ The solution

📶 Feeds

The solution to this is to leverage on the RSS feeds provided by YouTube itself, and a few Python libraries:

📶 feedparser: the core dependency element
💻 PyYAML: parses the existing front matter
🐌 slugify: not crucial for this use case but useful

Every YouTube channel, in-fact, has feeds in this URL format:

https://www.youtube.com/feeds/videos.xml?channel_id={channel_id}

⚙️ Algorithm

Essentially the algorithm work like this:

get and parse the feed file
for each news item (entry) in the feed file, extract:
1. URL
2. title
3. published date
4. tags (via the video description, using a regex)
create a new markdown file in the _posts directory using variables of step 2, and avoid changing existing auto-posts

Concerning step 1 and 2 it's quite simple thanks to list comprehensions:

def extract_feed_youtube_data(feed_source: str) -> list[dict]:
    d = feedparser.parse(feed_source)
    data = [
        {
            'url': e['link'],
            'title': e['title'],

            # feedparser generates a Python 9-tuple in UTC.
            'published': datetime.datetime(*e['published_parsed'][:6]),

            # Use a regex for this.
            # If there are no hash tags available just use default ones.
            'tags': get_video_tags(e['summary'])
                if ('summary' in e and e['summary'] != '')
                else STANDARD_TAGS,

            # The video description is not used at the moment.
            'sm': e['summary'],
        }
        for e in d.entries if 'link' in e
    ]

Before returning the data you can also perform a cleanup:

    for e in data:                                                              
        # Always use default tags. This also works in case videos do not have
        # a description.
        e['tags'] += STANDARD_TAGS

        # Unique.
        e['tags'] = sorted(list(set(e['tags'])))

        e['summary'] = ' '.join(e['tags'])

    return data

Step 3 involves an f-string. We need to take care of specific fields to avoid Jekyll throwing YAML parsing errors when using quotes. This can happen in the title and description fields specifically.

def create_markdown_blog_post(posts_base_directory: str, data: dict):
    return f"""---
title: "|-"
    {data["title"]}


tags: [{youtube_tags_to_jekyll_front_matter_tags(data["tags"])}]
related_links: ['{data["url"]}']
updated: {datetime_object_to_jekyll_front_matter_utc(data["published"])}
description: "|-"
    {data["title"]}


lang: 'en'
---
{generate_youtube_embed_code(get_youtube_video_id(data["url"]))}

<div markdown="0">
{data["summary"]}
</div>

*Note: post auto-generated from YouTube feeds.*
"""

🎯 Result

As you see, the content of each auto post is very basic: beside the standard fields, in the body we can find an HTML YouTube embed code, and a list of hash
tags extracted from the video description using a regex. The description (summary) was left out on purpose. The idea to implement in the future is to
get the video transcription and let a LLM (via Ollama) generate a summary. Of course I then need to manually proofread it. I also cannot copy the video
description verbatim because of SEO.

Another improvement could involve replicating a subset of the fields of the
auto-post to send toots to Mastodon using its API.

▶️ Running

At the moment the script is triggered by a local pre-commit hook which also installs the Python dependencies in a separate environment:

- repo: local
  hooks:
    - id: generate_posts
      name: generate_posts
      entry: python ./.scripts/youtube/generate_posts.py
      verbose: true
      always_run: true
      pass_filenames: false
      language: python
      types: [python]
      additional_dependencies: ['feedparser>=6,<7', 'python-slugify[unidecode]>=8,<9', 'PyYAML>=6,<7']

🎉 Conclusion

And that's it really. This script saves time and it's the kind of automation I like. Thankfully YouTube still provides RSS feeds, although I already had to fix the script once to adapt to their new structural changes.

If you are interested in the source code you can find it on its repo.

You can comment here and check my YouTube channel for related content!