DEV Community: Lars Quentin

Use XQuery for HTML (Web Crawling)

Lars Quentin — Fri, 06 Dec 2024 16:02:58 +0000

This is a very very niche post. If you don't know why this is a pain point, don't waste your time reading this.

XQuery is a great language for high level XML processing, providing a fully turing complete declarative language leveraging XPath. Unfortunately, it is not used often.

My personal take is that this is the case because most HTML out there is not XML compliant, mostly because of tags that are never closed (such as <link ...> instead of <link .../>). Thus your Saxon/BaseX parser will fail.

The solution is TagSoup, which provides a SAX-compliant HTML parser, pretending that it just parses XML.

BaseX just uses it for you if you have it in Path!
Saxon provides saxon:parse-html if TagSoup is in the classpath, which can be used after fetching with fn:unparsed-text

With that, you can now do actual web crawling! The rest is just plain XQuery.

Hack: Forcing Puppet to apt update

Lars Quentin — Fri, 09 Apr 2021 16:50:58 +0000

Short hack: As seen in m a n y Dockerfiles, the apt cache is located at /var/lib/apt/lists.

Therefore, we can use it's existence to create an idempotent exec-Ressouce!

exec { 'apt update':
  # Only if the update files dont exist
  unless => '[ "$(ls -A /var/lib/apt/lists)" ]',
  # /bin is needed for bash
  # /usr/bin is needed for [
  path => '/bin:/usr/bin',
}

Now whenever you need to have the apt cache first, just do something like

package { 'your-package':
  ensure => 'present',
  require => Exec['apt update']
}

Easy as 🥧

Using Grub without Linux for Windows 7/10 Dual Boot Configuration

Lars Quentin — Fri, 21 Aug 2020 20:50:50 +0000

I throw together things until it works
-- Rasmus Lerdorf, Creator of PHP

TL;DR at the bottom

This is part 1 on a 2 part series about manual grub configuration

Installing Standalone Grub for Windows 7/10 Dual Boot Configuration
The Absolute Minimum Every Linux User Absolutely, Positively Must Know About grub.cfg (No Excuses!) (coming soon®)

The Problem

Create a dual boot with
- A hard drive with Windows 7
- A SSD with Windows 10
I am not allowed to edit the BCD
I am not allowed to install any other OS

The Idea: Installing Grub with 2 Windows bootloader entries.

Preparation is Half the Battle

Making Space for the boot Partition

Since we shrink NTFS, we use Windows for that.

First, press <Win-r> and run compmgmt.msc and use the "Disk Management" utility.

Right click on the partition you want to shrink and make at least 200MB space for grub.

After you made some free space, right click the free space and create a new partition using "New Simple Volume". Which settings you select doesn't matter since we will overwrite it later.

After that, the last preparation step is to

Create a neat Ubuntu Stick

Just use rufus.

Installing Grub

After you booted up, grab a root shell, and find out which the new partition is (for example by listing all partitions via fdisk -l and comparing size. You can ignore all /dev/loopXX.)

Now we format it with a grub compatible file system. From now on, the new partiton is /dev/sda2.

(for simplicity, I assume bios/mbr from now on)

mkfs.ext4 /dev/sda2

Afterwards, we mount it, create a boot directory and try to install grub:

mount /dev/sda2 /mnt && mkdir /mnt/boot
# The --boot-directory is needed since we don't manipulate
# our own bootloader, therefore we don't have the default
# location /boot
grub-install --boot-directory=/mnt/boot /dev/sda1

If that didn't work, just force it, we have to configure it manually anyways:

grub-install --force --boot-directory=/mnt/boot /dev/sda1

It can't be that easy! And it isn't!!!! When you try to generate the config with

grub-mkconfig -o /mnt/boot/grub/grub.cfg

it should cry about something like

/usr/sbin/grub-probe:error:failed to get canonical path of /cow.

This happens because grub-probe, the tool called by the grub-mkconfig script, can't handle ubuntu's live system root.

Improvise, Adapt, Overcome

But that's no problem! Writing one's own minimal grub.cfg is easier than you'd expect!

First, find out which UUID's your C:\ partitions have. Again, just compare the size and file system with fdisk -l and throw the /dev/sdxx into blkid.

Now, let's start writing a grub.cfg by opening your favorite editor

nano -w /mnt/boot/grub/grub.cfg

We just have to do 2 things. First, we set some sane defaults:

# Set first as default
set default="0"

# Set timeout for 10 seconds that we have enough time
set timeout=10

# sane IO
terminal_input console
terminal_output gfxterm

# minimalistic color sheme
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray

Now, we have to create the entries. I'll comment as I go

# Here, the "Windows 10" is just the representative name
# it can be chosen as you want to
menuentry "Windows 10" --class windows --class os {
        # insmod ntfs loads ntfs support
    insmod ntfs
        # finds the right partition by the UUID you found earlier
    search --no-floppy --set=root --fs-uuid <YOUR UUID>
        # Starts the windows NT loader
    ntldr /bootmgr
}

# Analogous for your other windows
menuentry "Windows 7" --class windows --class os {
    insmod ntfs
    search --no-floppy --set=root --fs-uuid <YOUR UUID>
    ntldr /bootmgr
}

Afterwards, just unmount everything, change the boot order if necessary and et voila, you solved the problem!

TLDR Code

Minimal grub.cfg

set default="0"
set timeout=10
terminal_input console
terminal_output gfxterm
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
menuentry "Windows 10" --class windows --class os {
    insmod ntfs
    search --no-floppy --set=root --fs-uuid <YOUR UUID>
    ntldr /bootmgr
}
menuentry "Windows 7" --class windows --class os {
    insmod ntfs
    search --no-floppy --set=root --fs-uuid <YOUR UUID>
    ntldr /bootmgr
}

Thanks

Thanks for reading and thanks to Anton Tagunov for his blog post on the topic.

Dynamic Group based LDAP authentication with django and RegEx

Lars Quentin — Sun, 01 Mar 2020 20:11:22 +0000

I had the following problem:

Authentication over Active Directory through LDAP
just certain groups with a specific suffix were allowed to login
There was no way to know how many of those groups exist
Later views depended on those groups, therefore they had to be mapped to django groups

tl;dr: If you just want the full code, feel free to jump to the end, but at least leave a upvote. :D

Prerequisites (You already know them)

Besides django, you need django-auth-ldap as well

pip3 install django-auth-ldap

with that installed, we are ready to go!

Writing our own basic LDAP Backend

At first, let's create our own Backend, by extending the default LDAPBackend from django-auth-ldap.
For the official documentation, see here.

<name_of_main_application>/ldap.py

class GroupLDAPBackend(LDAPBackend):
  pass

Now before we do further customisation, let's just fill it with our parameters for the Search/Bind authentication:

<name_of_main_application>/ldap.py

import ldap

from django_auth_ldap.backend import LDAPBackend
from django_auth_ldap.config import LDAPSearch

class GroupLDAPBackend(LDAPBackend):
  default_settings = {
    # All those settings are overwriting base class values
    "SERVER_URI": "ldaps://url.to.our.dc.domain.com",
    "CACHE_TIMEOUT": 3600,

    # Those settings should probably be overwritten by the settings.py
    "BIND_DN": "",
    "BIND_PASSWORD": "",
    "USER_SEARCH": LDAPSearch(
      "OU=<OU_OF_USER_LOGGING_IN>, OU=..., DC=..., DC=domain, DC=com",
      # If you know, that all your users logging in are on that 
      # exact ou depth specified above, you can get better performance
      # by using ldap.SCOPE_BASE or for that depth and its direct 
      # children ldap.SCOPE_ONELEVEL, see python-ldap documentation for
      # more.
      ldap.SCOPE_SUBTREE,
      "(uid=%(user)s)"
    ),
  }

Those settings are the same as outside of the class with the AUTH_LDAP Prefix.

Here a short explaination of all those parameters:

SERVER_URI: The URL to the domain controller
CACHE_TIMEOUT: How long the data from the LDAP Server is cached. Here set to one hour.
BIND_DN: The fully qualified Distinguished Name, which means that dc's/ou's cant be before the cn logging in mustn't be omitted. You'll see an example later.
USER_SEARCH: The search, which will be performed on in the authentication process

Now we can add the ldap authentication in the settings.py, together with the other settings.

<name_of_main_application>/settings.py

...
AUTH_LDAP_BIND_DN = "CN=<user_which_handles_all_binds>, OU=its_ou, ..., DC=domain, DC=com"
AUTH_LDAP_BIND_PASSWORD = "password_for_that_account"

...

AUTHENTICATION_BACKENDS = [
  "<name_of_main_application>.ldap.GroupLDAPBackend",
  # And if wanted, the django one as well
  "django.contrib.auth.backends.ModelBackend",
]

In production, instead of hard coding (and even worse, committing) the credentials please export them into your linux environment and use os.getenv(KEY). This way, you don't save any critical information in your repository.

Also, if you have any problems finding the credentials, you can enable logging as well with

<name_of_main_application>/settings.py

LOGGING = {
  "version": 1,
  "disable_existing_loggers": False,
  "handlers": {"console": {"class": "logging.StreamHandler"}},
  "loggers": {"django_auth_ldap": {"level": "DEBUG", "handlers": ["console"]}},
}

This should already be enough to get the normal authentication working!
Now we just have to implement the group-based authentication.
Let's get right into the "fun"!

Extending our badass Backend with Group Authentication

Let's say, we want to allow any user, which has a group with the suffix "django".

"some_group_django": ✔️
"admindjango": ✔️
"djangogrp": ❌

For educational purposes, they all groups are located in
OU=Groups, DC=domain, DC=com
and its sub organisational units.

With that, we can extend our backend with our own RegEx-Setting as well as GROUP_SEARCH and GROUP_TYPE

import re
import ldap

from django_auth_ldap.backend import LDAPBackend
from django_auth_ldap.config import LDAPSearch, GroupOfNamesType

class GroupLDAPBackend(LDAPBackend):
  default_settings = {
    # Our new settings
    # Lets call the RegEx GROUP_REGEX for simplicity
    "GROUP_REGEX": re.compile(r"^*django$"),
    "GROUP_SEARCH": LDAPSearch(
      "OU=Groups, DC=domain, DC=com",
      ldap.SCOPE_SUBTREE,
      "(cn=*django)"
    ),
    "GROUP_TYPE": GroupOfNamesType(),

    # All those settings are overwriting base class values
    "SERVER_URI": "ldaps://url.to.our.dc.domain.com",
    "CACHE_TIMEOUT": 3600,

    # Those settings should probably be overwritten by the settings.py
    "BIND_DN": "",
    "BIND_PASSWORD": "",
    "USER_SEARCH": LDAPSearch(
      "OU=<OU_OF_USER_LOGGING_IN>, OU=..., DC=..., DC=domain, DC=com",
      # If you know, that all your users logging in are on that 
      # exact ou depth specified above, you can get better performance
      # by using ldap.SCOPE_BASE or for that depth and its direct 
      # children ldap.SCOPE_ONELEVEL, see python-ldap documentation for
      # more.
      ldap.SCOPE_SUBTREE,
      "(uid=%(user)s)"
    ),
  }

Finally, we can now override the LDAPBackend.authenticate_ldap_user with our own version!

I'll use some type hinting here, since the Documentation for _LDAPUser is more or less nonexisting so that you know where to find the source code.

# Just needed for type hinting lol
from django_auth_ldap.backend import _LDAPUser

class GroupLDAPBackend(LDAPBackend):
  default_settings = {
    ...
  }

  def authenticate_ldap_user(self, ldap_user: _LDAPUser, password: str):
    # This is the default implemented authentication
    user = ldap_user.authenticate(password)

    # If the authentication was denied, we have to return None
    if not user:
      return None

    ldap_groups = ldap_user.group_names
    ldap_groups = {x for x in ldap_groups if self.settings.GROUP_REGEX.match(x)}

    if len(ldap_groups) == 0:
      return None
    return user

I mean, at least for me, that was way easier than expected.
Lastly, we just have to create the django groups from the ldap ones.

Creating Django Groups on the fly!

Luckily, django has the get_or_create pattern, which makes things a lot easier.

from django.contrib.auth.models import Group

class GroupLDAPBackend(LDAPBackend):
  ...
  def create_groups_and_assign_user_to_it(self, user, ldap_groups):
    for group_name in ldap_groups:
      django_group, was_created = Group.objects.get_or_create(name=group_name)
      django_group.user_set.add(user)

That's it! If it helped you or you have further questions, feel free to hook me up on twitter.
Also, if there is a even neater way to implement it, please tell me and I'll correct it.

The whole source code with some logging

<name_of_main_application>/settings.py

...
AUTH_LDAP_BIND_DN = "CN=<user_which_handles_all_binds>, OU=its_ou, ..., DC=domain, DC=com"
AUTH_LDAP_BIND_PASSWORD = "password_for_that_account"

AUTHENTICATION_BACKENDS = [
  "<name_of_main_application>.ldap.GroupLDAPBackend",
  # And if wanted, the django one as well
  "django.contrib.auth.backends.ModelBackend",
]

LOGGING = {
  "version": 1,
  "disable_existing_loggers": False,
  "handlers": {"console": {"class": "logging.StreamHandler"}},
  "loggers": {"django_auth_ldap": {"level": "DEBUG", "handlers": ["console"]}},
}

<name_of_main_application>/ldap.py

import re
import ldap
import logging

from django_auth_ldap.backend import LDAPBackend, _LDAPUser
from django_auth_ldap.config import LDAPSearch, GroupOfNamesType

from django.contrib.auth.models import Group

class GroupLDAPBackend(LDAPBackend):
  default_settings = {
    # Our new settings
    # Lets call the RegEx GROUP_REGEX for simplicity
    "GROUP_REGEX": re.compile(r"^*django$"),
    "GROUP_SEARCH": LDAPSearch(
      "OU=Groups, DC=domain, DC=com",
      ldap.SCOPE_SUBTREE,
      "(cn=*django)"
    ),
    "GROUP_TYPE": GroupOfNamesType(),

    # All those settings are overwriting base class values
    "SERVER_URI": "ldaps://url.to.our.dc.domain.com",
    "CACHE_TIMEOUT": 3600,

    # Those settings should probably be overwritten by the settings.py
    "BIND_DN": "",
    "BIND_PASSWORD": "",
    "USER_SEARCH": LDAPSearch(
      "OU=<OU_OF_USER_LOGGING_IN>, OU=..., DC=..., DC=domain, DC=com",
      # If you know, that all your users logging in are on that 
      # exact ou depth specified above, you can get better performance
      # by using ldap.SCOPE_BASE or for that depth and its direct 
      # children ldap.SCOPE_ONELEVEL, see python-ldap documentation for
      # more.
      ldap.SCOPE_SUBTREE,
      "(uid=%(user)s)"
    ),
  }
  def authenticate_ldap_user(self, ldap_user: _LDAPUser, password: str):
    # This is the default implemented authentication
    user = ldap_user.authenticate(password)

    # If the authentication was denied, we have to return None
    if not user:
      return None

    ldap_groups = ldap_user.group_names
    ldap_groups = {x for x in ldap_groups if self.settings.GROUP_REGEX.match(x)}

    if len(ldap_groups) == 0:
      return None

    self.create_groups_and_assign_user_to_it(user, ldap_groups)
    return user

  def create_groups_and_assign_user_to_it(self, user, ldap_groups):
    for group_name in ldap_groups:
      django_group, was_created = Group.objects.get_or_create(name=group_name)
      django_group.user_set.add(user)