DEV Community: Wewake

When JNI Meets 'Write Once, Run Anywhere': Navigating Java's Multi-Architecture Reality

Wewake — Fri, 26 Dec 2025 14:59:57 +0000

Introduction
The "Write Once, Run Anywhere" Philosophy
- The Problem It Solved
- Java's Brilliant Solution
The Exception: When Architecture Matters
- Enter JNI: The Bridge to Native Code
- The Problem: Native Libraries Are Platform and Architecture-Specific
How Libraries Solve the Multi-Arch Problem
- The "Bundle Everything" Pattern
- The "Platform/Architecture-Specific JAR" Pattern
- When to Use Each Pattern
Conclusion
- Further Reading
References

Introduction

One of Java's most celebrated promises is "Write Once, Run Anywhere" (WORA). The idea is simple and powerful: write your code once, compile it, and run it on any platform—Windows, Linux, macOS, x86, ARM—without modification. This promise has made Java the language of choice for enterprise applications, Android development, and countless other use cases.

For pure Java code, this promise holds beautifully. But there's a caveat.

When you venture into the world of JNI (Java Native Interface), which is the bridge between Java and native code written in C, C++, or Rust—the cross-platform simplicity disappears. Suddenly, we need to worry about platform-specific binaries, and get treated with the cryptic UnsatisfiedLinkError error messages.

In this article, we'll explore:

How Java is normally platform-independent
When and why that independence breaks down with JNI
How modern libraries cleverly solve the multi-architecture challenge
Peak under the hood of a production multi-architecture library

The "Write Once, Run Anywhere" Philosophy

Java's WORA promise isn't just marketing—it's a fundamental architectural decision that revolutionized software development in the mid-1990s.

The Problem It Solved

Before Java, cross-platform software required painful compromises:

Separate codebases for each platform (Windows, Mac, Unix)
Abstraction layers like POSIX, but still compile separately for each platform
Interpreted languages like Perl or Python, which were slower and had limited ecosystem

Java's Brilliant Solution

Java introduced an intermediate layer: bytecode. Instead of compiling directly to machine code, Java compiles to a universal, platform-neutral instruction set that can run on any machine with a JVM.

Source Code (.java)  →  Bytecode (.class)  →  JVM (x86/ARM/etc)  →  Machine Code
   [Developer]           [Compiler]            [Runtime]              [CPU]

When you compile Java code with javac, it creates .class files containing bytecode—universal instructions that aren't specific to x86, ARM, or any particular CPU architecture. These bytecode instructions (like aload_0, getstatic, invokevirtual) are understood by the JVM, not directly by the hardware.

The magic happens at runtime: each platform has its own JVM implementation (compiled for x86-linux, aarch64-darwin, etc.) that reads this universal bytecode and translates it to native machine code via Just-In-Time (JIT) compilation. The same .jar (think of it as a zipped folder of .class files) file can run on Linux x86_64, macOS ARM64, or Windows x86_64 without any modification.

This architecture provides True portability, since the same .jar file runs everywhere.

The Exception: When Architecture Matters

Now comes the interesting part: when does this beautiful abstraction break down?

Enter JNI: The Bridge to Native Code

JNI (Java Native Interface) allows Java code to call functions written in C, C++, Rust, or other languages that compile to native machine code.

Why would we want to do this?

Performance: Critical operations like cryptography are often faster in native code
Hardware Access: Low-level hardware operations that Java can't do
Legacy Integration: Calling existing C/C++ libraries
System APIs: Accessing OS-specific features not exposed in Java

I feel it's important to mention that using JNI is not always beneficial. JNI calls incur an overhead. So, if JNI is invoked frequently, for low-work operations, it's not going to help reduce latency. We should only use it if the Java native operation is much slower.
{:.prompt-info}

The Problem: Native Libraries Are Platform and Architecture-Specific

When you compile native code, you must compile it for each target platform (Windows, MacOS, Linux, etc) and target architecture (x86-64, ARM64, armv7, etc).

We can run this simple test to explore this ourselves:

% mvn dependency:get -Dartifact=org.xerial:sqlite-jdbc:3.44.1.0 # Download sqlite jdbc jar from maven
% jar xf ~/.m2/repository/org/xerial/sqlite-jdbc/3.44.1.0/sqlite-jdbc-3.44.1.0.jar # Extract its contents

% # Checkout its target platform & architecture
% file native/Mac/x86_64/libsqlitejdbc.dylib
native/Mac/x86_64/libsqlitejdbc.dylib: Mach-O 64-bit dynamically linked shared library x86_64

% file native/Mac/aarch64/libsqlitejdbc.dylib
native/Mac/aarch64/libsqlitejdbc.dylib: Mach-O 64-bit dynamically linked shared library arm64

% file native/Windows/x86_64/sqlitejdbc.dll
native/Windows/x86_64/sqlitejdbc.dll: PE32+ executable (DLL) (console) x86-64 (stripped to external PDB), for MS Windows

% file native/Windows/x86/sqlitejdbc.dll
native/Windows/x86/sqlitejdbc.dll: PE32 executable (DLL) (console) Intel 80386 (stripped to external PDB), for MS Windows

% file native/Windows/armv7/sqlitejdbc.dll
native/Windows/armv7/sqlitejdbc.dll: PE32 executable (DLL) (GUI) ARMv7 Thumb, for MS Windows

% file native/Windows/aarch64/sqlitejdbc.dll
native/Windows/aarch64/sqlitejdbc.dll: PE32+ executable (DLL) (GUI) Aarch64, for MS Windows

The x86 .so file won't work on ARM, and vice versa. The Linux .so won't work on Windows. When you try it, you will get the famous java.lang.UnsatisfiedLinkError.

The "Write Once, Run Anywhere" promise is broken, in a way.

How Libraries Solve the Multi-Arch Problem

Modern Java libraries that use JNI have developed clever patterns to maintain cross-platform compatibility.

The "Bundle Everything" Pattern

The most common solution: include all native libraries for all platforms in a single JAR. We saw this in the last section with sqlite-jdbc.

At runtime, the Java code detects the platform and extracts the appropriate native library:

Detect OS and architecture from system properties
Extract the correct native library from the JAR to a temporary location
Load the library using System.load()
Clean up temporary files on JVM exit

The JAR file is larger (containing multiple native libraries), but the developer experience is seamless.

The "Platform/Architecture-Specific JAR" Pattern

An alternative approach: publish separate JARs for each platform, and let the build tool select the right one at build time.

Instead of one fat JAR containing all native libraries, libraries publish multiple artifacts:

For instance, checkout netty-tcnative:

% curl -s https://repo1.maven.org/maven2/io/netty/netty-tcnative/2.0.74.Final/ | grep -o 'netty-tcnative-[^"]*\.jar' | sort -u
netty-tcnative-2.0.74.Final-javadoc.jar
netty-tcnative-2.0.74.Final-linux-x86_64-fedora.jar
netty-tcnative-2.0.74.Final-linux-x86_64.jar
netty-tcnative-2.0.74.Final-osx-aarch_64.jar
netty-tcnative-2.0.74.Final-osx-x86_64.jar
netty-tcnative-2.0.74.Final-sources.jar
netty-tcnative-2.0.74.Final.jar

One slight hiccup:

If we specify the platform manually in our pom.xml, it becomes platform-specific:

<!-- BAD: Hard-coded platform -->
<dependency>
    <groupId>io.netty</groupId>
    <artifactId>netty-tcnative</artifactId>
    <version>2.0.74.Final</version>
    <classifier>linux-x86_64</classifier>  <!-- Won't work on macOS! -->
</dependency>

Our pom.xml now only works on Linux x86_64. Developers on macOS can't build, and we can't deploy to ARM servers.

The Solution: Maven's os-maven-plugin

The os-maven-plugin automatically detects your build platform and selects the correct JAR:

<!-- Add the plugin -->
<build>
    <extensions>
        <extension>
            <groupId>kr.motd.maven</groupId>
            <artifactId>os-maven-plugin</artifactId>
            <version>1.7.1</version>
        </extension>
    </extensions>
</build>

<!-- Use auto-detected classifier -->
<dependency>
    <groupId>io.netty</groupId>
    <artifactId>netty-tcnative</artifactId>
    <version>2.0.74.Final</version>
    <classifier>${os.detected.classifier}</classifier>  <!-- Auto-detected! -->
</dependency>

The plugin sets ${os.detected.classifier} based on your build environment. This way, the same pom.xml works on every developer's machine and in every CI/CD environment!

When to Use Each Pattern

Use "Bundle Everything" when:

Distributing desktop applications to unknown end-users
We want a single universal artifact that works everywhere
We deploy the same JAR to multiple different architectures
Simplicity is more important than JAR size

Use "Platform-Specific JARs" when:

Building server applications deployed to known platforms
Using containerization (each container image is already platform-specific)
JAR size matters (e.g., AWS Lambda cold start times)
We have a consistent CI/CD pipeline per architecture

Conclusion

Java's "Write Once, Run Anywhere" promise isn't a myth, it just comes with a practical asterisk when we venture into the native code territory. But with the patterns we've explored, modern Java libraries have found elegant ways to maintain cross-platform compatibility even when using architecture-specific native code.

References

This article was originally published on my blog at https://wewake.dev/posts/when-jni-meets-write-once-run-anywhere-in-multi-arch-world/

Find more of my content on my blog, where I share my software development experiences and learnings: https://wewake.dev

Why Your Python Logs Vanish in Docker (and How PYTHONUNBUFFERED=1 Saves the Day)

Wewake — Fri, 26 Dec 2025 14:55:04 +0000

The Mystery: Why are my logs disappearing ?
What is Output Buffering?
- Why Buffering Exists ?
- The Three Types of Buffering
Why Docker Makes This Worse
The Solution: PYTHONUNBUFFERED=1
Alternative Solutions
- Use python unbuffered commandline option
- Flush Manually
- Use stderr in Python's logging module
Conclusion
References

The Mystery: Why are my logs disappearing ?

This has unfortunately happened almost every time I have come back to using python in my little side projects, or experiments.

I create a neat little Python app. It runs perfectly, printing helpful logs at every step:

# app.py
print("Starting application...")
print("Connecting to database...")
print("Processing request...")
print("All done!")

I run it locally:

$ python app.py
Starting application...
Connecting to database...
Processing request...
All done!

All good! I containerize it:

FROM python:3.12-slim
COPY app.py .
CMD ["python", "app.py"]

Build and run it in Docker:

$ docker build -t myapp .
$ docker run myapp
$ docker logs <container-id>
#

But the logs are nowhere to be seen. Where did it go ? If running k8, I check kubectl logs , same thing. No logs.

The logs, like a magician's trick, have vanished into thin air. What sorcery is this?

Spoiler: It's Python's output buffering, and it's been frustrating developers since Docker became popular.

What is Output Buffering?

Output buffering is when your program doesn't immediately write output to its destination (like your terminal or Docker logs). Instead, it collects output in a temporary buffer (a chunk of memory) and writes it all at once.

Think of it like this:

Without buffering:

[Your code] ---> "Hello" ---> [Terminal]
            ---> "World" ---> [Terminal]
            ---> "!!!"    ---> [Terminal]

With buffering :

[Your code] ---> "Hello" --\
            ---> "World" ---}--[Buffer]---> [Terminal] (when full or flushed)
            ---> "!!!"   --/

Why Buffering Exists ?

Buffering is a performance optimization. Writing to output (terminal, files, network) is slow compared to memory operations.

The Three Types of Buffering

Python (and most languages) uses three buffering modes:

Unbuffered: Write immediately (no buffer)
- Used for: stderr by default
- Behavior: Every print() appears instantly
Line-buffered: Write after each newline (\n)
- Used for: stdout when connected to a terminal (TTY)
- Behavior: Output appears after each line
Block-buffered: Write when buffer is full (default: 8KB)
- Used for: stdout when NOT connected to a terminal (pipes, files, Docker)
- Behavior: Output appears in chunks, or when program exits

Note: Python detects if stdout is connected to a terminal. If yes, it line-buffers. If no (like in Docker), it block-buffers.

Want to see the actual buffer size? You can check it yourself:

import io
print(f"Default buffer size: {io.DEFAULT_BUFFER_SIZE} bytes")
# Output: Default buffer size: 8192 bytes

This io.DEFAULT_BUFFER_SIZE constant (defined in Python's io module) is what Python uses for block-buffering. It's typically 8192 bytes (8KB), though it can vary based on the system's block size.

Why Docker Makes This Worse

When you run Python in a Docker container:

stdout is NOT a TTY (it's a pipe to Docker's logging driver)
Python switches to block-buffering (8KB buffer by default)
The logs sit in memory until:
- The buffer fills up, OR
- The program exits, OR
- You explicitly flush

It's a classic case of "works on my machine" syndrome—because your machine is a TTY!

The Solution: PYTHONUNBUFFERED=1

Set the PYTHONUNBUFFERED environment variable to any non-empty value. This forces Python to use unbuffered mode for stdout and stderr.

In Dockerfile:

FROM python:3.12-slim

# Add this line!
ENV PYTHONUNBUFFERED=1

COPY app.py .
CMD ["python", "app.py"]

In docker-compose.yml:

version: '3.8'
services:
  app:
    build: .
    environment:
      - PYTHONUNBUFFERED=1

In Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:latest
        env:
        - name: PYTHONUNBUFFERED
          value: "1"

At runtime:

docker run -e PYTHONUNBUFFERED=1 myapp

Now our logs will appear instantly:

$ docker logs myapp
Starting application...
Connecting to database...
Processing request...
All done!

Fun fact: The value doesn't matter. PYTHONUNBUFFERED=1, PYTHONUNBUFFERED=true, even PYTHONUNBUFFERED=banana all work. Python just checks if the variable is set and non-empty.

Alternative Solutions

While PYTHONUNBUFFERED=1 is the easiest fix, here are other approaches:

Use python unbuffered commandline option

The -u flag forces unbuffered mode:

FROM python:3.12-slim
COPY app.py .
CMD ["python", "-u", "app.py"]  # Note the -u flag

This is equivalent to PYTHONUNBUFFERED=1, just more explicit.

Pros: No environment variable needed

Cons: Have to remember to add -u everywhere you run Python

Flush Manually

Call flush() explicitly when you want output to appear:

import sys

print("Critical log message", flush=True)  # Appears immediately

# Or flush everything:
sys.stdout.flush()

Pros: Fine-grained control over when to flush

Cons:

Easy to forget
Clutters your code
Doesn't help with third-party libraries that use print()

Use stderr in Python's logging module

stderr is unbuffered by default.

import logging
import sys

# Configure logging to stderr (which is unbuffered by default)
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    stream=sys.stderr  # stderr is unbuffered!
)

logger = logging.getLogger(__name__)

logger.info("Application starting")
logger.info("Connected to database")
logger.error("Something went wrong!")

Why this works:

stderr is unbuffered by default (even in Docker)
Structured logging is better for production anyway
You get timestamps, log levels, and better formatting

Conclusion

The PYTHONUNBUFFERED=1 environment variable is a tiny fix that solves a huge headache. It forces Python to stop buffering output, so your logs appear immediately in Docker and Kubernetes.

References

This article was originally published on my blog at https://wewake.dev/posts/why-your-python-logs-vanish-in-docker-pythonunbuffered-explained/ .

Find more of my content on my blog, where I share my software development experiences and learnings: https://wewake.dev

Impress Recruiters and Upskill using these 10 Unconventional Software Engineering Projects

Wewake — Sun, 05 May 2024 21:36:22 +0000

Introduction

It's a tough market out there, and the ability to stand out is very important. Most (if not all) resumes for software engineering roles contain several projects. But most of them are generic clones that all recruiters (and even interviewers) have seen hundreds of times. Such projects may well help you to fill up that resume, but they don't really add much of a value.

How to come up with a project idea

This may seem like a daunting task, but it is also incredibly rewarding. Some of my tips:

Identify your interests and strengths: Start by identifying your interests and strengths in software engineering. What technical or programming language do you like? What skills are you good at?
Research Current Products and Technologies Research: Keep up to date with current products and technologies in software technology. Look for emerging technologies like AI, blockchain, or AR/VR, that have the potential to disrupt industries or solve complex problems. Choosing such technology can set your project apart and reflect your forward-thinking approach.
Think unique solutions to common problems: Think outside the box and come up with unique solutions to common problems. Consider how you can use technology to automate tasks, improve productivity, or enhance the user experience in new and innovative ways.
Seek feedback and validation: Once you have a few ideas in mind, ask for feedback from classmates, mentors, or industry professionals. Their feedback can help you refine your idea and choose the most promising one for your project.
Be open to iterative development: Be open to iteration and refinement, and be willing to adapt if necessary. The final project may look different from your initial idea, but as long as it aligns with your goals and showcases your skills, it can be a successful and impactful project

List of Ideas

The following curated list contains more uncommon projects, that take a bit more time and effort to create. Naturally, if you have such projects in your resume, the chances of getting selected for the interviews is much higher.

Interactive Resume Website: Create a dynamic website that showcases your skills, experience, and services using HTML, CSS, and JavaScript. Include interactive features like animations and infographics to keep visitors engaged.
Virtual Reality (VR) Portfolio: Create a VR experience that showcases your projects and accomplishments. Use frameworks like A-Frame or Unity to create an immersive environment.
AI-Powered Personal Chatbot: Build a chatbot that can answer common questions about your industry, skills and experiences. Use natural language processing (NLP) and machine learning to improve its responsiveness over time.
Gesture-driven web applications: Create web applications that can be operated via gestures. Use libraries like TensorFlow.js or OpenCV.js for gesture recognition.
Augmented Reality (AR) Visualization Tools: Visualize an AR application that visualizes complex data or concepts. Use AR.js or Unity AR Foundation to create interactive AR experiences.
Blockchain-based credential verification system: Develop a system that uses blockchain technology to verify credentials and certificates. Use smart contracts to manage the verification process.
Real-time code collaboration platform: Create a platform that allows multiple developers to collaborate on code in real time. Use WebSockets for real-time communication with code editors like Monaco Editor or CodeMirror.
Voice-Controlled Task Manager: Create a task management application that can be operated using voice commands. Use a speech recognition API such as Web Speech API or Google Cloud Speech-to-Text.
Automated Code Review Tool: Create a tool that automatically reviews code to look for common problems and suggest improvements. Use static code analysis tools such as ESLint or SonarQube.
Artificial Intelligence Art Generator: Develop an AI-powered tool that generates unique pieces of art based on user inputs such as colors, styles, and themes. This project demonstrates your ability to create innovative and artistic applications of AI.

Conclusion

While it's important to have come up with, and worked on unconventional projects to showcase and set you apart, it's equally important to be able to replicate the design, reason and defend the design and technology choices & trade offs, during interviews. When explained and defended well, it leaves a positive impression on the interviewer, and highly increases the chances of selection.

If you like this, I write and publish technical content on my personal blog.

Practical and Beginner friendly guide for speeding up your web-apps

Wewake — Wed, 01 May 2024 15:07:34 +0000

Introduction
Setup
Starting State
Tree Shaking
Minification
Use only what is needed
Code Splitting
Lazy loading
Production mode
Compression
Other options
Bundle analyzer
Conclusion
References

Introduction

In modern web development, all webpage assets like html, css, js, etc are bundled together into a smaller set of assets. This helps in reducing network calls from the client/user agent, and gives a much smaller file to download. This in turn gives a fast and efficient user experience, which is what we all want.

There are various tools available that manage the size of bundled assets. We are going to use the example of a popular and widely used bundler named Webpack, and practically look at many of the optimization techniques it offers.

Setup

Let's start with a basic node + react setup. It's okay if you don't have expertise in this. Ensure you have node and npm and npx installed.

We are using webpack5, but the concepts are more or less transferable across versions with minor modifications.

My node and npm version details:

webpack-optimization-demo % node --version
v20.11.1
webpack-optimization-demo % npm --version
10.2.4

You can get the initial code from repo. Follow these steps:

git clone git@github.com:viv1/blog-code-examples.git
git checkout dab817ec0a502adff4c82070c58ee36bb79e7bc0
cd blog-code-examples/webpack-optimization-demo , then run npm install ..
You can go directly to section Starting State.

If you would like to do it from scratch, follow along:

Let's initiate a node project:

mkdir webpack-optimization-demo
cd webpack-optimization-demo
npm init -y

Install libraries and dependencies:

npm install --save-dev webpack webpack-cli webpack-dev-server style-loader css-loader babel-loader @babel/core @babel/preset-env @babel/preset-react html-webpack-plugin
npm install --save prop-types process lodash react react-dom webpack

Create the following files in this directory structure:

This is a simple single page app with 2 components.
App.js is cretaed using Component1 and a Component2. Component1 and Component2 are simple text displays. Component1 is shown by default with a button.
When button is clicked, it displays Component2.
unUsedComponent is not really used anywhere.

- src
    - components
        - Component1.js
        - Component2.js
        - unUsedComponent.js
    - App.js
    - index.html
    - index.js
- webpack.config.js
- package.json
- package.lock.json

Component1.js

import React from 'react';
import PropTypes from 'prop-types';
import _ from 'lodash';

const Component1 = () => {
  let x = _.join(['C', 'o', 'm', 'p', 'o', 'n', 'e', 'n', 't', ' 1'], '');
  return <div>{x}</div>;
}

Component1.propTypes = {
  title: PropTypes.string,
  description: PropTypes.string,
  count: PropTypes.number,
};

export default Component1;

Component2.js

import React from 'react';

const Component2 = () => {
  return <div>Component 2</div>;
}

export default Component2;

const unUsedComponent = () => {
    console.log("Using-unUsedComponent");
    return (
    <div>unUsedComponent</div>
    )
}
export default unUsedComponent;

index.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8" />
    <title>Webpack Optimization</title>
  </head>
  <body>
    <div id="root"></div>
  </body>
</html>

App.js

import React, { useState } from 'react';
import Component1 from './components/Component1';
import Component2 from './components/Component2';
import unUsedComponent from './components/UnUsedComponent';

const App = () => {
  const [showComponent2, setShowComponent2] = useState(false);

  const handleClick = () => {
    setShowComponent2(true);
  }

  return (
    <div>
      <Component1 />
      <button onClick={handleClick}>Load Component 2</button>
      {showComponent2 && <React.Suspense fallback={<div>Loading...</div>}>
        <Component2 />
      </React.Suspense>}
    </div>
  );
}

export default App;

index.js

import React from 'react';
import ReactDOM from 'react-dom';
import App from './App';

ReactDOM.render(<App />, document.getElementById('root'));

webpack.config.js

const path = require('path');
const HtmlWebpackPlugin = require('html-webpack-plugin');
const webpack = require('webpack');

module.exports = {
  mode: 'none',
  entry: {
    main: { import: "./src/index.js" }
  },
  module: {
    rules: [
      {
        test: /\.(js|jsx)$/,
        exclude: /node_modules/,
        use: {
          loader: 'babel-loader',
          options: {
            presets: ['@babel/preset-env', '@babel/preset-react']
          }
        }
      }
    ]
  },
  plugins: [
    new HtmlWebpackPlugin({
      filename: "index.html",
      template: "./src/index.html",
    }),
    // fix "process is not defined" error:
    new webpack.ProvidePlugin({
      process: 'process/browser',
    }),
  ]
};

Starting State

Let's first build the assets and see if our starting set is all good.

Run npx webpack to build the assets:

webpack-optimization-demo % npx webpack
asset main.js 1.8 MiB [compared for emit] (name: main)
asset index.html 201 bytes [compared for emit]
runtime modules 1.25 KiB 6 modules
modules by path ./node_modules/ 1.78 MiB
  modules by path ./node_modules/prop-types/ 28.6 KiB 6 modules
  modules by path ./node_modules/react/ 92.5 KiB 3 modules
  modules by path ./node_modules/react-dom/ 1.11 MiB 3 modules
  modules by path ./node_modules/scheduler/ 21.4 KiB 3 modules
  modules by path ./node_modules/react-is/ 9.5 KiB 3 modules
  + 3 modules
modules by path ./src/ 3.27 KiB
  modules by path ./src/components/*.js 710 bytes
    ./src/components/Component1.js 342 bytes [built] [code generated]
    + 2 modules
  modules by path ./src/*.js 2.58 KiB
    ./src/index.js 181 bytes [built] [code generated]
    ./src/App.js 2.4 KiB [built] [code generated]
webpack 5.91.0 compiled successfully in 743 ms

Notice the bundle size as 1.8MiB (Your bundle sizes might be slightly different)

Let's also see how our page looks:

Run npx http-server -p 8000. Now, open your web browser, and open developer tools. Go to Network tab.

Now load http://localhost:8000/dist on your web browser. Your page should look like this:

Your network tab:

If we click on button for Loading Component 2, we see that no new network call was made even though Component 2 was loaded.

Now that our starting point is all good, we are going to look into how we can optimize this code.

At any point, feel free to complete delete the dist folder since it gets generated automatically when npx webpack is run.

Tree Shaking

Let's start with [Tree Shaking]{https://webpack.js.org/guides/tree-shaking/}.
It's a method to remove any unused modules.

unUsedComponent is not really used in the display. However, it is still present in the code:

If we go to dist/main.js. Search for Using-unUsedComponent, we see that it is present.

Let's start fixing this:

Add this in webpack.config.js under modules.exports and then run npx webpack:

optimization: {
    usedExports: true,
  },

webpack-optimization-demo2 % npx webpack
asset main.js 1.79 MiB [emitted] (name: main)
asset index.html 201 bytes [compared for emit]
runtime modules 1010 bytes 5 modules
modules by path ./node_modules/ 1.78 MiB
  modules by path ./node_modules/prop-types/ 28.6 KiB 6 modules
  modules by path ./node_modules/react/ 92.5 KiB 3 modules
  modules by path ./node_modules/react-dom/ 1.11 MiB 3 modules
  modules by path ./node_modules/scheduler/ 21.4 KiB 3 modules
  modules by path ./node_modules/react-is/ 9.5 KiB 3 modules
  + 3 modules
modules by path ./src/ 3.27 KiB
  modules by path ./src/components/*.js 710 bytes
    ./src/components/Component1.js 342 bytes [built] [code generated]
    + 2 modules
  modules by path ./src/*.js 2.58 KiB
    ./src/index.js 181 bytes [built] [code generated]
    ./src/App.js 2.4 KiB [built] [code generated]
webpack 5.91.0 compiled successfully in 735 ms

Not much change in file size. However, if we go to dist/main.js. Search for unUsedComponent, we see it is NOT present.

Just like that, we have removed an unused component we created.

Minification

Next, we will do some minification. Minification is a process to reduce the total size of file contents by removing characters and parts which are deemed useless for machines.
FOr example, spaces in codes are simply a way for them to be readable by humans. However, from machine point of view, these are harmless. Minification helps remove these additional spaces among other things. Let's use an external plugin to help minify further.

We will install a minimizer plugin::
npm install --save-dev terser-webpack-plugin

Now, add this in webpack.config.js under modules.exports and then run npx webpack:

const TerserPlugin = require('terser-webpack-plugin');
module.exports = {
  // ... other config
  optimization: {
    // ...
    minimize: true,
    minimizer: [new TerserPlugin()],
  },
}

webpack-optimization-demo % npx webpack
asset main.js 520 KiB [emitted] [minimized] (name: main) 1 related asset
asset index.html 201 bytes [compared for emit]
runtime modules 1010 bytes 5 modules
modules by path ./node_modules/ 1.78 MiB
  modules by path ./node_modules/prop-types/ 28.6 KiB 6 modules
  modules by path ./node_modules/react/ 92.5 KiB 3 modules
  modules by path ./node_modules/react-dom/ 1.11 MiB 3 modules
  modules by path ./node_modules/scheduler/ 21.4 KiB 3 modules
  modules by path ./node_modules/react-is/ 9.5 KiB 3 modules
  + 3 modules
modules by path ./src/ 3.27 KiB
  modules by path ./src/components/*.js 710 bytes
    ./src/components/Component1.js 342 bytes [built] [code generated]
    + 2 modules
  modules by path ./src/*.js 2.58 KiB
    ./src/index.js 181 bytes [built] [code generated]
    ./src/App.js 2.4 KiB [built] [code generated]
webpack 5.91.0 compiled successfully in 3011 ms

Notice the reduction in file size. From 1.79MiB to 520KiB. Around 70% savings in size.

Use only what is needed

Notice that in Component1.js, we only use join but we are instead importing the whole lodash library.
lodash has split each function in their own library file, so we can just download lodash/join instead of the whole things.

Make the following code change in Component1.js:

Component1.js

// ...
import join from 'lodash/join';
// ...
let x = join(['C', 'o', 'm', 'p', 'o', 'n', 'e', 'n', 't', ' 1'], '');

And then run npx webpack:

webpack-optimization-demo % npx webpack
asset main.js 451 KiB [emitted] [minimized] (name: main) 1 related asset
asset index.html 201 bytes [compared for emit]
runtime modules 786 bytes 4 modules
modules by path ./node_modules/ 1.27 MiB
  modules by path ./node_modules/prop-types/ 28.6 KiB 6 modules
  modules by path ./node_modules/react/ 92.5 KiB 3 modules
  modules by path ./node_modules/react-dom/ 1.11 MiB 3 modules
  modules by path ./node_modules/scheduler/ 21.4 KiB 3 modules
  modules by path ./node_modules/react-is/ 9.5 KiB 3 modules
  + 3 modules
modules by path ./src/ 3.34 KiB
  modules by path ./src/components/*.js 779 bytes
    ./src/components/Component1.js 411 bytes [built] [code generated]
    + 2 modules
  modules by path ./src/*.js 2.58 KiB
    ./src/index.js 181 bytes [built] [code generated]
    ./src/App.js 2.4 KiB [built] [code generated]
webpack 5.91.0 compiled successfully in 2543 ms

Notice the further reduction is asset size.

Code Splitting

Code Splitting is a way to split the bundled asset into multiple bundled assets, which can be loaded on demand, or in parallel.
We will see how to code split here, and take advantage of it in this and the next section.

Do demonstrate this better, let's create a new page which only loads Component1. Remember that Component1 uses lodash/join

Create 'NewApp.js' inside src, which only loads Component1:

NewApp.js

import React from 'react';
import Component1 from './components/Component1';

const AppNew = () => {
  return (
    <div>
      <Component1 />
    </div>
  );
}

export default AppNew;

Create 'newindex.js' inside src: newindex.js

import React from 'react';
import ReactDOM from 'react-dom';
import AppNew from './AppNew';

ReactDOM.render(<AppNew />, document.getElementById('root'));

Let's now update webpack.config.js to the following:
webpack.config.js

const path = require('path');
const HtmlWebpackPlugin = require('html-webpack-plugin');
const TerserPlugin = require('terser-webpack-plugin');
const webpack = require('webpack');

module.exports = {
  mode: 'none',
  entry: {
    main: { import: "./src/index.js", dependOn: "shared" },
    newindex: { import: "./src/newindex.js", dependOn: "shared" },
    shared: "lodash/join",
  },
  output: {
    filename: '[name].js',
    path: path.resolve(__dirname, 'dist'),
  },
  module: {
    rules: [
      {
        test: /\.(js|jsx)$/,
        exclude: /node_modules/,
        use: {
          loader: 'babel-loader',
          options: {
            presets: ['@babel/preset-env', '@babel/preset-react']
          }
        }
      }
    ]
  },
  plugins: [
    new HtmlWebpackPlugin({
      filename: "index.html",
      template: "./src/index.html",
      chunks: ["main", "shared"],
    }),
    new HtmlWebpackPlugin({
      filename: "newindex/index.html",
      template: "./src/index.html",
      chunks: ["newindex", "shared"],
    }),
    // fix "process is not defined" error:
    new webpack.ProvidePlugin({
      process: 'process/browser',
    }),
  ],
  optimization: {
    usedExports: true,
    minimize: true,
    minimizer: [new TerserPlugin()],
    splitChunks: {
      chunks: 'all',
    },
  },
};

Here, we are doing the following things:

Modified entry to have 2 new values, main & newindex
lodash/join is used in both pages, so we have taken out lodash/join into a new common shared asset
output specifies how would we name our new assets.
Add HtmlWebpackPlugin to be able to load newindex page
Added splitChunks in optimization

Let's run npx webpack again:

webpack-optimization-demo % npx webpack
assets by path *.js 453 KiB
  asset 3.js 449 KiB [compared for emit] [minimized] (id hint: vendors) 1 related asset
  asset main.js 2.14 KiB [compared for emit] [minimized] (name: main)
  asset shared.js 1.25 KiB [compared for emit] [minimized] (name: shared)
  asset newindex.js 736 bytes [compared for emit] [minimized] (name: newindex)
asset newindex/index.html 287 bytes [compared for emit]
asset index.html 274 bytes [compared for emit]
Entrypoint main 451 KiB = 3.js 449 KiB main.js 2.14 KiB
Entrypoint newindex 449 KiB = 3.js 449 KiB newindex.js 736 bytes
Entrypoint shared 1.25 KiB = shared.js
runtime modules 3.18 KiB 6 modules
modules by path ./node_modules/ 1.27 MiB
  modules by path ./node_modules/prop-types/ 28.6 KiB 6 modules
  modules by path ./node_modules/react/ 92.5 KiB 3 modules
  modules by path ./node_modules/react-dom/ 1.11 MiB 3 modules
  modules by path ./node_modules/scheduler/ 21.4 KiB 3 modules
  modules by path ./node_modules/react-is/ 9.5 KiB 3 modules
  + 3 modules
modules by path ./src/ 3.76 KiB
  modules by path ./src/*.js 3 KiB
    ./src/newindex.js 190 bytes [built] [code generated]
    + 3 modules
  modules by path ./src/components/*.js 779 bytes
    ./src/components/Component1.js 411 bytes [built] [code generated]
    + 2 modules
webpack 5.91.0 compiled successfully in 2620 ms

Notice how multiple different asset files are getting created, including one for shared.js which only contains lodash/join. We can find these files inside the dist/ folder.

At this point, let's run the server again and see how it all looks: npx http-server -p 8000:

If we open https://localhost:8000/dist, we see that newindex.js is not loaded:

If we open https://localhost:8000/dist/newindex, we see that main.js is not loaded:

Lazy loading

Lazy loading simply means that loading of an asset should happen only on demand.

In our case, we have seen that Component2 gets shown to user only when the button has clicked. However, it is loaded upfront. We are goin to load this only on demand.

Update App.js with following 2 changes:
App.js

import React, { useState, lazy } from 'react';
// ...
const Component2 = lazy(() => import('./components/Component2'));
// ...

Run npx webpack, we see a new asset (4.js in my case, it might be named differently for you) is created in the dist:

webpack-optimization-demo % npx webpack
assets by path *.js 455 KiB
  asset 3.js 449 KiB [compared for emit] [minimized] (id hint: vendors) 1 related asset
  asset shared.js 3.2 KiB [compared for emit] [minimized] (name: shared)
  asset main.js 2.06 KiB [compared for emit] [minimized] (name: main)
  asset newindex.js 736 bytes [compared for emit] [minimized] (name: newindex)
  asset 4.js 254 bytes [compared for emit] [minimized]
asset newindex/index.html 287 bytes [compared for emit]
asset index.html 274 bytes [compared for emit]
Entrypoint main 451 KiB = 3.js 449 KiB main.js 2.06 KiB
Entrypoint newindex 449 KiB = 3.js 449 KiB newindex.js 736 bytes
Entrypoint shared 3.2 KiB = shared.js
runtime modules 8.12 KiB 12 modules
modules by path ./node_modules/ 1.27 MiB
  modules by path ./node_modules/prop-types/ 28.6 KiB 6 modules
  modules by path ./node_modules/react/ 92.5 KiB 3 modules
  modules by path ./node_modules/react-dom/ 1.11 MiB 3 modules
  modules by path ./node_modules/scheduler/ 21.4 KiB 3 modules
  modules by path ./node_modules/react-is/ 9.5 KiB 3 modules
  + 3 modules
modules by path ./src/ 3.8 KiB
  modules by path ./src/*.js 3.04 KiB
    ./src/newindex.js 190 bytes [built] [code generated]
    + 3 modules
  modules by path ./src/components/*.js 779 bytes
    ./src/components/Component1.js 411 bytes [built] [code generated]
    + 2 modules
webpack 5.91.0 compiled successfully in 2529 ms

What is this asset ? Le's open it (Find it in dist/4.js).
We see that 4.js contains our Component2 code.

Let;s now run npx http-server -p 8000, open Developer Tools and go to Network Tab and open https://localhost:8000/dist

We see that 4.js is not yet loaded.
Click on the button Load Component 2 and notice that 4.js gets loaded now.

There you have it, Lazy (On-demand) Loading.

Production mode

We can provide different modes in the webpack config. We are currently running with none mode. We can specify development and production mode.
In development mode, the purpose is ease of development. In production mode, the purpose is to give small, fast, optimized assets.

Let's change the mode in:
webpack.config.js

// ...
mode: 'production',
// ...

Run npx webpack and notice how the asset size is reduced in size by a huge number.

webpack-optimization-demo % npx webpack
assets by path *.js 143 KiB
  asset 248.js 137 KiB [emitted] [minimized] (id hint: vendors) 1 related asset
  asset shared.js 3.13 KiB [emitted] [minimized] (name: shared)
  asset main.js 1.94 KiB [emitted] [minimized] (name: main)
  asset newindex.js 644 bytes [emitted] [minimized] (name: newindex)
  asset 860.js 248 bytes [emitted] [minimized]
asset newindex/index.html 282 bytes [emitted]
asset index.html 269 bytes [emitted]
Entrypoint main 139 KiB = 248.js 137 KiB main.js 1.94 KiB
Entrypoint newindex 137 KiB = 248.js 137 KiB newindex.js 644 bytes
Entrypoint shared 3.13 KiB = shared.js
runtime modules 8 KiB 11 modules
orphan modules 2.88 KiB [orphan] 3 modules
cacheable modules 148 KiB
  modules by path ./node_modules/ 144 KiB
    modules by path ./node_modules/prop-types/ 2.6 KiB 3 modules
    modules by path ./node_modules/react/ 6.95 KiB 2 modules
    modules by path ./node_modules/react-dom/ 130 KiB 2 modules
    modules by path ./node_modules/scheduler/ 4.33 KiB 2 modules
    + 1 module
  modules by path ./src/ 3.61 KiB
    modules by path ./src/*.js 3.04 KiB 2 modules
    modules by path ./src/components/*.js 579 bytes
      ./src/components/Component1.js 411 bytes [built] [code generated]
      ./src/components/Component2.js 168 bytes [built] [code generated]
webpack 5.91.0 compiled successfully in 1664 ms

Compression

We can also compress the files further to ensure a smaller asset is transferred over the network.

Run npm install compression-webpack-plugin --save-dev, and then run npx webpack.

webpack-optimization-demo % npx webpack
assets by path *.js 143 KiB
  assets by status 5.7 KiB [emitted]
    asset shared.js 3.13 KiB [emitted] [minimized] (name: shared) 1 related asset
    asset main.js 1.94 KiB [emitted] [minimized] (name: main) 1 related asset
    asset newindex.js 644 bytes [emitted] [minimized] (name: newindex) 1 related asset
  assets by status 137 KiB [compared for emit]
    asset 248.js 137 KiB [compared for emit] [minimized] (id hint: vendors) 2 related assets
    asset 860.js 248 bytes [compared for emit] [minimized] 1 related asset
asset newindex/index.html 282 bytes [emitted] 1 related asset
asset index.html 269 bytes [emitted] 1 related asset
Entrypoint main 139 KiB = 248.js 137 KiB main.js 1.94 KiB
Entrypoint newindex 137 KiB = 248.js 137 KiB newindex.js 644 bytes
Entrypoint shared 3.13 KiB = shared.js
runtime modules 8 KiB 11 modules
orphan modules 2.88 KiB [orphan] 3 modules
cacheable modules 148 KiB
  modules by path ./node_modules/ 144 KiB
    modules by path ./node_modules/prop-types/ 2.6 KiB 3 modules
    modules by path ./node_modules/react/ 6.95 KiB 2 modules
    modules by path ./node_modules/react-dom/ 130 KiB 2 modules
    modules by path ./node_modules/scheduler/ 4.33 KiB 2 modules
    + 1 module
  modules by path ./src/ 3.61 KiB
    modules by path ./src/*.js 3.04 KiB 2 modules
    modules by path ./src/components/*.js 579 bytes
      ./src/components/Component1.js 411 bytes [built] [code generated]
      ./src/components/Component2.js 168 bytes [built] [code generated]
webpack 5.91.0 compiled successfully in 1656 ms

There is no difference in this small example. But the difference can be more significant in larger projects. (It's also possible the compression done in the production mode is comparable in this case, so does not make any difference here).

Other options

There are still many other steps we can take like External Dependency where we loaded common large dependencies during run-time from CDN, Purge Unsued CSS, Module Concatenation and many more, depending on our specific needs. I will not go into it for the sake of keeping this article consumeable.

Bundle analyzer

I will however mention one very useful plugin Webpack Bundle Analyzer. It graphically helps us visualize which modules are taking how much space and where we should look to optimize. It's also interactive.

npm install --save-dev webpack-bundle-analyzer

Update the webpack.config.js:
webpack.config.js

// ...
const BundleAnalyzerPlugin = require('webpack-bundle-analyzer').BundleAnalyzerPlugin;
// ...
module.exports = {
    // ...
  plugins: [
    // ...
    new BundleAnalyzerPlugin(),
  ],
  // ...
}

Run npx webpack and then open http://localhost:8888/ and we should see something like this:

Conclusion

Optimizing the bundle size of any web application is essential for ensuring a seamless and performant user experience. By leveraging the power of webpack and its optimization techniques, we can significantly reduce the amount of data transferred over the network, leading to faster load times and improved overall performance. Continuously monitoring and refining the webpack configuration will help in maintaining a high-performing and responsive application over time.

References

This article was originally published in my personal blog

Practical Experiments for Optimizing Django query with the power of SQL joins

Wewake — Sat, 27 Apr 2024 03:42:12 +0000

Introduction
Concepts
Practical
- Project Setup
- Experiment with a small number of records
- Experiment with a larger number of records

Introduction

Python is a remarkable programming language - it's simple, powerful, and abstracts away several complicated internal concepts, which makes it easy to code with. Django framework extends this simple powerful nature of python to the web.

However, the very abstraction and simplicity that make Python and Django so appealing can sometimes lead developers, especially those without formal computer science training, to overlook (or maybe just forget about) the intricacies of how things work under the hood.

One concept that I have often seen several developers overlook in their professional lives is the concept of joins in database queries. This might not be apparent on a small scale, but when handling vast amounts of data, the programs can become unweildy.

We will walk through how a simple optimization concepts can make a huge difference as the scale increases.

Concepts

Django's ORM is tightly coupled with the SQL (relational) tables. Each Model defined in Django has a correponding table defined in SQL. When we query in Django Model, we are actually querying the corresponnding SQL table. So, it is obvious that any optimization techniques used in SQL would also be beneficial when querying Django models.

Another important point to note is that making a SQL call (i.e. a Database call) is expensive. So, we want to minimize the number of calls to the Database.

We assume you already know what SQL joins are. Informally, SQL joins allows you to query 2 or more related tables faster. To read more, you can checkout SQL JOIN .

As we would expect, Django has corresponding methods to optimize queries:

select_related

Performs a SQL JOIN to fetch related data in the same query.
Suitable for one-to-one and one-to-many relationships.
More efficient than separate queries when you know you'll need the related data.

prefetch_related:

Uses separate queries to fetch related data, but batches the queries to minimize database hits.
Suitable for many-to-many and many-to-one relationships.
More efficient than separate queries when you'll need the related data.

In summary, select_related is optimized for one-to-one and one-to-many relationships, while prefetch_related is optimized for many-to-many and many-to-one relationships. # The choice depends on the type of relationship and the likelihood of needing the related data.

Practical

Project Setup

Let's set up a very simple Django project so that we can run our experiments.

1.Assuming you have python3 installed, run the following to start virtualenv and install Django :

python3 -m venv venv
source venv/bin/activate
pip install Django
pip install Faker

If you don't want to go through the setup, you can clone the
repository and go to the example directory: cd django_join after the above step. And then continue reading from Experiment 1

2.Run the following to start a new project named django-join and create an app/module inside it called myapp :

django-admin startproject django_join
django-admin startapp myapp

3.Now, enable the myapp by including it under INSTALLED_APPS in settings.py :

vim django_join/settings.py

Here:


...
INSTALLED_APPS = [
    'myapp', # ADD YOUR APP HERE

    # EXISTING MODELS
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]
...

4.Create New Django models in myapp/models.py:

from django.db import models
from django.core.validators import MaxValueValidator

# Create your models here.
class Author(models.Model):
    name = models.CharField(max_length=100)
    age = models.IntegerField(validators=[MaxValueValidator(100)])

class Book(models.Model):
    title = models.CharField(max_length=200)
    author = models.ForeignKey(Author, on_delete=models.CASCADE)

5.Run migration commands to create the model tables.

python manage.py makemigrations myapp
python manage.py migrate

6.Let's also create a command to populate our tables with random data.

Install Faker library:

pip install Faker

Create the file in this location (Create subfolders as needed):

vim myapp/management/commands/populate_data.py

Paste the following. This basically generates random data given author count and max number of books for an author.

from django.core.management.base import BaseCommand
from faker import Faker
from myapp.models import Author, Book
import random

class Command(BaseCommand):
    help = 'Populates the database with random authors and books'

    def add_arguments(self, parser):
        parser.add_argument('authors', type=int, help='The number of authors to create')
        parser.add_argument('books', type=int, help='The number of books to create per author')

    def handle(self, *args, **kwargs):
        faker = Faker()

        authors_count = kwargs['authors']
        max_books_per_author = kwargs['books']

        for _ in range(authors_count):
            author = Author.objects.create(name=faker.name())
            # Generate random number of books upto max_count
            book_count_for_this_author = random.randint(1, books_per_author)
            for _ in range(book_count_for_this_author):
                Book.objects.create(title=faker.sentence(), author=author)

        self.stdout.write(self.style.SUCCESS(f'Successfully added {authors_count} authors and {authors_count * books_per_author} books'))

7.Create another file to run the 1st experiment. This experiment will check the difference in number of queries.

vim myapp/management/commands/get_db_query_count.py

Paste the following content:

from django.db import connection, reset_queries
from myapp.models import Author, Book

# Measuring for select_related
reset_queries()
books = Book.objects.select_related('author').all()
for book in books:
    temp = book.author.name
print(f"Queries with select_related: {len(connection.queries)}")

reset_queries()
books = Book.objects.all()
for book in books:
    temp = book.author.name
print(f"Queries without select_related: {len(connection.queries)}")

# Measuring for prefetch_related
reset_queries()
authors = Author.objects.prefetch_related('book_set').all()
for author in authors:
    temp = author.book_set.count()
print(f"Queries with prefetch_related: {len(connection.queries)}")

reset_queries()
authors = Author.objects.all()
for author in authors:
    temp = author.book_set.count()
print(f"Queries without prefetch_related: {len(connection.queries)}")

8.Create another file to run the 2nd experiment. This experiment will measure the time difference while running the commands.

vim myapp/management/commands/get_time_difference.py

Paste the following content:

from django.core.management.base import BaseCommand
from django.db import connection, reset_queries
from django.db.models import Count
from myapp.models import Author, Book

import timeit

class Command(BaseCommand):
    help = 'Measure average time taken to run commands. It gives average run time of 10 commands.'

    def handle(self, *args, **kwargs):
        # With prefetch_related
        execution_time1 = timeit.timeit('get_total_book_count_with_prefetch_related()', globals=globals(), number=10) / 10
        print(f"Avg time with prefetch_related: {execution_time1}")

        # Without prefetch_related
        execution_time2 = timeit.timeit('get_total_book_count_without_prefetch_related()', globals=globals(), number=10) / 10
        print(f"Avg time without prefetch_related: {execution_time2}")

        print("prefetch_related performance improvement RATIO:", execution_time2 / execution_time1)

        print()
        # With select_related
        execution_time3 = timeit.timeit('get_total_book_count_with_select_related()', globals=globals(), number=10) / 10
        print(f"Avg time with select_related: {execution_time3}")

        # Without select_related
        execution_time4 = timeit.timeit('get_total_book_count_without_select_related()', globals=globals(), number=10) / 10
        print(f"Avg time without select_related: {execution_time4}")

        print("select_related performance improvement RATIO:", execution_time4 / execution_time3)



# Without prefetch_related
def get_total_book_count_without_prefetch_related():
    authors = Author.objects.all()
    count = 0
    for author in authors:
        count += author.book_set.count()
    # print(count)

# With prefetch_related
def get_total_book_count_with_prefetch_related():
    authors = Author.objects.prefetch_related('book_set').all()
    count = 0
    for author in authors:
        count += author.book_set.count()
    # print(count)

# Without select_related
def get_total_book_count_without_select_related():
    books = Book.objects.all()
    total_age = 0 
    for book in books:
        total_age += book.author.age
    # print(total_age // books.count())

# With select_related
def get_total_book_count_with_select_related():
    books = Book.objects.select_related('author').all()
    total_age = 0
    for book in books:
        total_age += book.author.age
    # print(total_age // books.count())

Experiment with a small number of records

We will first populate a small dataset, and then run the commands to get db query count and time.

(venv) viv1@Viveks-MacBook-Pro django_join % python manage.py populate_data 5 5
Successfully added 5 authors and 18 books

(venv) viv1@Viveks-MacBook-Pro django_join % python manage.py get_db_query_count
Queries with prefetch_related: 2
Queries without prefetch_related: 6
Additional Queries: 4

Queries with select_related: 1
Queries without select_related: 19
Additional Queries: 18
(venv) viv1@Viveks-MacBook-Pro django_join % python manage.py get_time_difference
Avg time with prefetch_related: 0.0008449875000000023
Avg time without prefetch_related: 0.0017883291999999996
prefetch_related performance improvement RATIO: 2.1163972248110117

Avg time with select_related: 0.00023677920000000075
Avg time without select_related: 0.0030041750000000004
select_related performance improvement RATIO: 12.68766428807932

Even with the small amount of records, we can see the performance improvement in terms of time and the number of db calls. In fact, what we are observing in the query part is the famous N+1 query problem , where there is N number of DB calls for each of the N records. With this optimization, we can reduce it to a single DB call.

Experiment with a larger number of records

(venv) viv1@Viveks-MacBook-Pro django_join % python manage.py populate_data 100 10
Successfully added 100 authors and 549 books
(venv) viv1@Viveks-MacBook-Pro django_join % python manage.py get_db_query_count
Queries with prefetch_related: 2
Queries without prefetch_related: 106
Additional Queries: 104

Queries with select_related: 1
Queries without select_related: 568
Additional Queries: 567
(venv) viv1@Viveks-MacBook-Pro django_join % python manage.py get_time_difference
Avg time with prefetch_related: 0.0058029208
Avg time without prefetch_related: 0.029672083399999993
prefetch_related performance improvement RATIO: 5.113301460188806

Avg time with select_related: 0.003976683299999994
Avg time without select_related: 0.099798425
select_related performance improvement RATIO: 25.09589461147186

With just a bit higher number of records, we can see the performance improvement is even higher.

Conclusion

Optimizing database queries is crucial for achieving high performance and scalability in Django applications. Being aware of very simple concepts like joins can make our application faster, more responsive and efficient, especially when dealing with large datasets and complex relationships between models.

References

prefetch_related

select_related

This article was originally published on my blog at https://wewake.dev/posts/practical-experiments-for-django-orm-query-optimizations/

Find more of my content of my blog, where I share my software development experiences and learnings: https://wewake.dev

Solving Coding Problems using the Inclusion-Exclusion Principle and its variations

Wewake — Mon, 15 Apr 2024 17:29:10 +0000

Problem Statement
Solving the inverse problem
Inclusion Exclusion Principle
Variations
- Variation 1: Counting Sequences with Required Elements
- Variation 2: Number of upper-bound integer sums
- Variation 3: Leetcode Problem Ugly Number III
- Other Variations
Conclusion
References and Further Reading
Appendix

Problem Statement

Let's start with the problem statement:

Count the number of permutations of digits from 0 to 9 such that the first digit is greater than 1, and the last digit is less than 8.

Solving the inverse problem

We will try to solve for the inverse problem instead. i.e. finding all permutations where the first digit is greater than 1, OR the last digit is less than 8

If we define X and Y as follows:

X: The set of permutations where the first digit is less than or equal to 1.
Y: The set of permutations where the last digit is greater than or equal to 8.

From venn diagram:

|X ∪ Y|

|X ∩ Y|:

it follows:

i.e.

|X ∪ Y| : Count of permutations with "first digit is less than or equal to 1" OR "last digit is greater than or equal to 8".

|X ∩ Y|: Count of permutations with "first digit is less than or equal to 1" AND "last digit is greater than or equal to 8".

|X ∪ Y| : Represents all the bad permutations.
This is the solution to the "inverse" problem.

So,
Our answer = All Permutations of [O, 9]  - All Bad permutations
=> answer = All Permutations of [O, 9] - |X ∪ Y|

Since, `|X ∪ Y| = |X| + |Y| - |X ∩ Y|`

Answer = `Total Permutations of [O, 9] - (|X| + |Y| - |X ∩ Y|)`

Here is the code:

from math import factorial

def count_permutations_ie():
    total_permutations = factorial(10)
    bad_first_digit = 2 * factorial(9)  # 2 options (0 or 1) for the first digit
    bad_last_digit = 2 * factorial(9)   # 2 options (8 or 9) for the last digit
    bad_first_and_last = 2 * 2 * factorial(8)  # 2 options for first, 2 for last

    bad_permutations = bad_first_digit + bad_last_digit - bad_first_and_last
    good_permutations = total_permutations - bad_permutations

    return good_permutations

print(f"Number of permutations (IE): {count_permutations_ie()}")

This can also be solved via dynamic programming. See section Appendix.

Inclusion Exclusion Principle

What we observed above was basically the Inclusion Exclusion Principle.

To find |X ∪ Y|, we used |X| + |Y| - |X ∩ Y|.

Here is a more complete definition:

The Inclusion-Exclusion Principle is a counting technique that allows us to find the cardinality (size) of the union of multiple sets by considering their individual cardinalities, their intersections, and their complements. The principle states that:

|A_1 ∪ A_2 ∪ ... ∪ A_n| = Σ(|A_i|) - Σ(|A_i ∩ A_j|) + Σ(|A_i ∩ A_j ∩ A_k|) - ... + (-1)^(n-1) * |A_1 ∩ A_2 ∩ ... ∩ A_n|

In simpler terms, the principle says that to find the cardinality of the union of sets, we need to:

Add the cardinalities of all the individual sets.
Subtract the cardinalities of all pairwise intersections of the sets.
Add the cardinalities of all triple intersections of the sets.
Subtract the cardinalities of all quadruple intersections of the sets, and so on, alternating the signs.

This formula might seem intimidating at first, but it becomes more intuitive with examples and practice.

Variations

Variation 1: Counting Sequences with Required Elements

Consider a problem where we need to count the number of sequences of length n consisting of only 0, 1, and 2, such that each number occurs at least once.

We can represent the set of "bad" sequences (those that don't satisfy the constraint) as the union of three sets:

A_0: The set of sequences that don't contain 0.
A_1: The set of sequences that don't contain 1.
A_2: The set of sequences that don't contain 2.
Using the Inclusion-Exclusion Principle, we can find the cardinality of the "bad" sequences as:

|A_0 ∪ A_1 ∪ A_2| = |A_0| + |A_1| + |A_2| - |A_0 ∩ A_1| - |A_0 ∩ A_2| - |A_1 ∩ A_2| + |A_0 ∩ A_1 ∩ A_2|

We can calculate the cardinalities as follows:

def count_sequences(n):
    total_sequences = 3 ** n
    bad_sequences = 3 * (2 ** n) - 3 * (1) + 0
    good_sequences = total_sequences - bad_sequences
    return good_sequences

n = 5
print(f"Number of good sequences of length {n}: {count_sequences(n)}")

This code will output the number of sequences of length n that contain all three elements (0, 1, and 2) at least once.

Variation 2: Number of upper-bound integer sums

For x1 + x2 + x3 + x4 + x5 + x6 = 20, where 0 <= xi <= 8, find the number of solutions.

Let's define the set of "good" solutions as the set of all tuples (x1, x2, x3, x4, x5, x6) that satisfy the equation x1 + x2 + x3 + x4 + x5 + x6 = 20 and the constraint 0 ≤ xi ≤ 8 for all i.
Also, let's define the set of "bad" solutions as the set of tuples that violate the constraint 0 ≤ xi ≤ 8 for at least one value of i.

Now, we can use the Inclusion-Exclusion Principle to count the number of "bad" solutions and then subtract it from the total number of solutions to get the count of "good" solutions.

Here is the code:

from functools import lru_cache

@lru_cache(maxsize=None)
def count_solutions(target, num_vars, lower_bound=0, upper_bound=8):
    """
    Count the number of solutions to the equation x1 + x2 + ... + xn = target,
    where lower_bound <= xi <= upper_bound for all i.
    """
    if num_vars == 1:
        return 1 if lower_bound <= target <= upper_bound else 0

    solutions = 0
    for x in range(lower_bound, upper_bound + 1):
        remaining_target = target - x
        solutions += count_solutions(remaining_target, num_vars - 1, lower_bound, upper_bound)

    return solutions

def count_solutions_with_inclusion_exclusion(target, num_vars, lower_bound=0, upper_bound=8):
    total_solutions = count_solutions(target, num_vars, 0, upper_bound)
    bad_solutions = 0

    for k in range(1, num_vars + 1):
        invalid_values = [0] * k + [-1] * (num_vars - k)
        bad_solutions += (-1) ** k * count_solutions(target, num_vars, lower_bound, upper_bound - 1, invalid_values)

    return total_solutions - bad_solutions

# Example usage
target = 20
num_vars = 6
print(f"Number of solutions: {count_solutions_with_inclusion_exclusion(target, num_vars)}")

Variation 3: Leetcode Problem Ugly Number III

This is a bit advanced. Here, the Inclusion Exclusion Principle helps us break down the problem so that we can apply binary search, and find the solution.

An ugly number is a positive integer that is divisible by a, b, OR c.

Given four integers n, a, b, and c, return the nth ugly number.

Basically, we can find count of such ugly numbers from 1...N by doing this:

+ count of numbers from 1..n which are divisible by a
+ count of numbers from 1..n which are divisible by b
+ count of numbers from 1..n which are divisible by c
- count of numbers from 1..n which are divisible by a AND b
- count of numbers from 1..n which are divisible by b AND c
- count of numbers from 1..n which are divisible by a AND c
+ count of numbers from 1..n which are divisible by a,b AND c

Let's call this f(K). Now the task is to find out the least K such that f(K) >= N.
We can do a binary search.

class Solution:
    def gcd(self, a, b):
        if b == 0:
            return a
        return self.gcd(b, a%b)

    def lcm(self, a, b):
        return (a * b) // self.gcd(a,b)

    def uglyCountFrom1toN(self, a, b, c, n):
        return n // a + n // b + n // c - n // self.lcm(a,b) - n // self.lcm(b,c) - n // self.lcm(a,c) + n // self.lcm(a,self.lcm(b,c))

    def nthUglyNumber(self, n: int, a: int, b: int, c: int) -> int:
        lo = min(a,b,c)
        hi = max(a*b*c, 2*pow(10,9))
        while lo < hi:
            mid = lo + (hi-lo)//2
            count = self.uglyCountFrom1toN(a,b,c,mid)
            # print(lo, hi, mid, count, n)
            if count < n:
                lo = mid+1
            else:
                hi = mid
            # print(lo, hi)
        return lo

Other Variations

You can find several other variations in this wonderful detailed article.

Conclusion

The Inclusion-Exclusion Principle is a powerful combinatorics technique that can be applied to a wide range of counting problems. While the principle's formula might seem daunting at first, its systematic approach can simplify complex counting scenarios by breaking them down into manageable parts.

By understanding the Inclusion-Exclusion Principle and practicing its application, developers can enhance their problem-solving skills and tackle intricate counting problems more effectively. Whether you're working on algorithms, probability theory, or combinatorics-related challenges, mastering this technique can be a valuable asset in your programming toolbox.

References and Further Reading

Appendix

Can this be solved using Dynamic Programming ?
Yes.
Let's define dp[i][j] that stores the count of permutations of length i where the first digit is j.

from math import factorial
def count_permutations_dp(n):
    dp = [[0] * 10 for _ in range(n + 1)]

    # Base case: For permutations of length 1, there's 1 permutation for each digit
    for j in range(10):
        dp[1][j] = 1

    # Fill the dp table
    for i in range(2, n + 1):
        for j in range(10):
            if j > 1:  # Include permutations where the first digit is greater than 1
                dp[i][j] = 9 * dp[i - 1][j] + factorial(9)
            else:
                dp[i][j] = 8 * dp[i - 1][j]

    # Calculate the desired result
    result = sum(dp[n][j] for j in range(2, 8))
    return result

print(f"Number of permutations (DP): {count_permutations_dp(10)}")

Time complexity: O(n * 10)
Space complexity:O(n * 10), where n is the number of digits (10 in this case).

This article was originally published on my blog at https://wewake.dev/posts/inclusion-exclusion-principle/

Find more of my content of my blog, where I share my software development experiences and learnings: https://wewake.dev