Deep drive into driver load mechanism of JDBC

JDBC is the classic API in Java for database connectivity (it stands for Java Database Connectivity api), and every good database driver follow the standart.

It encompasses many years of history and evolution, that today, the driver detection and loading is almost magical, because everything happens behind the scenes.

Now, I want to address how this magical thing works, and how you can mimic the mechanism to build good plugin frameworks with it.

The classloading process

First of all, we need to understand that the JVM load classes lazily.

It means that, first, the JVM bootstrap some basic classes, using the bootstrap classloader, and then, execute the main method of the application. Let's remember that every application needs to have a main method, in any language. Maybe you don't see a method named main, but it exists, even behind the scenes.

Well, JVM only can read bytecode instruction, so the Java Compiler will enforce some type and rule checking and procceed to transform all your source .java files into .class files.

The .class files are similar to .exes or .elfs, even .sos or .dlls. They contain low level instructions that tells the target machine what it will execute, and the machine will read and execute every instruction almost sequentially (it's not strictly sequentially, because of processing unit semantics, like instruction reordering, branch prediction, memory layout reordering, etc. They are optimizations, and JVM has a lot of them).

So, remember that JVM read classes lazily. It is more efficient to do that because, lets supose the JVM read all of them eagerly. An average application have, almost, 100k classes. I am taking the fact that, not only the application code is loaded, but library (and standart library too) code is loaded. So, when the JVM find an instruction that deals with an unknown class, it asks the Application Classloader to load the class, that way it can knows how the object is, and which code it should execute.

Now that the JVM load the class the first time, it keeps them in memory.

Execution blocks

But, the life is not that simple.
Classes are the only top level construct allowed in Java, so in JVM. As an example, Scala generates class files for top level functions because JVM don't supports no-class constructs (your processor don't supports structs, so, they only purpose it's to be a shape for how the data will be disposed in some memory segment).

The following scala code:

@main
def FrenchDate: Unit =
  val now = LocalDate.now
  val df = getFormatter(Locale.FRANCE)
  println(df.format(now))

def getFormatter(locale: Locale): DateTimeFormatter =
  DateTimeFormatter.ofLocalizedDate(FormatStyle.LONG).withLocale(locale)

Will generate the following class with main method:

public final class frenchDate {
   public static void main(final String[] args) {
      try {
         .MODULE$.frenchDate();
      } catch (ParseError var2) {
         scala.util.CommandLineParser..MODULE$.showError(var2);
      }

   }
}

That references the following class, with our methods:

public final class FrenchDate$package$ implements Serializable {
   public static final FrenchDate$package$ MODULE$ = new FrenchDate$package$();

   private FrenchDate$package$() {
   }

   private Object writeReplace() {
      return new ModuleSerializationProxy(FrenchDate$package$.class);
   }

   public void frenchDate() {
      LocalDate now = LocalDate.now();
      DateTimeFormatter df = this.getFormatter(Locale.FRANCE);
      .MODULE$.println(df.format(now));
   }

   public DateTimeFormatter getFormatter(final Locale locale) {
      return DateTimeFormatter.ofLocalizedDate(FormatStyle.LONG).withLocale(locale);
   }
}

And that said, we have a third class that will represent the Scala package (that haves different semantic than Java packages), but it's not the focus here.

Since the JVM only supports classes as top level construct, what if I need to run some code when a class is loaded? Or when an object is instatiated.

Well, that are some initialization blocks in Java classes that provide me that functionality.

First, we have static initializations blocks:

static {
 // your code here
}

This block is executed once, when the class is loaded by the JVM.

The next is an initialization block:

class Smartphone {

  {
    // this is a initialization block
  }

  Smartphone() {
    // no-args constructor
  }
}

An initialization block is executed when the object is instantiated, before the constructor.

After that we have the constructor, and the methods. Methods are executed everytime they are called by another code. Static methods follows this logic too. Constructors are normal methods, but they don't have any return value, because the new object reference is implicitly return by them. But, they allocate some memory to accomodate the object in it.

Driver Manager

With the basic knowledge in our hands, let's start by investigating the initial class of JDBC: the DriverManager.

Everything starts by getting the connection from DriverManager where we pass the parameters to it as properties optionally:

val url = "jdbc:derby://localhost:1527/jaLiDb"

def getConnection: Connection =
  DriverManager.getConnection(url)

However, we should note that we didn't specificate any driver for it. The method getConnection have a documentation that states:

Attempts to establish a connection to the given database URL. The DriverManager attempts to select an appropriate driver from the set of registered JDBC drivers.

So, we should know that, for JDBC understand and be able to select some driver, the driver should be registered to JDBC in the application.

The next question is, how is the process of driver register?
Let's take a look into the DriverManager class, but we gonna look closely into its static initialization block:

   static {
        loadInitialDrivers();
        println("JDBC DriverManager initialized");
    }

So, when DriverManager is first referenced in the application, it gonna execute this code. The second line is trivial, and will only display the message on console.

But the first method is very interesting.

Loading the drivers

Ok, let's go through the method, but I'll advice you: we gonna take a ride into history.

This method is very short, and really easy to understand.

First of all, how the list of registered drivers is maded of? First by getting a system property:

     String drivers;
        try {
            drivers = AccessController.doPrivileged(new PrivilegedAction<String>() {
                public String run() {
                    return System.getProperty("jdbc.drivers");
                }
            });
        } catch (Exception ex) {
            drivers = null;
        }

So, when these lines is runned, the code inside the Runnable will run with privileged permissions. Let's remember the so forgeted SecurityManager that manages the permissions of what the code can or cannot access, create, modify, etc.

What these lines do is simple: recover what is referenced by the -Djdbc.drivers method.

This a very legacy approach, when no magic was invented yet, the way a user have to register the driver was by, either, loading explicitly the class:

Class.forName("oracle.jdbc.driver.OracleDriver");

Or passing as a system property:

java -Djdbc.drivers=oracle.jdbc.driver.OracleDriver

So, when it catch the drivers and it have any valid classname, it makes the same process we already do manually:

 println("DriverManager.initialize: jdbc.drivers = " + drivers);

        if (drivers == null || drivers.equals("")) {
            return;
        }
        String[] driversList = drivers.split(":");
        println("number of Drivers:" + driversList.length);
        for (String aDriver : driversList) {
            try {
                println("DriverManager.Initialize: loading " + aDriver);
                Class.forName(aDriver, true,
                        ClassLoader.getSystemClassLoader());
            } catch (Exception ex) {
                println("DriverManager.Initialize: load failed: " + ex);
            }
        }

We skipped some lines of code, but don't worry, it only to show the first legacy primitive mechanism to load a driver.

Service Provider Interface

Well, the other approach is by loading the drivers using the Service Provider Interface (SPI).

The SPI provides some hack to load classes that are declared as a resource file that implements some type of interface. The fact is, every JDCB Driver implements the Driver interface. So it's very simple to make this using SPI.

In the same DriverManager.loadInitialDrivers method, we skipped the following lines of code:

 AccessController.doPrivileged(new PrivilegedAction<Void>() {
            public Void run() {

                ServiceLoader<Driver> loadedDrivers = ServiceLoader.load(Driver.class);
                Iterator<Driver> driversIterator = loadedDrivers.iterator();

                /* Load these drivers, so that they can be instantiated.
                 * It may be the case that the driver class may not be there
                 * i.e. there may be a packaged driver with the service class
                 * as implementation of java.sql.Driver but the actual class
                 * may be missing. In that case a java.util.ServiceConfigurationError
                 * will be thrown at runtime by the VM trying to locate
                 * and load the service.
                 *
                 * Adding a try catch block to catch those runtime errors
                 * if driver not available in classpath but it's
                 * packaged as service and that service is there in classpath.
                 */
                try{
                    while(driversIterator.hasNext()) {
                        driversIterator.next();
                    }
                } catch(Throwable t) {
                // Do nothing
                }
                return null;
            }
        });

Look how it's very simple. It loads the classes using the SPI, and then, iterate over them to allow it be catchable by the classloader. When the classloader see the references, it gonna load the class to register the driver, and that is the true magic for loading JDBC drivers by only putting them on classpath.

Driver registration: a two-way process

So, we already see how the drivers are loaded automatically and manually, but the register process are unseen in this context. So, they are only needed to be loaded and that's it? Not that really.

The first part of driver registration is load the Driver implementation on classpath. The second part happens on the side of Driver implementation.

Let's take as an example the PostgreSQL Driver implementation.

In every Driver, is supposed to have a static initialization block too, that will run when the class is loaded:

 static {
    try {
      // moved the registerDriver from the constructor to here
      // because some clients call the driver themselves (I know, as
      // my early jdbc work did - and that was based on other examples).
      // Placing it here, means that the driver is registered once only.
      register();
    } catch (SQLException e) {
      throw new ExceptionInInitializerError(e);
    }
  }

As the comment make us know, some drivers can make a choice to put the registering code on the driver constructor. But, what we are looking for is the method register:

  public static void register() throws SQLException {
    if (isRegistered()) {
      throw new IllegalStateException(
          "Driver is already registered. It can only be registered once.");
    }
    Driver registeredDriver = new Driver();
    DriverManager.registerDriver(registeredDriver);
    Driver.registeredDriver = registeredDriver;
  }

This method will make the following things:

1. Check if the driver is already registered, to make guarantees that it will registered only once;
2. Instantiate the driver;
3. Calls DriverManager to register the driver

And now, the only thing DriverManager will do is put the driver on a list. And now the driver is fully registered:

 public static synchronized void registerDriver(java.sql.Driver driver,
            DriverAction da)
        throws SQLException {

        /* Register the driver if it has not already been added to our list */
        if(driver != null) {
            registeredDrivers.addIfAbsent(new DriverInfo(driver, da));
        } else {
            // This is for compatibility with the original DriverManager
            throw new NullPointerException();
        }

        println("registerDriver: " + driver);

    }