DEV Community: Enrique Zamudio

j8583 version 2.0

Enrique Zamudio — Tue, 29 Nov 2022 05:43:15 +0000

It's been a long time since I've done any work on the j8583 library. Mostly because it works, but I have received requests for improvements, which I haven't worked on because I just don't have the time for it.

But, I think it's high time for a major release. Version 2.0 of j8583 will contain the following features:

First and foremost, Java 8. Or, more likely, Java 11. I mean, Java 7's been dead for a while, and while I understand that Enterprise environments are not using the most recent Java versions, I think even 8 is really old now.
What does it mean to use Java 8? Well, CustomEncoder and CustomDecoder can now be functional interfaces. IsoMessage can have accessors for fields and values that return an Optional, and, most importantly...
Support for the new Java Date/Time API. The fields of type DATEx, DATE_EXP and TIME should use the new classes such as LocalDate, ZonedDateTime and OffsetTime. Obviously backwards compatibility with the old Date API needs to be maintained, so support will have to be programmatically enabled at first.
Internally, upgrade to junit 5 and Gradle 7.

If things go well, the release might be ready before the year ends.

Useful comments in your code

Enrique Zamudio — Thu, 10 Dec 2020 10:10:42 +0000

It's your first day at your new job. You get assigned a task from the backlog, which requires you to read tons of code so you can fix a small bug and therefore is an excellent way to become familiar with the codebase.

The bug is somewhere in the gardening subsystem, so you start looking at the code gardening modules.

You run into this:

/** The model for flowers */
interface FlowerModel {
}

And also this:

/** Represents a flower */
class Flower {
}

You keep digging and find out that the codebase is full of these self-referential useless pseudo-tautologies, which tell you absolutely nothing.

Wouldn't you find this infuriating if it happened to you? And yet, so many people do this. If you look at the git log for those files, you can probably glimpse a story more or less like this:

Code was committed with zero comments.
Reviewer asked for javadoc.
These comments were begrudgingly added.
Reviewer didn't care enough to ask for real, useful documentation and they were all in a hurry to deploy this so it was considered acceptable, or a promise was made to properly document that code later and of course that day never came.

Another possibility is that the comments were added like that from the start, maybe as a placeholder to add real documentation later. Maybe the person didn't even write those; they could have very well been generated by a plugin.

In any case, it's clear nobody thought of the person who would have to read this code for the first time, three years later, maybe even on their first day. It doesn't really tell them anything: what are flowers for? Where are they used? Why are they necessary?

So this person is now forced to go asking around a bunch of people, starting with the ones that git blame shows, and is keeps getting redirected to someone else until they finally ask someone who goes oooooooh, yeah, I remember that thing, wow, it was back when... and gets all starry eyed and tells a long story about how things were done back in the day, and it's probably very enlightening, but by now it's taken the better part of someone's day, when a simple paragraph describing the reason d'etre of the flower would have taken the writer less than an hour and the reader just a minute.

So, be mindful when documenting your code. Java programmers have at our disposal this nifty feature called Javadoc which allows us to add references to other declarations and other metadata. Even if you never generate actual docs from it, it's extremely useful when navigating code because IDEs can render javadoc on the fly, showing readers very useful information that saves a lot of time because they don't have to dive into other files all the time.

/** The contract for a component that
 * handles the storage of flowers.
 *
 * @see Flower
 */
interface FlowerModel {

    /** Inserts a new flower into the database.
     * Expects the flower to have no ID yet, and will assign a new one to it.
     * @param flower A new flower with no ID.
     * @throws DuplicateKeyException if a similar flower already exists
     *         (flowers are considered identical if they have the same scientific name)
     */
    void insert(@Nonnull Flower flower);

    /** Creates a group of flowers of the same kind.
     * @param flower The kind of flower to use
     * @param count How many flowers are included in the bouquet.
     * @returns A bouquet, which is a group of flowers of the same kind.
     * @throws IllegalArgumentException if count of less than 1.
     */
    Bouquet arrange(@Nonnull Flower flower, int count);
}

Ever heard that old saying about writing code as if a dangerous psychopath who knows where you live is going to read it five years from now? Well it applies for documentation too. But instead of writing out of fear, document your code out of empathy. Your future self will thank you when you go back to that code after a couple of years, and when the new hire documents their code properly too because they saw yours and took it as an example.

Database Views vs Table Functions

Enrique Zamudio — Mon, 22 Jul 2019 20:27:50 +0000

Views are a very useful feature of relational databases. You can define a complex query and store it as a view, and then query that view as if it were a table.

I have a complex query involving a join across 4 tables. That's a lot of columns. So I created a view to simplify things, and it works beautifully. Except, one day we had to change the type of a column, because we had screwed up, and it required us to drop the view. The situation is similar to this:

--A simple table as an example
CREATE TABLE example(
  table_id BIGSERIAL PRIMARY KEY,
  created  TIMESTAMP,
  other_stuff VARCHAR(20)
);

--An example of a view, as a simple query to the table
CREATE VIEW example_view AS SELECT * FROM example;

--If we try to modify the table, we get an error
ALTER TABLE example ALTER COLUMN created TYPE TIMESTAMP WITH TIME ZONE;

--We need to drop the view so we can modify the table
DROP VIEW example_view;

--Then we can alter it
ALTER TABLE example ALTER COLUMN created TYPE TIMESTAMP WITH TIME ZONE;

--Finally, re-create the view
CREATE VIEW example_view AS SELECT * FROM example;

In my case, the table we need to modify had over 8 million rows, so the ALTER TABLE command took several seconds to run, during which there was no view, so all functionality that depended on that view had to be taken offline.

After altering the table, creating the view takes no time at all, but the downtime during the table modification was an inconvenience I wanted to eliminate. So I started looking for alternatives.

Table Functions

Reading the PostgreSQL documentation I came across table functions. These are functions that return a table-like structure and you can query them as you would query a table. Following the previous example, we can create a table function instead of the view like so:

CREATE FUNCTION example_function()
  --The return type must include column names and types, like a table definition
  RETURNS TABLE(table_id BIGINT, created TIMESTAMP, other_stuff VARCHAR) AS
$$
BEGIN
  RETURN QUERY SELECT * FROM example;
END
$$
language plpgsql;

And then we can query it like this:

SELECT * FROM example_function();

If we later decide to alter the table, we can do so without getting an error. But, next time we query the table function we will get an error, because the table structure no longer matches the return type of the function. So we still have a problem in which queries stop working after altering a table.

A bit of metaprogramming

One approach to solve this is with a bit of metaprogramming. Instead of directly creating the table function, we can create a function to create the table function, dynamically defining the return type from the metadata of the table. This way, when we alter the table, we can simply run the generator function again and the table function will be updated with the new table metadata. It's something like this:

CREATE FUNCTION create_table_function() RETURNS INT AS
$META$
DECLARE
  columns TEXT;
  cmd     TEXT;
BEGIN
  --Here we create a string with all the column names and their types
  --separated by commas, to use as the return type of the function
  SELECT (SELECT STRING_AGG(column_name || ' ' || data_type, ', ')
    from information_schema.columns where table_name = 'example'
    group by table_name) INTO columns;

  --Here we create the whole function definition
  SELECT 'CREATE FUNCTION example_function() RETURNS TABLE(' ||
    columns || ') AS $$ BEGIN ' ||
    'RETURN QUERY SELECT * FROM example; END ' ||
    '$$ language plpgsql;'
    INTO cmd;
  --First we need to drop the function, because we can't replace
  --an existing one due to the different return type
  DROP FUNCTION IF EXISTS example_function();
  EXECUTE cmd;
  RETURN 1;
END
$META$
language plpgsql;

The function above creates the command to create example_function, constructing the return type from the example table metadata. When we call this generator function, it will create the table function we need. So after altering the table, we only need to call this again and the table function will be updated instantly. Well, almost instantly.

There is still a minor problem: what if someone alters a column from a table used by our table function, and forgets to regenerate the table function itself? A lot of errors will start to occur. One workaround for this is to make the query performed by the table function cast every column to the type defined in the function's return type. This requires some tweaking to the table function, again using the table's metadata:

CREATE FUNCTION create_table_function() RETURNS INT AS
$META$
DECLARE
  columns TEXT;
  query   TEXT;
  cmd     TEXT;
BEGIN
  --This remains the same
  SELECT (SELECT STRING_AGG(column_name || ' ' || data_type, ', ')
    from information_schema.columns where table_name = 'example'
    group by table_name) INTO columns;

  --Now we create a string with each column, casting it to the type
  --it currently has, so it always matches the return type of the function
  SELECT (SELECT STRING_AGG('CAST(example.' || column_name || ' AS ' || data_type, '), ')
    from information_schema.columns where table_name = 'foo'
    group by table_name) INTO query;

  --we create the function with the new query
  SELECT 'CREATE FUNCTION example_function() RETURNS TABLE(' ||
    columns || ') AS $$ BEGIN ' ||
    --we need a closing parens because of the way STRING_AGG works
    'RETURN QUERY SELECT ' || query || ') FROM example;' ||
    'END $$ language plpgsql;'
    INTO cmd;
  DROP FUNCTION IF EXISTS example_function();
  EXECUTE cmd;
  RETURN 1;
END
$META$
language plpgsql;

Because every column is cast to the data type that the table had when the table function was created, altering one of these columns does not affect the function's return type, so now the function is not affected at all by the modification (except if you drop a column from the table, but this is also a problem with views). The table function can be updated later.

Everything's a trade-off

Of course there are disadvantages to this approach. The most evident one is the clunkiness of creating the table function dynamically, compared to the simplicity of creating a view.

A big disadvantage however, is that you can't index table functions; the only way I've found around this limitation is to have one function per query, or add parameters to your function so it can perform different queries depending on the arguments passed to it.

In any case, altering the types of table columns is not something you should be doing often. The most common case is extending VARCHAR columns, which would not affect the table function (but it can affect a view, so this is yet another advantage of table functions vs views).

I'm not trying to advocate for table functions instead of views for every case, but I think this is a very useful trick to have at hand if you find yourself in a situation where you need to alter a table that's involved in a view and need to do something to minimize downtime.

Streaming large queries in Java

Enrique Zamudio — Thu, 15 Nov 2018 18:29:28 +0000

I've been using the JdbcTemplate class since version 1.0, and it's evolved nicely, but I was hoping that for version 5 it would include some streaming capabilities for queries with large results. Alas, that didn't happen.

Still, sometimes I need to perform queries that return millions of rows, and I can't use the JdbcTemplate methods that return lists for it. A RowCallbackHandler is perfect for it, but it would so much nicer to just receive a Stream, wouldn't it? Especially if you have custom RowMappers...

So, I decided to write my own Stream generator to use with a JdbcTemplate. In the process, I ended up creating a more generic Stream generator, which I think is good, and so I want to share it with anyone who needs something similar. I don't think it's enough material for a library, though, so I decided to write a post about it instead.

The challenge

First of all, we need to consider that streams are lazy, and when you get a stream and define the operations to be done on it, nothing is happening yet, until you realize a final operation, which needs to actually traverse the elements and apply the operations on it. There are operations which go through the entire stream (such as count, or collecting the elements into another collections), and there are short-circuit operations (such as determining if any element passes some filter).

So we want to get a stream, and define operations on it, and nothing happens, up until the moment when the stream needs to be traverse, then the query needs to be run (which implies having an open connection to the database). If something bad happens, the query needs to stop (and JdbcTemplate will take care of cleaning up the connection and other resources).

The only way I found I could make this work is by using two threads: a producer thread in which the query is run and the rows are somehow fed to the stream, and a consumer thread which is the reader of the stream.

We will need a buffer in which the producer will store elements and from which the consumer will take elements from. A LinkedBlockingQueue seems perfect for this.

So, without further ado, here it is:

    public static <T> Stream<T> streamForQuery(int bufferSize, T endOfStreamMarker,
                                               Consumer<Consumer<T>> query) {
        final LinkedBlockingQueue<T> queue = new LinkedBlockingQueue<>(bufferSize);
        //This is the consumer that is usually passed to queries;
        //it will receive each item from the query and put it in the queue
        Consumer<T> filler = t -> {
            try {
                //Try to add to the queue, waiting up to 1 second
                //Honestly if after 1 second the queue is still full, either the stream consumer
                //needs some serious optimization or, more likely, a short-circuit terminal
                //operation was performed on the stream.
                if (!queue.offer(t, 1, TimeUnit.SECONDS)) {
                    //If the queue is full after 1 second, time out.
                    //Throw an exception to stop the producer queue.
                    log.error("Timeoud waiting to feed elements to stream");
                    throw new BufferOverflowException();
                }
            } catch (InterruptedException ex) {
                System.err.println("Interrupted trying to add item to stream");
                ex.printStackTrace();
            }
        };
        //For the stream that we return, we use a Spliterator.
        return StreamSupport.stream(() -> new Spliterators.AbstractSpliterator<T>(Long.MAX_VALUE, Spliterator.ORDERED) {
            //We need to know if the producer thread has been started
            private boolean started = false;
            //If there's an exception in the producer, keep it here
            private volatile Throwable boom;
            /** This method is called once, before advancing to the first element.
             * It will start the producer thread, which runs the query, passing it our
             * queue filler.
             */
            private void startProducer() {
                //Get the consumer thread
                Thread interruptMe = Thread.currentThread();
                //First time this is called it will run the query in a separate thread
                //This is the producer thread
                new Thread(() -> {
                    try {
                        //Run the query, with our special consumer
                        query.accept(filler);
                    } catch (BufferOverflowException ignore) {
                        //The filler threw this, means the queue is not being consumed fast enough
                        //(or, more likely, not at all)
                    } catch (Throwable thr) {
                        //Something bad happened, store the exception and interrupt the reader
                        boom = thr;
                        interruptMe.interrupt();
                    }
                }).start();
                started = true;
            }
            @Override
            public boolean tryAdvance(Consumer<? super T> action) {
                if (!started) {
                    startProducer();
                }
                try {
                    //Take an item from the queue and if it's not the end of stream maker, pass it
                    //to the action consumer.
                    T t = queue.take();
                    if (t != endOfStreamMarker) {
                        action.accept(t);
                        return true;
                    }
                } catch (InterruptedException ex) {
                    if (boom == null) {
                        System.err.println("Interrupted reading from stream");
                        ex.printStackTrace();
                    } else {
                        //Throw the exception from the producer on the consumer side
                        throw new RuntimeException(boom);
                    }
                }
                return false;
            }
        }, Spliterator.IMMUTABLE, false);
    }

And this is how you use it, with a JdbcTemplate:

final MyRow marker = new MyRow();
Stream<MyRow> stream = streamForQuery(100, marker, callback -> {
    //Pass a RowCallbackHandler that passes a MyRow to the callback
    jdbcTemplate.query("SELECT * FROM really_big_table_with_millions_of_rows",
                       rs -> { callback.accept(myRowMapper.mapRow(rs, 0)); }
    );
    //Pass the marker to the callback, to signal end of stream
    callback.accept(marker);
});

At this point, the query hasn't been performed. You can do stuff such as:

stream = stream.filter(row -> row.isPretty());

And still nothing happens. When you do something like this though:

Optional<MyRow> row = stream.skip(100_000).limit(1000).findAny();

Then the query is executed, the first hundred thousand rows will be read (and skipped), and each row will be passed through the filter, until one is pretty or a thousand rows have been read.

With great power...

Please, please, PLEASE don't use this as a substitute for a good WHERE clause and properly indexing your tables. I've used this thing mainly to generate reports, concatenating streams of disjoint types by mapping the elements to a common type for further processing (basically, making up for the lack of union types in Java).

Having said that, it is pretty cool to be able to read rows from a database in a streaming fashion.
I guess this could be integrated into Spring's JdbcTemplate, and/or jOOQ...

I hate checked exceptions

Enrique Zamudio — Fri, 10 Aug 2018 06:15:43 +0000

I hate checked exceptions. I really do. They should have really got rid of them in Java 8. They get in the way of a lot of the new stuff; it's not a coincidence that most of the other JVM languages don't have checked exceptions.

BEGIN RANT

I have a class in which I'm overriding an abstract method:

@Override
protected Runnable createTask(Request req) {
  Response resp = new Response(Errors.CANT_CONNECT);
  if (req.getType() == Request.Something) {
    return () -> {
      try {
        oneThing(req, resp);
      } catch (IOException ex) {
        log.error("Doing something with {}", req, ex);
      } finally {
        process(resp);
      }
    };
  } else if (req.getType() == Request.SomethingElse) {
    return () -> {
      try {
        somethingElse(req, resp);
      } catch (IOException ex) {
        log.error("doing something else with {}", req, ex);
      } finally {
        process(resp);
      }
    };
  } else if (req.getType() == Request.YetAnotherThing) {
    return () -> {
      try {
        anotherThing(req, resp);
      } catch (IOException ex) {
        log.error("Doing yet another thing with {}", req, ex);
      } finally {
        process(resp);
      }
    };
  }
  return null;
}

As you can see, it's very repetitive. A block is code is basically repeated 3 times with just a very small variation. An obvious opportunity to put it elsewhere, right? Something like this looks way nicer:

private Runnable execute(BiConsumer<Request, Response> m,
        Request req, String msg) {
  Response resp = new Response(Errores.CANT_CONNECT);
  return () -> {
    try {
      m.accept(req, resp);
    } catch (IOException ex) {
      log.error(msg, ex);
    } finally {
      process(resp);
    }
  };
}

@Override
public Runnable createTask(Request req) {
  if (req.getType() == Request.Something) {
    return ejecutar(this::oneThing, req, "Doing something with {}");
  } else if (req.getType() == Request.SomethingElse) {
    return ejecutar(this::somethingElse, req, "doing something else with {}");
  } else if (req.getType() == Request.YetAnotherThing) {
    return ejecutar(this::anotherThing, req, "Doing yet another thing with {}");
  }
  return null;
}

Way nicer. Yeah.

Except I can't do this. Why not? Well, BiConsumer doesn't throw IOException, therefore I can't catch it. Also, I can't pass refs to my methods because they throw IOException.

In order to do this, I need to catch IOException inside each of my methods, only to convert it into a RuntimeException, which I can then catch in execute.

I really don't want to do that right now. So... ⌘+Z ⌘+Z ⌘+Z ⌘+Z ⌘+Z ⌘+Z ⌘+Z ⌘+Z ⌘+Z ⌘+Z

END RANT

ebooks: Valid use case for blockchain?

Enrique Zamudio — Mon, 27 Nov 2017 22:56:30 +0000

I've seen ideas thrown around about X but on the blockchain and they usually are overengineered, buzzword-driven things that could easily be solved with digital signatures in a regular, central database. I've heard of very few examples of problems that really require the blockchain.

Then a few days ago, while on a layover between 2 long flights, a thought popped into my mind: Using the blockchain for validating ebooks.

When it comes to books, I still prefer the paper versions over electronic ones. I don't even own a Kindle. I have bought very few ebooks, and all of them have been technical. And the main reason for this is that I don't trust that system. I clearly remember a debacle with Amazon some years ago when they deleted copies of 1984 from a lot of devices without permission from the users. And it's known that the books you have on these devices can be updated without your consent, which is great for fixing typos but it's got some real potential for Ministry of Truth style revisionist editing. I'd rather have typos any day of the week.

The solution to this, however, is rather simple: Just have a hash of the book handy. Maybe you can calculate it yourself when you download the book, and check it periodically. That way you can be sure it hasn't been edited without your knowledge; but what if it was somehow edited right before you download it?

So, it may be better to have these hashes available online. A simple database where you lookup by ISBN and get a hash, and you can compare it to the one you just calculated. Simple.

But, who's in charge of this database? It needs to be either someone I trust, or someone I don't need to trust. So: Blockchain! A decentralized database with all the hashes of ebooks everywhere. Of course, it needs to be work a bit different from the ones I'm familiar with, which are for cryptocurrencies. My first idea is this: when you get an ebook, you get its hash and then send it along with the ISBN to this blockchain. A lookup will be done first, and if it's there, you get back the hash that's already stored there, and you can see if it maches. Otherwise, your submission is kept in a pool, until you get X number of similar submissions (by ISBN). If the hashes in all these submissions match, then it's added in the next block. If a small number of submissions don't match, they get discarded. If there seems to be no consensus at all (lots of different hashes with no significant majority), then all get discarded, and maybe the ISBN is flagged for good measure.

Now, who makes the blocks? That's always the tricky part. And the problem is, we usually ask the wrong question. Because the answer to "but who makes the blocks?" should always be: anybody. Same goes for "but who can send transactions?" (or, in this specific case, "but who can submit ISBN+hash?")

With any cryptocurrency, you can just download the software and become a node in that blockchain. Same here: anybody should be able to run a node for this blockchain. So the question, then, is: Why would anyone want to run this node? What incentive do they have to do this?

And here's where I ran out of ideas. I don't know the answer to this. Bookstores, maybe... Publishing houses? Even authors might have a vested interest in this. And there's also the detail of how blocks are created. Proof of Work? Proof of Stake? Is there a cost to submitting ISBN+hashes?

I don't know. Maybe I was just too jetlagged and this seemed like a good idea. But it's been 4 days since that, and at least it doesn't sound completely bogus. What do you think?