DEV Community: Kuldeep Yadav

Python Extension Modules

Kuldeep Yadav — Sun, 16 Jul 2023 05:12:29 +0000

Disclaimer: I am still exploring this space and might have
interpreted few things wrongly here, hence I would advise readers to
take it with a pinch of salt and explore themselves and don't use the
examples in production :-)

Recently in my company, I was asked to analyse a python package which
can be used to connect with kafka cluster as my team is moving away
from tech stack which involves languages i.e. Java, C# to languages
like python & node.I have some experience with python as once in my
previous company, I used flask to write an API gateway (not
completely, it was just a bad wrapper) and it was working quite well
but then due to maintenance burden of a different tech (which I only
knew at that time), we decided to ditch it completely.

At first I thought, it should be easy as their must be some package on
pypi, which can be used but there was a catch in my requirement.The
cluster I had to connect was using Kerberos authentication, which to
be honest till date I have not understand completely due to its
opaqueness in some regards and compairing it with other authN & authZ
patterns i.e. OAuth2, token etc. Still, I thought there would be some
packages available which can be leveraged, but to my surprise, I could
only find 2 packages, kafka-python & confluent-kafka. It was fine,
I just needed one to work, but here comes the challenge, both packages
required SASL feature to be enabled which internally would use GSS
API.

This is little bit involved as GSS (Generic Security Services API)
allows application to communicate securely using kerberos and requires
native library libkrb5 to be present on client (and its dependencies).
These native dependencies are available via most package managers i.e.
brew, yum, apt etc to any linux or similar platform. In enterprises,
where Microsoft ecosystem is ubiquitous, kerberos protocol is used
heavily which provides SSO capabilities and employees only require
their system credentials to access any service over internet. Now this
post will become very long if describe the every piece of kerberos
authentication and at the time of writing this I am not fully
convinced that I undestand it.

I first gave a try to kafka-python as I found an example online
demonstrating producer example using kerberos, but when I tried, it
didn't work for me and error message were not clear enought what was
happening, so I checked the package's github issues and I found
similar issues posted and in one of the issue, the author himself
confirmed that he doesn't have much clue about the SASL implementation
as it was patched by community only and doesn't have idea around how
to debug or test it.

Finally, I decided to give up on this and turned towards
confluent-python-kafka package, which was also an official package
from the Confluent (company behind kafka now), this package had one
big issue, it was not supporting SASL out of the box. On github, it's
README file mentioned that this is a wrapper package on their existing
C/C++ library librdkafka and for SASL, simply installing it from
pypi and using in our code, won't work. We would need to install using
source distribution, which then link the SASL feature.Normally, you
would expect from any package manager to install the library and then
you can start using it, but in above case we have to do some extra
steps.

Packages

Languages like java has a concept of bytecode, so every dependency you
would include in your code would eventually have a jar file hosted on
some repository, and JVM will take care of converting and executing it
on any platform, hence abstracting away the complexity of creating
platform dependent binaries.

Javascript is also same in that regard as most modern js frameworks
are using nodejs, which provides engine to run the js on any machine
as well as abstracts away the complexity of running your js program on
any platform. Similarly python also uses interpreter Cython to compile
and execute the code on any platform. Since both node and Cython are
written in C/C++, its natural to have the API available so that you
can extend the capabilities of these languages.

Python has two types of packages pure python or wrapper packages
around native libraries written in c, c++, rust etc etc. As you might
have got the idea that pure packages are written completely in python
and you just install and use them. Wrapper packages requires more to
setup and use because of inherited complexity of the platform on which
it will run. These packages are mostly written in C/C++ and hence
require platform specific tools to compile these.

My toy example

Before we dive into how these packages are installed and setup, let's
understand the python C/C++ extension API with a small example.

Let's say you have your own module which offers a function sort, which
will take a list and will return the sorted list.

from wildsort import sort

if __name__ == "__main__":
    list = [21, 3, 48, 12, 18,45]
    list = sort(list)

Here, I named the module wildsort and it will expose the sort
function, which will do the sorting and return a new sorted list.

Now this is very simple module with just a simple function but let's
see how I develop this using C API.

First, we need to create the sort function, so I created a
wildsort.c file and declare my function as below:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject* wildsort_sort(PyObject *self, PyObject *args) {
    return NULL;
}

For people who have not written a single line of C, this can be
difficult to understand and I cannot help much here so I would suggest
to first write a hello world in C to relate some of things written
above.

Now for those, who knows C, first two lines should be clear atleast
from syntax point of view, since we will be using Python's C API, we
need Python.h, which define the contract for whole API. You can
ignore the first macro for now. The function above just return NULL
pointer as of now.

After creating the function (this will be our sort function in python
code), we need to create a defination of it to register with our
module wildsort later.

//...previous code above it will remain here
static PyMethodDef WildsortMethods[] = {
    {"sort", "wildsort_sort", METH_VARARGS, "Sort a number of integers "
    + "but not in performant way"}
}

In above code we are using PyMethodDef struct to create a method
definiton in our module and here we are mapping the function name
going to be in python sort with wildsort_sort declared above and
defining the parameters type for the function METH_VARARGS, meaning
it will take single argument and no keyword arguments which python
offers.

Similarly, we need to create module definition in which we will map
the method definition and then link it to the module creation.

static struct PyModuleDef WildsortModuleDef = {
    Py_ModuleDef_HEAD_INIT, "wildsort", NULL, -1, WildsortMethods
}

You can read about the some of the params passed above in python C API
docs. Now we will create the function, which will be called to create
and initiaze our module when python interpreter starts executing our
python code.

static PyMODINT_FUNC PyInit_wildsort(void) {
    return PyModule_create(&WildsortModuleDef);
}

Here, one thing to note that the function name should always follow
naming standard PyInit_* and * should match with your module name
defined in module definition.

Now, you can write the sort implementation in function wildsort_sort
method or can delegate it to any external shared library, which will
involve some nitty gritties of converting python objects from and back
to data structures in C.

Building the package

Now, in main directory of your code, you need to add setup.py with
below code

from distutils.core import setup

setup(name = "pywildsort",
      version = "0.1.1",
      description = "Wrapper package in C to be used as python package")

You must have some of the module installed in your python distribution
to run above script i.e. setuptools etc. Most of these modules comes
pre-installed with latest python distribution and I guess you won't
have to install explicitly.

Now, we need to run below steps to build a python package

python setup.py build

I would suggest you do these steps by creating a virtual env (check
python docs) otherwise you may run into error using just python
because on some systems i.e. Mac OS it points to python 2.x version
and command may fail. In that case you may have to explicitly use
python3.x variant of command and please check if its on path but I
would recommend using virtual environment.

This will build the source code with available C compiler on your
machine and will produce the object file(s) and other metadata
information into build directory in root directory of your source
code.

Now if you need to test the package, you can instead run the below,
which will install the package into your python installation directory
with your have run the setup.py file.

python setup.py install

Note if you are using virtual env, then it will install inside the
virtual environment installation location.

You can test the package with the code mentioned in the starting or
with python REPL, keeping in mind that you installed with same python
version or virtual environment.

Back to our confluent-kafka-python package

Above was just to create the native wrapper package for python, in my
case, I wanted to understand why I would need to install using the
source distribution for confluent-kafka-python package and not just
install and use it, irrespective of if it is written using C library
or pure python.

Some python packaging history

The answer lies in to understanding how python packages are build and
uploaded to pypi and why there are multiple options to install the
packages.

In early days python offerred only one type of packaging format and
that was source distribution, I guess this was easiest choice
considering the complexity involving different platforms.

In this distribution your package's source code will be deployed as
zip file to pypi and basically when user will install it using pip,
it will be downloaded on user's machine and then bascially python setup.py install will be executed like in our example. Package author
would run the below command to create source distribution and upload
it to pypi.

python setup.py sdist

Since this process will be slow if there are projects which require
many packages and then their dependent packages will all be compiled
at user's machine, a binary distribution concept was introduced, in
which the author will create platform dependent binary distribution
for major platforms and deploy it on pypi. So when an user install, it
will receive the precompiled files and pip would just link it
with current python interpreter.

In python ecosystem, this binary distributon is called wheel and
this is the default mechanism for pip. So pip checks for the binary
distribution matching your machine OS and platform and if found it
would just download it and unzip and just link the pre-compiled
sources.

But when it doesn't find any matching package to your machine
requirement, it will fallback to source distribution and old ways of
compiling and linking as per your machine arch and tools available. In
case tools are not available on machine to compile and link the source
then the package installation would fail and I guess this is another
reason why binary distributon makes sense.

You may check the naming convention of wheels available on pypi and
python docs, you will understood why these are named that way.

Why confluent-kafka-python require source distribution for

kerberos?

Now confluent kafka python package already had a binary distribution
available for my machine but it was mentioned that to enable SASL
feature, I would need to install package using below:

pip install --no-binary confluent_kafka confluent_kafka

The reason is that the package authors decided to disable the SASL
feature while creating binary distribution. I guess they might have
thought that not everyone will be using kerberos to authenticate with
kafka and they might have thought that generally the older languages
like c/c++, java etc. only are used in that case to connect to kafka
cluster and not python and it is required that the librdkafka (their
c/c++) lib and its dependencies should be present on the machine to
utilise this feature. Hence the source distribution was necessity so
that pip can link the librdkafka dynamic library on OS while
installing the package with SASL feature enabled and in your code you
can easily use the kerberos authentication.

If you want to learn more about how it all works, you may check the
confluent-kafka-python
and if you feel overwhelmed by the source code, then you can check my
above example's source
code in which I created
a shared library libwildsort and used it to complete the
wildsort_sort implementation in our example above.

Conclusion

It was exciting for me to touch base around how extension C API works
and doing something in c/c++ after very long and I learn few things
around python and its ecosystem. Also, I found that the most famous
python libs are using C API and major chunk of the packages is written
in c, c++, rust etc etc.

I think there are some good use cases of doing this i.e. reusing
component already written in other language stack and already famous,
performance etc., but again it also makes the ecosystem complex for
user who are not much aware of these things and generally work in
python. Sometimes I have seen the error messages emitted by pip are
too cryptic to understand what has gone wrong if you don't know all
this.

I have mixed feeling about the whole ecosystem as of now and not sure
if this is good thing but I guess this will be a trade off for easy
language syntax and ease of use for scripting....I don't know yet!

You can also read this on my blog, thanks!!

References

Chain Of Responsibility Design Pattern

Kuldeep Yadav — Mon, 16 Aug 2021 12:41:51 +0000

Chain of responsibility pattern is behavioural pattern from G.O.F book about design patterns, this pattern is very useful when you need to decouple the sender from its receivers and each receiver has some responsibility to fulfill.

Let's understand this with an example, suppose you have a mail server, and you have already identified ways to identify spam, fan, business, complaining mails from normal mails, and you are asked to implement this logic into the mail server, so that it automatically filters these mails and perform some logic, for example:

Spam needs to be ignored.
Fan mails needs to be forwarded directly to CEO of company.
Business mails needs to be forwarded to business email in company.
Complaint mails need to be directed to Consumer department.

Now, we can simply put if and else-if checks in our logic and accomplish the requirement, but it will be problematic for
few reasons:

We will be violating open-close principle, because whenever we need to add new rule we would change our function.
We will also be breaking the single responsibility principle as the function would be doing a lot of things on its own.

Better approach would be if we have different handlers for each responsibility i.e. filtering spam, fan and then some how combine them together, which will then handle the request object one by one. This is what COR helps us to do. Let's take a look at the implementation of this pattern to solve our problem at hand, given below is the UML diagram of the classes involved.

Let's refer the Chain of responsibility pattern with COR now on wards.

Here, MailFilter is the functional interface with abstract filter method, which will be then implemented by every concrete class i.e. SpamFilter, FanFilter, so each filter will provide its own implementation for filter logic.

MailServer class is the sender class which will send the mail object to handler assigned to it, in actual implementation it would not know if there are more than one handler and which handler it is interacting with.

Implementation

For the implementation I have chosen the functional interface instead of abstract class for MailHandler above since there is only one abstract method.


@FunctionalInterface
public interface MailFilter {
    default MailFilter appendNext(MailFilter next) {
        return mail -> {
            filter(mail);
            next.filter(mail);
        };
    }

    void filter(Mail mail);
}

Notice that appendNext function is giving same interface(implementation) back to the caller here, also it is calling first the filter method(which would be of concrete class) and then it calls the next.filter(mail) method on next handler in chain.

Now, lets look at one of the concrete implementation of this interface SpamFilter

class SpamFilter implements MailFilter{
    private final List<String> allowedMails = Arrays.asList(
            "abc@xyz.com",
            "zyz@abc.com",
            "kuldeep.yadav@xyz.com",
            "johnoliver@xyz.com",
            "vincent@rocketmail.com"
    );

    @Override
    public void filter(Mail mail) {
        if(isSpam(mail)) {
            System.out.println("Ignoring spam mail");
        }
    }

    private boolean isSpam(Mail mail) {
        return !allowedMails.contains(mail.from());
    }
}

As you can see the SpamFilter class implements the MailFilter interface, so it provides the logic for filter method which will then check if the Mail object can be filtered and if true then it will process (In this case only printing on console).

The filtering logic in above example is just for demo purpose, please don't use it in production :)

Similarly, there would be other filters in the chain which will process the Mail object and work on it if the logic/condition passes. Let's see how it is being used by the sender/client.

class MailServer {
    private final MailFilter filter;

    public MailServer() {
        filter = init();
    }

    private MailFilter init() {
        return new SpamFilter()
                .appendNext(new FanFilter())
                .appendNext(new BusinessFilter());
    }

    public void process(List<Mail> mails) {
        mails.forEach(filter::filter);
    }
}

Here for the demo purpose I have chosen to define the chain in the same class but in real scenario, you may want to have
some strategy or factory which will generate this chain and give you the filter object to deal with in your code. Here, we need to check only the process method which is receiving list of mails and process them one by one.

You can see that this is much cleaner solution than if-else chain in your code and also decouples the sender from the
handler/filter(s) in this case, for example if there was a separate class to initialise the chain, the MailServer
class would not even know how many filters the mail object would go through, hence we can directly add or remove filters
from chain any time without touching the MailServer class.

Also, every filter logic resides in its own class, which follows the SRP(Single Responsibility principle) as well.

Disadvantages

As with every thing in software world, this pattern is also not a magic bullet and comes with cons as well

If the chain is very long, it is hard to debug and verify where the problem lies.
With very long chain of handlers, the call stack will grow as well, and it could lead to performance issues.
It can lead to duplicate code among handlers, although duplication can be avoided.

Points To Consider

Use it when you don't need the guarantee that the request object must be executed by all the handlers, otherwise use the decorator pattern.
Use it when multiple handlers must be able to process a request object, and you don't want sender to have any idea about multiple handlers.
Don't use it when the client must know the handler which will execute the request.

Conclusion

COR design pattern helps to decouple the sender from its receivers, the request sent by sender can be handled by
multiple handlers or receivers without sender knowing it, which will then gives flexibility to remove or add the receivers without touching the code of sender.javax.servlet.Filter API is nice example of this pattern implemented in java world, another good case of this pattern is logging, wikipedia mention this example with implementation. But COR pattern also have its downsides so as a developer we must be aware of these.

You may read the same on my personal blog.

References

Functional Programming in Java 8, Part 2

Kuldeep Yadav — Sun, 14 Mar 2021 14:15:58 +0000

This post is in continuation with my last post, where we saw an example of builder pattern using Java 8 functional library, in this post I will share few more examples with you and discuss the pros and cons of functional programming.

Let's start with an example

Suppose you have a list of persons with their name, age and salary, the person object would be like

@Value(staticConstructor = "of")
public class Person {
    String name;
    int age;
    Double salary;
}

@value(staticConstructor = "of") is the lombok annotation to produce static factory method of

Now, you have a use case where you may want to find out all the persons who have either age more than 30 or have salary more than 100K but must not be both, how would you do that?

Simple traditional approach

Simple solution could be the one which I have been doing since my college days (although I didn't know java then only C ;-) )

// solution list will have our persons with criteria
List<Person> solution = new new ArrayList<Person>();

for(Person person : persons) {
    if(person.getAge() >= 30 && person.getSalary() < 100000) {
        solution.add(person);
    }else if(person.getAge < 30 && person.getSalary() >= 100000) {
        solution.add(person);
    }
}

This approach is also called imperative approach, where we are specifying how our solution should arrive, you may hear this term a lot when reading about functional programming.

Functional Programming approach

Using Java 8 predicate API to filter with given criteria.

As you can see in previous solution we have two if condition, which in simple terms means that we have two mutually exclusive (set theory) conditions, hence we need to find xOr of two conditions.

Predicate API does't not provide xOr operation, though it has or and and default implementation. So we will extend the Predicate API into our own functional interface Specification

@FunctionalInterface
public interface Specification<T> extends java.util.function.Predicate<T> {
    default Specification<T> xOR(Specification<T> other) {
        return t -> test(t) ^ other.test(t);
    }
}

We can have default function definition inside interfaces !!!

Now we can easily use this new interface to filter while using stream over our persons list.

Specification<Person> ageGreaterThanThirty = person -> person.getAge() >= 30;
Specification<Person> salaryMoreThanHundredThousand = person -> person.getSalary() >= 100000;

Specification<Person> xorAgeAndSalary = ageGreaterThanThirty.xOR(salaryMoreThanHundredThousand);

List<Person> solution = persons.stream()
                                                             .filter(xorAgeAndSalary)
                                             .collect(toList());

Isn't it cleaner and readable than the traditional approach? we have no more those if and for loop statements and also the criteria is now re-usable, if you need similar thing at some other place.

"Why" Functional Programming

This question often comes to my mind as well and as with everything in programming, there is no straight forward answer to this and due to our brain being used to of doing imperative programming for so many years, It's very difficult sometimes to see how functional programming helps.

Here, I will share few pros of functional programming with you, which might help you to embrace this style of programming.

Pros of functional programming

Functional programming revolves around the usage of pure functions and immutable objects, so many benefits of both applies to functional programming as well.

Pure functions let you write deterministic functions.
Pure functions are easier to test.
Debugging is easy.
Parallel and concurrent programming becomes easier.
Function signature are meaningful and clear.

Pure Functions

Pure functions are those functions which takes some input and produces an output on the basis of encapsulated algorithm and input parameters, these do not perform any IO, Database Calls, UI interaction or any other side effects.

Due to this definition of pure functions, they are deterministic in nature, that means if that function is called any number of times with same parameters, it would produce the same output every time.

Let's look at some examples

public List<T> sort(List<T> ele, Comparator<T> cmp) {
    // Some sorting algorithm omitted for brevity
    return sortedList;
}

public List<Person> findAllPersonUsing(List<Person> persons, Specification<Person> specification) {
    return persons.stream()
                                .filter(xorAgeAndSalary)
                              .collect(toList());
}

Both above functions when passed with same arguments, will produce same output, no matter how many times these are called.

What would be impure functions, then?

// Function does not return anything, considered to have side effects
public void changePersonName(Person person) {
    // doing some thing with person object here
}

// Person output will depend on what personRepository will give back
public Person getPersonWithName(String name) {
    return personRepostitory.findPersonWithName(name);
}

// Function output will depend on the user input
public String readUserInput() {
    Scanner s = new Scanner(System.in);
    return s.nextLine();
}

First function is interesting one, it does not return anything. In functional programming, these kind of functions are considered having side effects.

Also as per comment, it is changing the state of person object, although this particular example is harmful in any programming style, one must not change parameter state inside function.

This was little simple example to illustrate the point but when you are working with some entity framework i.e hibernate, you may find yourself doing this for some use case.

Pure functions as expressions

Pure functions are deterministic in nature, so they can be written as algebraic equation or composed together like an expression, for example

Integer x = f(any) // f is pure function which returns integer
Integer y = g(x) // g is also pure function which returns integer
Integer z = x + y

Because, f(any) and g(x) are deterministic functions, we can write above as

Integer y = g(f(any))
Integer z = f(any) + y

Integer z = f(any) + g(f(any))

Also, z can be replaced with the expression directly wherever it is used but if f and g were not deterministic in nature it would not have been possible to replace in this manner.

Testing is easier

With pure functions and immutable objects, testing is easier, because you don't have side effects, you won't be mocking objects and their behaviour to test your function.

Debugging is easier

Debugging is easy because you just need to follow the function values in stacktrace, you don't need to worry about other parts of applications.

Parallel and Concurrent programming

Since parallel and concurrent programming involves lot of threads, which might be executing your same function then if you have functions which are mutating states and you are not using immutable objects, then you would receive a panic call in the mid of your vacation, enjoying your time at beautiful beach, saying that everything is blown in production because of your function.

Functional programming have pure functions and immutable objects, thats why you don't have to worrry about multithreaded environment.

Function signatures are meaningful and clear

Since, we write pure functions with functional programming, most of the time function signature tells you what that function might be doing or you wouldn't care what would be happening inside that function.

Cons of Functional Programming

Functional programming is not a new concept though, it's roots are as old as the great depression (1930s), however, programming languages like C++, Java etc promoted OOPs and imperative programming since starting and atleast I leant programming from these 2 languages, I generally found functional programming concepts new and little hard to grasp in the begining.

Following are the cons of functional programming

It's very hard to reason about initially for people, who have been writing programs using imperative approach for many years and often very intimidating.
Due to its connotations with mathematical concepts i.e monads, functor, combiner etc, it's very hard to comprehend sometimes.
Real world software application involves IO, Database, UI, Reading/Writing To Files and since functional programming only uses pure functions, you cannot apply pure functional programming to the whole application, you would need to mix your approach.
Also, fp uses immutable objects, therefore if written carelessly, it would be bad for performance and may eat up lot of memory.
With immutable objects, you would need to cover lot of update case, in that case you could use the pattern like "Update your copy" but again it would work for simple objects, with nested objects it becomes little hard to follow. Also, unlike Scala, its difficult to in Java, you would end up with lot of copy methods inside your class.

I think, we can overcome some of these cons upto some extent

Wrapping impure over pure

We can limit the impact of impure part on pure part by wrapping the pure with a thin layer of wrapper around it to perform i.e IO, DB etc. This looks like we are doing the pure functions only. There are quite a few techiniques i.e monads HOF to do this in fp world, discussing them will be out of scope of this post.

Learning curve

How much difficult it was to write in C++ or Java when you started the language and adopted the style, I am sure you must have written lot of programs to be confident about loops, statements expressions etc . I think same applies to functional programming as well, the more you do, more you get comfortable with it.

Mathematical Connotations

I am telling you this from my personal experience, initially it intimidated me but as I kept doing it, I found none of those concepts are compulsory to get started with fp, yes it is good to learn those if you can understand but it doesn't mean you can't fp without learning those. (This is true for programming as general as well).

Memory and Performance issues

These issues are always debatable and I think these are mostly due to programmers' mistake not due to some programming style, unless you are processing a very huge number of data and you find that the immutable objects are problem, luckily I haven't encountered such case till now and I will appreciate, if someone can share their experiences about it.

Function programming decouple state and behaviours

Functional programming prefers you decouple your data and behaviour that means you must have immutable classes with their behaviors(pure functions) in separate files or modules but this is not mandatory you can have it in same file as well but immutable classes/objects is important point.

This is opposite to what OOPs suggest and might resembles anemic model but it is not.

Conclusion

Functional programming let you write pure functions which will operate on immutable objects, which are easier to test, debug and maintain and are free from side effects. But you cannot write whole real world application which involves lot of moving part just using functional programming.

It has some learning curve but helps to write clean APIs, I prefer to use it whenever I find the use case for it and mix with imperative style of programming to use best of both versions.

So that's it, you can find the source code used in both post on my github account.

Now, you can also read the same post on my blog here

References

Functional Programming in Java 8, with Example(s)

Kuldeep Yadav — Thu, 18 Feb 2021 05:23:38 +0000

It has been almost 7 years since Oracle released Java 8, with it, they also released lambda expressions and Functional interfaces, which lets you do functional programming in Java, but it's not very intuitive when compared to languages i.e javascript (functions are not the first-class citizen in java)

In this post, I will give a brief introduction of functional interfaces and lambda expression in Java and try to explain the little strange syntax with one important example.

Functional Interfaces

Some of the important functional interfaces Java 8 has to offer, which we will discuss in a while.

Function
Consumer
Supplier
BiFunction
Predicate

Function

Represents a function which accepts one argument and produces a result.

It is a very simple and understandable definition, right? But when you go and try to use it you would get confused at first, because as per definition, this represents something like

R someFunction(T param) {
    return r;
}

// Using it would be
R r = someFunction(t);

But in your code, you would write

Function<Integer> convertIntToStr = t -> "Some String from " + t;

String str = convertIntToStr.apply(1);

Got confused, right?

This is because interfaces and classes are the first-class citizens in Java, that's why they have provided these Functional Interfaces and their usage is similar to creating an object and calling functions on that object.

So, what are functional interfaces?

Functional interface is the interface in Java 8 which has only one abstract method, any interface which qualifies this condition is a functional interface and you can also denote it with @FunctionalInterface annotation to explicitly state that, so that compiler can generate an error if some try to add another abstract method to it.

If you checkout the implementation of Function, you would see something like this

@FunctionalInterface
interface Function<T,R> {
    R apply(T t); //only abstract method
    // default method omitted for brevity
}

So, our code example above initialises the object of this interface with lambda expression which is in this case just a function

Function<Integer, String> convertIntToStr = t -> "Some String from " + t;

t -> "Some String from " + t 

       equivalent to

String someFunction(Integer value) {
            return t -> "Some String from " + t;
}

And since convertIntToStr is an object of type Function, we call the apply method on this object to execute our lambda expression or someFunction and you must have got an idea by now that you provided the implementation for apply function using a lambda expression, because the same statement can be written as anonymous object declaration

Function<Integer, String> convertIntToStr = new Function<Integer, String>() {
        @Override
        String apply(Integer t) {
                return "Some String from " + t;
        }
}

So, next time think the statement on left to be equivalent to the right

R someFunction(T param) {
    return r;
}
R r = someFunction(t);

Function<T, R> someFunction = t -> r;
someFunction.apply(t);

All other, interfaces work in a similar manner, those represent some other useful functions, Let's see what they have to offer.

Consumer

Represents an operation that accepts a single input argument and returns no result.

You can think of Consumer as void function, which takes a parameter and it has an abstract method void accept(T t)

Supplier

Represent a supplier of results.

You can think of Supplier as a function which doesn't take any argument and return a value and it has an abstract method T get()

BiFunction

Represents a function that accepts two arguments and produces a result.

You can think of BiFunction as a function which takes two arguments and return a value and has an abstract method R apply(T t, U u)

Predicate

Represents a predicate (boolean-valued function) of one argument.

You can think of Predicate as a function which takes an argument and returns a boolean value, it has an abstract method boolean test(T t)

There are more interfaces than mentioned, you can check docs

Why Functional Programming?

Functional programming lets you compose your logic through pure functions in a declarative manner avoiding shared state, mutable data and side effects. Let's see this with a very simple example.

Suppose we have an item and we need to calculate different taxes on the item's base price, we would write something like this

I would be using lombok project to generate some boilerplate code in my examples, so please check its documentation for more info.

@Value
class Item {
    String name;
    Double price;
}

class TaxCalculator {
        // I have passed the boolean flags for the taxes needs to be calculated
    public Double calculateTaxablePrice(Item item, boolean excise, boolean vat, boolean cess, boolean gst) {
        Double basePrice = item.getPrice();
        if(excise) basePrice += basePrice * 0.02;
        if(vat) basePrice += basePrice * 0.04;
        if(cess) basePrice += basePrice * 0.01;
        if(gst) basePrice += basePrice * 0.1;
        return basePrice;
    }
}

// And we would call like this
Double newPrice = calculator.calculateTaxablePrice(item, true, false, true, true);

You might have spotted the problems in the above code, let me point out a few below

If we need to add another tax tomorrow, we would change this method and so have to be the every calling method in client code.
Client will have to dig into the implementation of a function to know, which flag is for which tax, so the client can make a mistake and this may not be captured by the unit test also.

So, what can we do? An alternative approach can be to create a TaxCalculatorBuilder which will create the TaxCalculator object for us, which will have all the flags as state variables and calculate function will take only price, providing fluent API to the client

TaxCalculator calculator = new TaxCalculatorBuilder()
                                                                .withExcise()
                                                                .withVat()
                                                                .withCess()
                                                                .withGst()
                                                                .build();
// And we would call like this, here calculator object knows which tax to be applied
Double newPrice = calculator.calculateTaxablePrice(item);

But, there is an issue with the builder as well, you would have to change the builder class every time a new tax is added. So, how can we solve this problem?

We can solve this issue using functional interface Function<T, R> and we will see in a moment how?, but before that let's define each tax as a function which would take one argument and give back the result, in new class TaxRules

@NoArgsConstructor(access = AccessLevel.PRIVATE)
class TaxRules {
    public static Double excise(Double price) {
        return price + price * 0.02;
    }

    public static Double vat(Double price) {
        return price + price * 0.04;
    }

    public static Double cess(Double price) {
        return price + price * 0.01;
    }

    public static Double gst(Double price) {
        return price + price * 0.1;
    }
}

Now, we would write our improved tax calculator as following

class ImprovedTaxCalculator {
    private final List<Function<Double, Double>> taxRules = new ArrayList<>();

    public ImprovedTaxCalculator with(Function<Double, Double> fn) {
        taxRules.add(fn);
        return this;
    }

    public Double calculateTaxablePrice(Double price) {
        Function<Double> fn = taxRules.stream()
                                      .reduce(f -> f, (firstFn, secondFn) -> firstFn.andThen(secondFn))
        return fn.apply(price);

    }
}

The calculate function here is little difficult to understand so let's apply divide and conquer to understand it

taxRules is a list, .stream() method call gives us a stream of the object in the list.
reduce() method takes a stream of elements and produce a single result.

Let's take an example to understand it

List<Integer> numbers = Arrays.asList(1,2,3,4,5);

Integer sum = numbers.stream()
                                            .reduce(0, (sum, number) -> sum + number);

The Integer value 0 is the identity value, Its the initial value provided to the reduction process and if the value of the list is empty then it will be the default value which will be returned.

The second argument is the accumulator which will add the previous sum to next number from the stream, initially sum would be 0 here

(sum, number) -> sum + number // Lambda expression

Lambda expression can also be written as a method reference

Integer sum = numbers.stream()
                                            .reduce(0, Integer::sum);

Now let's get back to our original example now

Function<Double> compositeFn = taxRules.stream()
    .reduce(f -> f, (firstFn, secondFn) -> firstFn.andThen(secondFn));

Here, first function in taxRules would be our identity and accumulator is a lambda expression returning a new composed Function, that first applies its input to firstFn and then applies the result to secondFn.

Now, Let's see how client would calculate tax using our improved calculator

ImprovedTaxCalculator calculator = new ImprovedTaxCalculator()
                                         .with(price -> TaxRules.excise(price))
                                         .with(price -> TaxRules.cess(price))
                                         .with(price -> TaxRules.gst(price));

Double newPrice = calculator.calculateTaxablePrice(item);

If tomorrow, we got a new tax, then we just have to add the new function into TaxRules class and the client can use it, we don't have to change our calculator.

Also, if you have a case where only one client has to apply some extra tax which doesn't have to be defined in TaxRules, then the client can very easily do that

ImprovedTaxCalculator calculator = 
                       new ImprovedTaxCalculator()
                                         .with(price -> TaxRules.excise(price))
                                         .with(price -> TaxRules.cess(price))
                                         .with(price -> TaxRules.gst(price))                                                                         
                                         .with(price -> price + price * 0.03); //Extra

Double newPrice = calculator.calculateTaxablePrice(item);

You can see that how functional interface Function can help us write better reusable code with clean fluent API, which is much better than the builder, we wrote earlier.
For now, I will leave you all with this example, we will see some more examples and functional programming pros and cons in the coming posts. So till then enjoy.....