Simplifying Multiple Null Checks in Databricks
Recently, I ran into a case where I needed to check if 11 different fields were null. Yes, I could have used Copilot (or my preferred Codeium) to generate it for me, but I knew it had to be easier. There had to be an easier way…
The Scenario
You have a list of conditions in a python list (in my case, I pasted it as separate lines and did the following. Doesn’t matter how you specify your conditions, just that it’s a list somehow
conditions = """
condition 1
condition2
...".split("\n")
You have a way of associating this list with a set of Databricks columns
Using reduce
from functools
Here’s what I found:
- There’s a python function
reduce
infunctools
that takes a list a reduces it down to a result - You specify how this list is “reduced”, such as “and each element of this list together”
Example Implementation
Here’s how it works. first, I have an array of join_columns
which are the common columns between two datasets. In my setup, each column of join_columns
maps to the corresponding filter:
conditions = [F.col(join_columns[i]) == filter_cols[i] for i in range(2)]
Now, we use the reduce
function to combine these into a condition for filtering:
condition = functools.reduce(lambda a, b: a & b, conditions)
Combining Conditions
So, in this case, we are saying all these conditions should be and
’d together – or rather all of them should match. Naturally, had I wanted any of them to match, this would have have been:
condition = functools.reduce(lambda a, b: a | b, conditions)
(the &
is now a |
)
Final Condition for Filtering
That’s it–I now have a condition that I can use:
res_df = input_df.filter(condition)
Top comments (0)