DEV Community: Deji

Ensuring High Data Quality for Accurate Analysis: A Guide for Busy Professionals

Deji — Thu, 11 May 2023 21:40:51 +0000

Data analysis is crucial for businesses to make informed decisions, but the accuracy of the analysis depends on high data quality. As a busy professional, it can be challenging to find the time to ensure data quality. Here are some tips to help ensure high data quality for accurate analysis.

Define Your Data Quality Metrics
Invest in Data Cleaning Tools
Implement Data Governance Policies
Ensure Data Accuracy through Verification
Ensure Data Consistency through Standardization
Monitor Data Quality Regularly
Ensure Data Relevance

When you follow these tips, you can ensure high data quality for accurate analysis. You need to invest time in ensuring data quality, then you can have confidence in the insights and decisions that are based off your analysis.

I wrote more about this on my blog here.

The Top Tools and Software Busy Professionals Can Use for Data Analysis

Deji — Thu, 27 Apr 2023 09:27:24 +0000

I started doing data analysis as a busy professional. As soon as I started, I realized how important it was to find the right tools and software to make data analysis more efficient for me.

I found those tools, and they simplified the process, helped streamline data visualization and ultimately achieve better results.

Here is a quick list of some of the top tools and software you, as a busy professionals you should be aware of for data analysis:

Excel
Tableau
Power BI
Google Analytics
Python
R
SQL

I tried to explore even a little bit of all these tools and wrote more about them on my blog. You can read more about it here.

The Danger of Overfitting: How to Recognize and Address Overfitting in Data Analysis

Deji — Sat, 22 Apr 2023 13:29:52 +0000

Overfitting is a common problem in data analysis that occurs when a model or algorithm is overly complex and fits the data too closely. This can result in inaccurate predictions and unreliable insights.

To avoid overfitting, data analysts need to recognize the signs of overfitting and implement strategies to address it.
Here are some steps data analysts can take to address overfitting.

The first step to recognizing overfitting is to understand the bias-variance tradeoff.
Another way to address overfitting is to use regularization techniques such as Lasso or Ridge regression.
Another is to address overfitting is to use cross-validation techniques such as k-fold cross-validation.
It's also essential to use appropriate evaluation metrics to assess the model's performance.

I wrote more about these steps and strategies on my blog here.

Ethical Considerations in Data Analysis: Avoiding Bias and Discrimination

Deji — Tue, 18 Apr 2023 15:21:44 +0000

Data analysis is a powerful tool that can uncover important insights and help make data-driven decisions. However, it's essential to consider ethical considerations when conducting data analysis to avoid bias and discrimination.

Bias occurs when a data analyst favours or discriminates against certain groups or individuals. Some ways in which bias can creep into data analysis include sampling bias, confirmation bias, and selection bias.

Sampling bias occurs when the sample used for analysis is not representative of the larger population. This can lead to inaccurate conclusions about the population as a whole.
Confirmation bias occurs when a data analyst looks for data that confirms their pre-existing beliefs or assumptions. This can lead to a skewed analysis that ignores important data that contradicts their beliefs.
Selection bias occurs when the selection of data is biased towards a certain group or individual, leading to inaccurate analysis.

I stumbled upon this not too long ago; It was very enlightening to see how these biases play out when conducting data analysis and how to avoid it. I wrote more about it on my blog here.

Reasons Why Data Visualization is Critical in Data Analysis

Deji — Sat, 15 Apr 2023 08:26:50 +0000

As a data analyst, it's essential to understand the importance of data visualization in presenting your findings.

Data visualization is the process of creating graphical representations of data to help you communicate insights to your audience effectively.

Here are five reasons why data visualization is critical in data analysis:

Simplifying Complex Data
Communicating Insights Effectively
Identifying Patterns and Trends
Spotting Anomalies
Making Better Decisions

There's a lot more I got into on my blog post here.

But one thing I always remember is that as a beginner data analyst, learning how to use data visualization tools can be a valuable asset in your career.

Common mistakes in data analysis and how to avoid them

Deji — Thu, 30 Mar 2023 22:27:18 +0000

Data analysis can be overwhelming, especially for beginners. Even experts can make mistakes that lead to incorrect insights and bad decisions. To avoid these mistakes, it's important to be aware of the common pitfalls in data analysis. In this blog post, we'll explore the top 5 blunders to avoid in data analysis from a beginner's perspective.

Firstly, relying on incomplete data can lead to biased conclusions that result in expensive mistakes. Make sure you have a complete and representative dataset to avoid this mistake.

Secondly, ignoring outliers can lead to incorrect conclusions. Identify and analyze outliers separately from the rest of the data to avoid this mistake.

Thirdly, overlooking data distribution can also lead to incorrect conclusions. Examine the distribution of the data before analysis to avoid this mistake.

Fourthly, failing to check for confounding variables is a common mistake that can lead to spurious conclusions. Examine the relationships between all variables in the dataset to avoid this mistake.

Lastly, overcomplicating the model can lead to overfitting, which occurs when the model fits the noise in the data rather than the signal. Use the simplest model appropriate for the data and assess the model's performance on new data to avoid this mistake.

By being aware of these common mistakes and taking steps to avoid them, you can ensure that your data analysis conclusions are accurate and reliable.

Compound interests

Deji — Mon, 11 Jan 2021 10:04:42 +0000

Compounded interests. One of several methods to grow value.

The beauty of investments is that small amounts will compound over a long time into huge fortunes, as long as you don't interrupt the process.

The quote above is from a wealth-tech application. Several other apps also have some narrative similar to that too: save for a long period to enjoy compound interests.

I've found that to be false, or at least, not entirely true.

TLDR; save, but over short periods at a time then reinvest, that is how you get compounded returns.

A case study. An individual deposits some amount of money—this could be one-time or periodically—and earns a specified interest, say 10%, per annum. All well and good, except one could be very disappointed at the meager 'returns' at maturity, especially when savings were made periodically. This is no fault of the apps, it's just how interests work, simple interests.

Simple interest

Very straightforward. If you invest a hundred thousand Naira (N100,000) at an interest of 10% per annum, at maturity, you would have a total of a hundred and ten thousand Naira (N110,000).

Usually, fintech apps would add interactivity by showing 'daily earnings'. That is just dividing your returns of ten thousand Naira (N10,000) by 365. In some cases, this also serves to show that you do not have to wait till a year is up before reaping benefits. If you make a withdrawal after three months, that is a quarter of the year, you would have earned approximately two thousand, five hundred Naira (N2,500).

Again, simple enough.

Periodical savings

The situation is different when you make periodic deposits.

Say you want to save a hundred thousand Naira at 10% p.a. But you do it in two installments: an initial deposit of fifty thousand Naira (N50,000), and another fifty at the 6-month mark. You would not be making your perceived return of ten thousand Naira (N10,000). Why?

Because your initial deposit was the only batch that earned interest for a full year, netting you five thousand Naira (N5,000). Your 6-month batch earned half of that, making two thousand, five hundred Naira (N2,500).

With that said, should your deposit frequency increase, say ten thousand Naira (N10,000) for 10 months, at maturity, your earned returns would be far lower than the expected ten thousand.

Compound interest

A simple way to put it is, interest earned on interest.

Using the example above, at your year's maturity with a total of a hundred and ten thousand Naira, you re-invest it's entirety for another year. This time earning eleven thousand Naira, netting a total of a hundred and twenty-one thousand Naira.

And there's the difference. If the said initial investment was only left for a two year maturity period, you would have earned a hundred and twenty thousand Naira alone.

Now, here's the magic. Take the case of a hundred thousand Naira (N100,000) for a year at 10% per annum. You stand to gain far more if you apply the principle of compound interests like so: save in 3-month intervals.

You leave it in for three months, earn interests of two thousand, five hundred Naira (N2,500) and put that back in for another three months. At 6 months, you would have earned, in interests, two thousand, five hundred and sixty-two Naira, fifty kobo (N2,562.50). A difference of 62.50. Just because you took it out and put it back in. Apply that to the remainder of 6 months, and you net a total of one hundred and ten thousand, three hundred and eighty-one Naira, twenty-nine kobo (N110,381.29).

The splendor of compound interests. Repeat that a few more times and you get a hundred and twenty-one thousand, eight hundred and forty Naira, twenty-nine kobo (N121,840.29).

It's really not linear.

Good business

I said earlier that these apps were not being fraudulent, or deceptive. Not really. Some encourage you to "re-invest" to earn that compound interest. While some just ask you to leave it in there for a longer period.

Oh, the joy if apps could just build that in by default: lock-in for a year, compound at 3-month intervals. Keep the benefits of a "lock-in" feature and earn compound interests too.

Has that never occurred to them? It most likely has. I'd love to then understand their rationale for not building it.

Deji Joseph

Understanding VPNs

Deji — Thu, 31 Dec 2020 10:14:18 +0000

Background

Sometime in October, Nigerians made their discomfort with the state of affairs in the country known to the government through public protests. The government did not respond favorably.

The nature of the protest was largely decentralized. Several groups and individuals supported these protests, though. One of them is the Feminist Coalition. I mention them in particular because, in the aftermath of the demonstrations, the government sought to identify and punish "those they could identify" (for the avoidance of using the term, leader).

Soon after, the Feminist Coalition mentioned that their website was being censored by the government, still is as of today, Dec 11, 2020. It is accessible via VPN though. And now, we're at the crux of the matter.

During the protests, several folks said to install and use VPNs to prevent the government from "censoring" us. The narrative was that VPNs would prevent the government from "taking away the internet"!

As at that time, I couldn't quite agree with that notion, even though I did not know enough to argue against it. I installed one but didn't use it. Its use came, though, when the Feminist Coalition announced their website being blocked. I turned on the VPN and they weren't lying.

Still, I couldn't reconcile the "taking away the internet" part. Your VPN depended on your internet connection being available, and the government still has reasonable control over that. I just had to (re-)learn the details of the internet and its limitations. At least in the context of how VPNs work.

Internet Service Providers

The first bone of contention.

I'll relate/simplify them to a router on a local network. Say, two networks.
Network A has the Johnsons on it. Network B has the Starks on it. Each "network" is their router, the family's. To connect both networks, creating an inter-network, you just need to connect both routers.

Now, in the event that a family's router fails, or rather, the connection between both routers fail, they would be cut off the "inter-network". Their networks will work fine but they won't be able to reach the other family's.

Using that example, say, the Stark Big Brother wanted to prevent his siblings from connecting to the media player on the Johnson's network, all he needs to do is block the resolution of that device's IP address. Or at least, that's one way, simple way, to go about it.

I'll relate a VPN then to an "allowed" device on the Johnson's network, that the Stark's network "allows" a connection to. This device can then connect to the Johnson's media player on behalf of the Stark siblings and route traffic back and forth. But you see, the effectiveness of this is dependent on the network connection between Johnson and Stark still existing and the Stark's network "allowing" that other device.

That pretty much summarizes, quite abruptly too, the purpose of this note. I'll recap how I got to that understanding.

How VPNs work

Doing internet research on that topic, I came across an analogy that stayed with me but just felt so wrong.

It related a VPN to an "underground" system of tunnels a person can use to avoid surveillance on the regular roads. That is, if there were people outside of your door watching when you come out and go in as well as where you go to, then you could travel underground instead and nobody will be the wiser.

The problem with this analogy is that it creates another Internet Service Provider. A better parallel would be using a tinted car and telling surveillance that you were going to your friend's house. When you get to your friend's house, you go where you need to go, and when you need to go back home, you use the tinted vehicle parked at your friend's place.

These images, from my investigation, show what I'm talking about.

In the images above, the VPN is off. Traffic is normal: DNS, TCP, and UDP. Making requests to and receiving from different devices on the internet.

Here the VPN has been turned on and it's attempting to connect to the "proxy" server.

And here, the VPN is connected. It now stands between me and all internet traffic.

Private VPNs then, following the same analogy, will require you to be specific about which house you're going to proxy from as well as a keyphrase to allow you entry into said house.

Censoring and taking away the internet

The VPN provides an alternative, a possible method to circumvent restrictions. I can't lose the thought of the government turning off the internet for its citizens, but now I know VPNs would not help against that.

The solution to that, having as an alternative an ISP that does not go through a government-controlled exchange point.

Cheers.

— Deji Joseph