DEV Community: Sıddık AÇIL

DB Design for Storing International Addresses

Sıddık AÇIL — Fri, 15 Nov 2019 21:55:42 +0000

Every country has a different address hierarchy with different fields. Some has prefectures, others have counties. USA addresses have a necessary ZIP Code field, etc.

This design has the country agnostic aspect I am looking for but it is absolutely not scalable at all.

On a RDBMS system, how would you go about your scheme design?
On the corresponding user interface, how would you design form fields?

What are your design choices on these problems?

Implementing Simple PCA using NumPy

Sıddık AÇIL — Tue, 12 Nov 2019 20:57:58 +0000

I am open to job offers, feel free to contact me for any vacancies abroad.

In this article, I will implement PCA algorithm from scratch using Python's NumPy. To test my results, I used PCA implementation of scikit-learn.


from sklearn.decomposition import PCA
import numpy as np
k = 1 # target dimension(s)
pca = PCA(k) # Create a new PCA instance

data = np.array([[0.5, 1], [0, 0]]) # 2x2 data matrix
print("Data: ", data)
print("Reduced: ", pca.fit_transform(data)) # fit and transform

This results in:


[[-0.55901699]
 [ 0.55901699]]

Centering Data Points

Make origin the centroid of our data.


data = data - data.mean(axis=0) # Center data points
print("Centered Matrix: ", data)

Get Covariance Matrix

Get covariance matrix of our features.


cov = np.cov(data.T) / data.shape[0] # Get covariance matrix
print("Covariance matrix: ", cov)

Perform Eigendecomposition on Covariance Matrix

Eigendecomposition extracts eigenvalues and corresponding eigenvectors of a matrix


v, w = np.linalg.eig(cov)

Sort Eigenvectors According to Eigenvalues

Most numerical libraries offer eigenvectors pre-sorted, However, this is not the case for NumPy. Therefore, we need to argsort the eigenvalue vector to get sorting indices and perform sorting on columns of eigenvalues.


idx = v.argsort()[::-1] # Sort descending and get sorted indices
v = v[idx] # Use indices on eigv vector
w = w[:,idx] # 

print("Eigenvalue vektoru: ", v)
print("Eigenvektorler: ", w)

Get First K Eigenvectors

Our aim in PCA is to construct a new feature space. Eigenvectors are the axes of this new feature space and eigenvalues denote the magnitude of variance along that axis. In other words, a higher eigenvalue means more variance on the corresponding principal axis. Therefore, the set of axes with the highest variances are the most important features in this new feature space since they hold most of the information. By getting the first K columns of eigenvector matrix, which have the K biggest eigenvalues, we form what is called a projection matrix. The dot product of our data matrix and projection matrix, which sounds pretty cool but it is actually pretty straightforward, is the reduced feature space, the result of PCA.


print("Sonuc: ", data.dot(w[:, :k]))
# Get the dot product of w with the first K columns of eigenvector matrix
# (a.k.a) projection matrix

Which also results in:


[[-0.55901699]
 [ 0.55901699]]

This is it. Any corrections or suggestions are welcome.

Extras

Square vs. Non-Square: Eigendecomposition and SVD

Original Turkish article at

CPP Eigen ve Python Numpy ile PCA Gerçeklemesi | by Sıddık Açıl | Medium

Sıddık Açıl ・ May 20, 2019 ・ 4 min read
Medium

My go to article on PCA

Simple .NET Core Benchmark for Whitespace Count

Sıddık AÇIL — Tue, 12 Nov 2019 08:28:55 +0000

I am open to job offers, feel free to contact me for any vacancies abroad.

In this installment, I will take you through a simple benchmark using BenchmarkDotNet. We are going implement and compare different functions that use different approaches to find how many whitespace characters a string has via the static function char.IsWhiteSpace. Let us get to work :)

Methods

These methods are pretty familiar to any programmer out there except, maybe, unsafe functions.

Naive For Loop with Index Accessor


    public int IndexAccessorVersion()
    {
        int cnt = 0;
        for(int i = 0;i < test.Length;i++)
        {
            if (char.IsWhiteSpace(test[i]))
                cnt++;
        }
        return cnt;
    }

Foreach Loop


    public int ForeachVersion()
    {
        int cnt = 0;
        foreach(char c in test)
        {
            if (char.IsWhiteSpace(c))
                cnt++;
        }
        return cnt;
    }

LINQ Expression on String


public int LINQStringVersion() => test.Count(c => char.IsWhiteSpace(c));

LINQ Expression on Char Array


public int LINQCharArrVersion() => test.ToCharArray().Where(c => char.IsWhiteSpace(c)).Count();

Unsafe Looping without Unrolling

An unsafe function is a block of function that enables managed code to use unmanaged facilities. We can obtain raw pointers with the fixed keyword to perform pointer arithmetic for iteration and dereference the acquired pointers to get the data at the address.


    public unsafe int UnsafeVersion()
    {
        int cnt = 0;
        fixed(char* p = test)
        {
            for (int i = 0; i < test.Length; i+=1) 
            {
                if (char.IsWhiteSpace( *(p+i) )) 
                    cnt++;
            }
        }
        return cnt;
    }

Benchmark

Setup

We need to do 3 things:

Create a simple class that holds a string which contains the methods above. These methods should be marked with attribute Benchmark.


[MemoryDiagnoser] // Report Allocations and GC Generations
public class WhiteSpaceCounter
{
    private string test {get; set;}
    [Benchmark]
    public int LINQCharArrVersion() => test.ToCharArray().Where(c => char.IsWhiteSpace(c)).Count();
    [Benchmark]
    public int ForeachVersion()
    {
        int cnt = 0;
        foreach(char c in test)
        {
            if (char.IsWhiteSpace(c))
                cnt++;
        }
        return cnt;
    }

}

This string should be initialized at the constructor.


 public WhiteSpaceCounter()
    {

        test = "Split your lungs with blood and thunder when you see the white whale";
        for (int i = 0; i < 10; i++)
        {
            test += test;
        }
    }

BenchmarkRunner's static Run method should be invoked with a template argument of our class.
```
BenchmarkRunner.Run<WhiteSpaceCounter>();
```

Results

Change project build target to release and run.

As you can see, LINQ version is the slowest. We can fasten it up with an explicit conversion to Char Array which in turn hogs more memory. Naive loop and foreach version are faster than LINQ. Finally, unmanaged version, as you can expect, is the fastest.

What is the fastest way to iterate through individual characters in a string in C#?

Jan 9 '12 Comments: 8 Answers: 6

The title is the question. Below is my attempt to answer it through research. But I don't trust my uninformed research so I still pose the question (What is the fastest way to iterate through individual characters in a string in C#?).

Occasionally I want to cycle through the characters…

Open Full Question

This is it for now. Any corrections are welcome.

Runs Test for Randomness Testing in Python

Sıddık AÇIL — Mon, 11 Nov 2019 20:13:04 +0000

Hello there,

This is my first post on this site. I've been running a Turkish/English blog on Medium platform for 28 months now and have just discovered this site. I absolutely fell in love with its retroesque design, so here we are. As you can see, I am looking forward for job opportunities abroad. Feel free to contact me for any vacancies.

Runs Test Implementation in Python | by Sıddık Açıl | Medium

Sıddık Açıl ・ Oct 28, 2019 ・ 2 min read
Medium

Methodology

Runs test is a hypothesis testing based methodology that is widely used in statistical analysis to test if a set of values are generated randomly or not. It is a hypothesis test so we have a pair of a null hypothesis and an alternative one.

Null hypothesis: The values are randomly generated.

Alternative hypothesis: The values are NOT randomly generated.

A Z score for hypothesis can be acquired by simply following the general formula:

(Observed-Excepted) / Standard Deviation

The score is then tested against the confidence interval(two-tailed) we specify. If the value is higher, we conclude that our alternative hypothesis holds. Otherwise, if the value is lower, we cannot say anything about the randomness of data at this significance level. We will be using %95 confidence interval(alpha = 0.05) through the rest of this article.

Definitions and Formulas

A run: A series of positive or negative values:

Data: [1, -2, -3, 4, 5, 6, –7]

Runs: [[1], [-2, -3], [4, 5, 6], [-7]]

Score Formula

(Number of runs - Excepted Value for Number of Runs) / Standard Deviation

Expected Value Formula for Runs Test

n_p = Number of positive values
n_n = Number of negative values
Excepted value of runs = (2 * n_p * n_n) / (n_p + n_n) + 1

Standard Deviation Formula for Runs Test

n_p = Number of positive values
n_n = Number of negative values
Excepted value of runs = (2 * n_p * n_n) / (n_p + n_n) + 1

Test Data

I will using the data provided by NIST.

Octave/Matlab Implementation and Results

Using online Octave:

Upload text file
Load ‘statistics’ package

pkg load statistics
Import data to array

x = importdata("test.txt")
Run runstest

[h, v, stats] = runstest(x, median(x))

Python Implementation

Implement the test using Python.

Running the code produces the results below (Calculated Z Score, Z Score at %95 confidence):

(2.8355606218883844, 1.6448536269514722)

Since our test score is higher, alternative hypothesis holds. This means that our values are genuinely random.

Thanks for reading. Any corrections are welcome.