DEV Community: Ajitesh Tiwari

How much I have paid to Uber?

Ajitesh Tiwari — Tue, 05 Jun 2018 19:34:33 +0000

Being a developer I feel thrilled to watch conferences on machine-learning, demonstrating life like chatbots or super awesome image analysis tools.

But personally I have failed multiple times in making a start towards handling/analyzing data, which as many say is the base of ML. Learning about data-analysis can be tough to start. Most of the available tutorials took me so deep in complex math formulas that I was not able to follow after a few lectures.

This weekend I decided to do something basic without any knowledge about python or other tools. Had to search solutions all along the way, but was a nice experience over-all.

So, I decided to scrape my historical data from Uber's website and try to find some answers like -

How much I have paid to Uber in total?
How much time I have spend on Uber rides or waiting for rides?
How much I have saved by choosing ride-sharing as an option?
How much I have paid per km for various Uber rides?

Although data may be obtained from Uber developer APIs. But for learning important concept of web-scraping I decided to take a longer path.

And much more, depending on how much data you scrape and how much investigative you are :D
So let's start

Step 1 - Scrape data from Uber's history

We can get uber's history from this link. This is a paginated endpoint which has a list of rides. Most of the information is available on this page itself.

To make things more simple we have a plugin available - Uber data extractor. We just need to open this link and click on the plugin. Within some-time we will have an excel sheet in our downloads with most of the information we need.

This plugin iterates through the paginated endpoint -https://riders.uber.com/trips?page={page-number} and fetches data to save it in an excel sheet. After collecting information we have following columns - trip_id, date, date_time, driver, car_type, city, price, payment_method, start_time, start_address, end_time, end_address.

But things are not finished yet. Two important parameters are still left to be captured - distance and trip-time.
This data is available on details page of each ride, which can be accessed from https://riders.uber.com/trips/{trip-id}.

Thankfully we have trip-ids in the data collected from above chrome plugin, using which we can generate an array of links we need to visit.

Now we can use selenium and chrome-web-driver to automate opening of links and printing of fields required. Since I am not so strong in python, please ignore if I have made few basic mistakes.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import time


option = webdriver.ChromeOptions()

browser = webdriver.Chrome(executable_path='/Users/atiwari4/Downloads/chromedriver', chrome_options=option)

browser.get("https://riders.uber.com/trips")
time.sleep(60)

Above code tries to take you to https://riders.uber.com/trips link, which in turn asks for authentication. We need to login into uber's account in order to process next steps.
Thus, I have added time.sleep(60). It provides us with a buffer of 60 secs to help us complete the login process.

for a in arr:
    row = []
    browser.get(a)
    time.sleep(3)
    distance = browser.find_elements_by_xpath("/html/body/div[2]/div/div[2]/div/div/div[2]/div[3]/div/div[1]/div[2]/div[2]/div/div[2]/h5")
    trip_time = browser.find_elements_by_xpath("/html/body/div[2]/div/div[2]/div/div/div[2]/div[3]/div/div[1]/div[2]/div[2]/div/div[3]/h5")
    row.append(a)
    distancex = [x.text for x in distance]
    row.append(distancex)
    trip_timex = [x.text for x in trip_time]
    row.append(trip_timex)
    print row

Here arr is array of ride detail page links.

Using XPath we pick the required elements from the page and print them on console. Above code may take time depending on number of trips you have completed. For more info on web-scraping using selenium, refer this article.

Now my data-sheet looks something like this.

Step 2 - Analyze data using Google Colaboratory

Google colabs is a cloud environment which can be used to run live code.

We can import sheets in colabs using various methods. Since we have our sheet already uploaded on google-sheets, we can directly access it from there.

First we need to authenticate our google account with colabs.

!pip install --upgrade -q gspread
!pip install -U matplotlib

from google.colab import auth
auth.authenticate_user()

import gspread
from oauth2client.client import GoogleCredentials

gc = gspread.authorize(GoogleCredentials.get_application_default())

Once we are authenticated this is how we import the sheet from google-sheets

# Open our new sheet and read some data.
worksheet = gc.open('Uber').sheet1

# get_all_values gives a list of rows.
rows = worksheet.get_all_values()
print(rows)

# Convert to a DataFrame and render.
import pandas as pd
df = pd.DataFrame.from_records(rows)
df.head()

df stands for DataFrame which is provided by Pandas. According to there documentation it is -

Two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns). Arithmetic operations align on both row and column labels. Can be thought of as a dict-like container for Series objects.

head(n=5) function returns top n (default 5) rows of the data-frame.

The first row in our sheet is wrong and should me made header for the sheet.

df.columns = df.iloc[0]
df = df.reindex(df.index.drop(0))

Now we need to define a column which has unique values.

df['trip_id'].is_unique
df = df.set_index('trip_id')
df.head()

The schema looks better now. But what about data. We need to clean it, before we start looking for answers. For example, the price column is something like ₹89.53. We need to remove first 3 characters from that column.

df['price'] = df['price'].str.slice(3)

Next we need to remove comma from price column in order to convert its datatype to number. Also we need the data-type of date column to be of date-time.

df['price'] = df['price'].str.replace(',','')
df['price'] = pd.to_numeric(df['price'])
df['date'] = pd.to_datetime(df['date'], format='%m/%d/%y')

Now we will have a lot of columns as NaN. Those are especially the cancelled rides, and we need to get rid of those.

import numpy as np
df['end_time'].replace('', np.nan, inplace=True)
df['end_time'].replace('Dropoff time unknown', np.nan, inplace=True)
df['start_time'].replace('', np.nan, inplace=True)

df.dropna(subset=['end_time'], inplace=True)
df.dropna(subset=['start_time'], inplace=True)
df.dropna(subset=['price'], inplace=True)

Finally the question of decade :D

How much I have paid to Uber?

Just type this to get your answer -

df['price'].sum()

Turns out that I have paid Rs. 26852.03 to Uber(involving Uber Eats). We can also segregate various uber services and see what have we paid for them individually.

To know exact number of uber services availed with count we need to run following code -

print pd.value_counts(df['car_type'].values, sort=True)

Here is the result -

We can also plot graphs in colabs using matplotlib library. Here is an example of a time vs price graph.

A thousand more answers can be retrieved from this dataset. Feel free to try new combinations and comment in the section below.

Click here for colabs sheet with many more inferences.

Go functional with Java

Ajitesh Tiwari — Thu, 05 Apr 2018 07:46:21 +0000

Java has been the most popular language since it's launch, because it's creators have made sure that the language doesn't miss on anything and sustains it originality with the emerging changes.

Functional Programming

Functional programming is a programming concept that is inspired from lambda calculus. In this concept each computation is considered as a function. Although these functions should not be allowed to change state/data outside their scope.

Why?

Software development is an iterative process, which involves not only writing code but also understanding code written by others.

You may find it challenging and vice-versa :D

It can take a lot of time just to figure out the state of a particular object, if it is been changed by functions that use it as an argument. Hence making it difficult to predict the behavior of a program.

Consider it like a third-party API without documentation. You are not sure of what behavior to expect when calling a particular function.

It is not same with concept of functional programming. By not allowing changing state and mutable data it avoids side-effects.

How?

All this theory is fine, but how do I use it as a Java developer?

Most of you must be thinking about this question. The answer is a new member of java family Lambda.

Lambda

Lambda is a function which acts like an object. Can be passed anywhere. Can be executed whenever needed. Can be defined in a single line. Can be created without a class. And much more. It helps to remove a lot of boilerplate code previously required by Java.

Syntax - (Parameter Declaration) -> {Lambda Body}

Examples -
Without parameters - () -> System.out.println("Hello lambda")
With one parameter - s -> System.out.println(s)
With two parameters - (x, y) -> x + y
With multiple line body - (x, y) -> {}

Cool right? but how Java knows which lambda gets mapped to which function?

Digging Deep

Since interfaces are the backbone of lambdas. Java has introduced a new concept known as Functional Interface against which a lambda is defined.

Functional Interface

They are similar to normal interfaces in java with one major difference. They follow SAM rule. According to SAM rule an interface is allowed only single abstract method (SAM).

This check can be enforced at compile time using @FunctionalInterface annotation.

For each lambda we need to create a new functional interface?
That's in-convenient. Right?

Looks like Java has already taken care of this issue as well.
Package java.util.function contains almost 50 functional interfaces. It is highly unlikely that you may need something out of their scope.

Family of functional-interfaces

Mastering all the interfaces may seem like a lot. Rather if we just understand their families, we can point to the right interface easily.

There are 4 families of functional-interfaces -

Consumer - Consume and Discard

@FunctionalInterface
public interface Consumer<T> {
    void accept(T t);

    default Consumer<T> andThen(Consumer<? super T> after) {
        Objects.requireNonNull(after);
        return (T t) -> { accept(t); after.accept(t); };
    }
}

It accepts an object, performs some action and returns no output. So mean :D

Example -

Consumer<String> stringConsumer = string -> System.out.println(string);

Since lambda function and println() both accept same arguments, this can also be written using Method reference, like this -

Consumer<String> stringConsumer = System.out::println;

Function - Map or Transform or Compute

@FunctionalInterface
public interface Function<T, R> {
    R apply(T t);

    default <V> Function<V, R> compose(Function<? super V, ? extends T> before) {
        Objects.requireNonNull(before);
        return (V v) -> apply(before.apply(v));
    }

    default <V> Function<T, V> andThen(Function<? super R, ? extends V> after) {
        Objects.requireNonNull(after);
        return (T t) -> after.apply(apply(t));
    }

    static <T> Function<T, T> identity() {
        return t -> t;
    }
}

It accepts one object, performs some action on it and returns another object.

Example -

Function<String, String> stringStringFunction = String::toUpperCase;

Predicate - Test or Filter

@FunctionalInterface
public interface Predicate<T> {
    boolean test(T t);

    default Predicate<T> and(Predicate<? super T> other) {
        Objects.requireNonNull(other);
        return (t) -> test(t) && other.test(t);
    }

    default Predicate<T> negate() {
        return (t) -> !test(t);
    }

    default Predicate<T> or(Predicate<? super T> other) {
        Objects.requireNonNull(other);
        return (t) -> test(t) || other.test(t);
    }

    static <T> Predicate<T> isEqual(Object targetRef) {
        return (null == targetRef)
                ? Objects::isNull
                : object -> targetRef.equals(object);
    }
}

It accepts one object, and returns a boolean. Usually defines rules.

Example -

Predicate<String> stringPredicate = String::isEmpty;

Supplier - Create

@FunctionalInterface
public interface Supplier<T> {
    T get();
}

It accepts nothing, but returns an object. So generous :D

Example -

Supplier<String> stringSupplier = () -> "Hello, World";

Real world

Now it's time to apply our knowledge to some real world challenge. Let's pick up a basic programming challenge from HackerRank - camelCase.

Problem - Count number of words in a camelCase string.

Example - saveChangesInTheEditor
Result - 5

So our motive is not only to solve this problem, but to solve it using the functional way.

The solution is straight forward, count the number of upper-case letter in the string and the result is count + 1.

To do this in functional way we need -

Stream of individual characters in the string, and
A predicate which helps filter the upper-case characters.

Solution -

/* Stream - s.chars()
Predicate - Character::isUpperCase */

static long camelcase(String s) {
    return s.chars().filter(Character::isUpperCase).count() + 1;
}

Hurrray !

A single line and the problem is solved.

This approach helps developers easily understand the behavior of the snippet. The above line can easily be understood to filter upper-case chars from a string and return value of (count of chars + 1), without any side-effects. Which is what we wanted.

Although becoming functional is not limited to the above example. It is something which is developed with experience, because it is about how we approach a problem.

Java developers may have to put in some extra effort to learn this approach, because our minds are trained to create more code than required. :D

Java 9 Flow API

Ajitesh Tiwari — Fri, 23 Mar 2018 04:45:36 +0000

Java 9 Flow API

Flow API is Java's official support for Reactive Streams Specification. It is a combination of both Iterator(Pull) and Observer(Push) patterns.

The Flow API is an inter operation specification and not an end-user API like RxJava.

If you are a developer(even an outdated one :D), you must have surely heard about reactive-streams, non blocking IO, asynchronous calls, etc.

So why Reactive?

Consider it as a secretary in a office -

If they makes you wait for meeting their boss - that's normal way.
If they asks you to go do something else and they contact you when their boss is free for meeting - that's the reactive way.

People generally tend to think any reactive programming as improvement in speed, but it is not true. Considering both scenarios above you have to spend equal time for meeting her boss (so no improvement in speed), but you can be more productive in the second approach.

Reactive programming increases scalability and stability but not speed.

Flow API

Flow API consists of 4 basic interfaces -

Subscriber - The Subscriber subscribes to the Publisher for the callbacks.

Publisher - The publisher publishes the stream of data items to the registered subscribers.

Subscription - Link between publisher and subscriber.

Processor - The processor sits between the Publisher and Subscriber, and transforms one stream to another.

Now it's time for some hands-on with our new Flow API

Let's create a basic subscriber which asks for one data object, prints it and asks for one more.

Now let's quickly use a publisher implementation provided by Java (SubmissionPublisher) to complete our session.

And here is our sweet output -

Here you can easily see how the submit(T item) function of publisher invokes onNext(T item) function of subscriber.

But it wouldn't have been possible without request(long n) function of the subscription.

The delay of 1 sec helps us see the reactive magic happening. Although it's no magic, just some code written by really smart people.

DEV Community: Ajitesh Tiwari

How much I have paid to Uber?

Step 1 - Scrape data from Uber's history

Step 2 - Analyze data using Google Colaboratory

How much I have paid to Uber?

Go functional with Java

Functional Programming

Why?

How?

Lambda

Digging Deep

Functional Interface

Family of functional-interfaces

Consumer - Consume and Discard

Example -

Function - Map or Transform or Compute

Example -

Predicate - Test or Filter

Example -

Supplier - Create

Example -

Real world

Hurrray !

Java 9 Flow API

Java 9 Flow API

So why Reactive?

Flow API

Now it's time for some hands-on with our new Flow API

Source Code