DEV Community: Wanjala

Walmart Store Sales Forecasting -DS Challenge

Wanjala — Sat, 28 Aug 2021 09:45:17 +0000

There are key points we need to understand while predicting the sales.

It is easier to make forecasts for the near future than for the future 200 days from now. Farther the future ,farther we are from being accurate.
It is good to use multiple techniques for forecasting than just relying on one.
It is challenging to predict sales when we obtain less historical data. More the history of sales at hand more we analyze deeper patterns in customer demands.

Business Problem

The sales prediction problem was launched by Walmart via Kaggle as a part of it’s recruitment process to predict their weekly sales for every department of all the stores.We were given historical sales data of 45 stores and 81 departments for every store.

Business Objectives and Constraints

Robustness Ensuring using Cross validation

Interpretability As we forecast sales and hand it over to buinesses,it is important to answer Why’s ? and How’s.

Accurate prediction matters as it might impact taking business decisions

No latency requirements (i.e) there is not much demand in predicting sales in milli seconds.

Models should be Evolutionary as consumer demands/supplies can change over time.

Data Overview

Data was collected from year 2010–2012 for 45 Walmart stores. We are tasked with predicting department wise sales for each store. We are provided with datasets : Stores.csv,train.csv,test.csv,fetaures.csv

Dataset Size

Trainset : 421570 rows (12.2MB)
Testset : 115064 rows (2.47MB)
Stores: 4KB
Features: 580 KB

Understanding the type of Machine Learning Problem that we are dealing with

It’s a regression problem where we are given time series(from 05–02–2010 to 26–10–2012) data with multiple categorical and numerical features.We are tasked to predict department wise sales of each store from 02- 11–2012 to 26–07–2013 . In addition, Walmart runs several promotional markdown events throughout the year. These markdowns precede prominent holidays, the four largest of which are the Super Bowl, Labor Day, Thanksgiving, and Christmas. The weeks including these holidays are weighted five times higher in the evaluation than non-holiday weeks. Part of the challenge is modeling the effects of markdowns on these holiday weeks in the absence of complete/ideal historical data.

Away with the theories lets get staright to the Practical work now.

First I imported the libraries

Then I changed the style that allows me to do the plotting. This is specifically useful when it comes to plotting of data.

After that, I went straight into loading the data (stores.csv) to the loading the data into the notebook to facilitate manipulation

After which, I had to display the datasets and then describe them accordingly.

Data Cleaning

Data cleansing or data cleaning is the process of detecting and correcting corrupt or inaccurate records from a record set, table, or database and refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data (Wikipedia)

I then removed empty values and re-checked to confirm if there were other empty values

To improve visualization, I then deployed charts. Here, I calculated the maximum size, figures and sizes then build a plot and showed it. This becomes easier especialy when one has to do a skimming of the whole dataset which makes it easier.

Plotting the data by Store

Plotting the data by year.

AUTOMATICALLY LOAD ENVIRONMENT VARIABLES IN FLASK

Wanjala — Sun, 15 Aug 2021 08:19:31 +0000

Get the code for this article here

With the introduction of the Flask command line interface, one of the more annoying things you have to do during development is set the environment variables each time you work on your app, namely FLASK_ENV and FLASK_APP. Well, Flask has a way for you to handle those things in a way to where you only have to do it once. And through the same functionality, you can also add other environment variables for your project. In this article, I'll show you how to use python-dotenv to have your environment variables loaded and ready to go every time you run your app.

The one package that makes this all work is python-dotenv

What We'll Build

I'm going to demo how this works through a very simple Flask app.

Let's start by creating the directories and empty files we'll need. This is what our project directory will look like:

demo/__init__.py
demo/settings.py
.env
.flaskenv
run.py

This app won't do anything special. It'll just show us what configuration values we have.

Set Up The Project

To start, on the command line create a new virtual environment and install flask and python-dotenv. I use pipenv, so I can do this all in one step:


pipenv install flask python-dotenv
pipenv shell

Next, in your demo/init.py, we'll need to create the basics of a Flask app, which means importing Flask, creating a application factory function (create_app), and instantiating Flask. Here's the code to do this.

# demo/__init__.py
from flask import Flask 

def create_app():
    app = Flask(__name__)

    return app

If you've worked with Flask at all, then you know exactly how this works.

Next, let's add in a simple route so we know our app actually works.

# demo/__init__.py
from flask import Flask 

def create_app():
    app = Flask(__name__)

    @app.route('/')
    def index():
        return '<h1>Hey There!</h1>'

    return app

Add FLASK_APP Environment Variable

Once we've done that, we can use the Flask CLI to run the app. To run the app, use the following command while in the top level directory of your project.

flask run

But when we do that, we already get an error!

Error: Could not locate a Flask application. You did not provide the "FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not found in the current directory.

This is a common error though. Normally, to fix this, we would export the environment variable FLASK_APP to be equal to the name of our app directory like this:

export FLASK_APP=demo

But since we're using python-dotenv and we want to avoid the inconvenience of using the command line, we'll have this load automatically by putting it in one of our dot files.

We have two dot files: .env and .flaskenv. We want to use .flaskenv for any Flask CLI configuration commands and use .env for our app configuration.

We'll put our FLASK_APP enviornment variable inside of the .flaskenv file.

#.flaskenv
FLASK_APP=demo

Now that we've added that one line, let's try running the app again.

flask run
We should be able to navigate to localhost:5000 or 127.0.0.1:5000 to see the result.

Note the message that appears on the command line that says we aren't in debug mode

Now if we were to stop working on this project and come back later, all we would need to do is use flask run and the app will start up without having to specify its location again.

Flask CLI Options

Let's say we want to modify some other Flask CLI options, we can put those in our .flaskenv file.

Here are some options:

FLASK_ENV - Controls the environment.
FLASK_DEBUG - Enables debug mode.
FLASK_RUN_EXTRA_FILES - A list of files that will be watched by the reloader in addition to the Python modules.
FLASK_RUN_HOST - The host you want to bind your app to.
FLASK_RUN_PORT - The port you want to use.
FLASK_RUN_CERT - A certificate file for so your app can be run with HTTPS.
FLASK_RUN_KEY - The key file for your cert.

I think the most common ones you'll use are FLASK_ENV, FLASK_RUN_PORT, and FLASK_RUN_EXTRA_FILES, so I'll create examples with them here.

Let's start with FLASK_ENV. By default, it's 'production', which doesn't do anything noticeable. The point of the environment though is so you can decide on what actions to take in your app depending on the environment. For example, you could do something like instantiate the Sentry library for error logging when in production but not development.

If you change your FLASK_ENV to development, the biggest change you'll see is the reloader starts working and your app gets put into debug mode. So let's set that value in our .flaskenv file.

#.flaskenv
FLASK_APP=demo
FLASK_ENV=development

Let's stop the server and run our app again.

flask run

Notice how the message changes now.

Let's add another one for the port.

#.flaskenv
FLASK_APP=demo
FLASK_ENV=development
FLASK_RUN_PORT=8080

After restarting, we'll have to access our on through port 8080:

What if we don't want to have to stop and restart our app when we change non-py files in our project? We can use FLASK_RUN_EXTRA_FILES for that. For example, to watch a files called config.yml and README.md

FLASK_RUN_EXTRA_FILES=config.yml:README.md

(If you are on windows, you'll need to use a semicolon (;) instead of a colon (:) to separate multiple files)

Adding in .env

Now that we've worked with .flaskenv, let's move on to adding variables to the .env file. We'll use these in our app.

One advantage of having .env hold our app environment variables is that we can easily have multiple versions of the file for each place we run our app. For example, we can have one for our local machine, one for a staging environment, and one for production, etc.

Let's add two:

#.env
SECRET_KEY=topsecretkey
API_KEY=donotsharethisapikeywithanyone

Unlike the values prepended with FLASK_ in our .flaskenv file, the variables in .env aren't meant to work for us automatically. Instead, we need to load them into our app. For that, we'll use the settings.py file.

Inside of settings.py, let's import environ from os so we have access to our environment variables.


#demo/settings.py
from os import environ

Next, we need to assign those variables to Python variables, and then later we'll load them into our app.


#demo/settings.py
from os import environ 
SECRET_KEY = environ.get('SECRET_KEY')
API_KEY = environ.get('API_KEY')

We don't need to use the same name for the variable in Python as we do for Python, but I like to do it for consistency purposes.

Next, let's add the configuration from this file in our init.py.

# demo/__init__.py
from flask import Flask 

def create_app():
    app = Flask(__name__)

    app.config.from_pyfile('settings.py')

    @app.route('/')
    def index():
        return '<h1>Hey There!</h1>'

    return app

Now our environment variables are ready to use. In the case of SECRET_KEY, normally some Flask extensions we use would pick this up and use it automatically. For the API_KEY, we would need to use this directly in our code.

In our code, we'll simply display the API_KEY in our route.

# demo/__init__.py
from flask import Flask 

def create_app():
    app = Flask(__name__)

    app.config.from_pyfile('settings.py')

    @app.route('/')
    def index():
        return f'API_KEY = { app.config.get("API_KEY") }'

    return app

Running in Production

If you want to use your .dotenv files in production, you have to make a small change because you won't be using the Flask CLI on a production server.

If you want to see this work, you can install gunicorn so you don't have to use the development server for Flask.

pipenv install gunicorn
We need to create a file that gunicorn can find the app object in. For this, we'll create run.py. Let's start with the standard code that doesn't load the environment variables, which is only two lines.

#run.py
from demo import create_app

app = create_app()
gunicorn run:app

This should run on port 8000 by default. If you navigate to the index, you'll see the value of API_KEY is None.


#run.py
from demo import create_app
from dotenv import load_dotenv

load_dotenv('.env') #the path to your .env file (or any other file of environment variables you want to load)

app = create_app()

Now if we start the server with gunicorn again we'll see our API_KEY value

Conclusion

Now you should understand how python-dotenv can be used in your Flask project to make handling Flask CLI configuration and general app configuration more convenient.

What we covered:

How to use Python-dotenv for Flask development server options.
How to use Python-dotenv for application specific configuration.
How to use it in production.

You'll be able to leave and come back to your project without having to re-set your environment variables, and you'll have a flexible way of having multiple configurations depending on where your app is running

Introduction To Python Functions

Wanjala — Mon, 02 Aug 2021 12:35:51 +0000

What is a Python function?

A Python function is a block of code which only runs when it is called. You can pass data, known as parameters, into a function.
A function can return data as a result.

How do we create this functions?

In Python a function is defined using the def keyword:

Example

def my_function():
    print("Enjoy functions")

In-built Functions.

These are defined as functions whose functionality is predefined in python. The python interpreter has several functions that are always readily available for use whenever the need arises. Some examples of these built-n functions are;

Python abs() function.

This function is used to return the absolute value of a number. It only takes one argument and that is the number whose absolute value is to be returned. The argument can either be an integer or a floating point number. If the argument happens to be a complex number, then the abs() function returns its magnitude.
The abs() function, furthermore, ignores a negative sign that precedes a number. This is to mean that if the number is for example -4 abs() returns 4. Returns positive numbers.

# Integer.
a = -34
print(abs(a))
# prints 34.
# floating point.
 b = -45.67
print(abs(b))
#prints 45.67

Python float() function.

This is a built in python function that is used to convert a string or integer to a float point value. The float() function takes one parameter, that is the value you want to convert to a float. Passing on the parameter in this case is optional. The default value is 0.0 .
Converting an integer to a float in python is achieved as in the following example.

a = 45
print(float(a))
# prints 45.0
And converting a string, which is nothing but just a sequence of characters, is don as in the example below.
b = '32'
print(float(b))
# prints 32.0

Python sum() function.

This is a python function that sums up all the numerical values in an iterable such as a list and returns the total of such values. The sum() function calculates the total of both floating point numbers and integers.
The sum() function can take two parameters;
i. The iterable object that you would like to calculate the total of, this parameter is required.
ii. An extra number you want to add to the value you are calculating, this is optional.

a = [1,2,3,4]
total = sum(a)
print(total)
# prints 10
# two parameters.
a = [1,2,3,4]
total = sum(a, 1)
print(total)
# prints 11.

User Defined Functions.

As the name suggests, these are created by the user. These functions are created in a certain way unlike the in-built functions. User defined functions involves the use of keyword. The def keyword is used to create these functions. It is followed by a function name declared by the user. The function name is subject to the rules governing the naming of variables.
The following is a simple syntax to define function;

def hello():
    print("Hello world!")
# prints hello world

The above code is a simple function with no parameters and arguments passed.

Arguments

Information can be passed into functions as arguments.
Arguments are specified after the function name, inside the parentheses. You can add as many arguments as you want, just separate them with a comma.
The following example has a function with one argument (fname). When the function is called, we pass along a first name, which is used inside the function to print the full name:

Example


def my_function(fname):
    print(fname + " Refsnes")

my_function("Brian")
my_function("Junior")
my_function("Lin")

Parameters and Arguments? Let's get started.

A parameter is the variable listed inside the parentheses in the function definition.

def my_function(fname):
#**fname** represents a parameter

An Argument?

An argument is the value that is sent to the function when it is called.

my_function("Brian")
#**Brian** represents the argument.

Number of Arguments

By default, a function must be called with the correct number of arguments. Meaning that if your function expects 2 arguments, you have to call the function with 2 arguments, not more, and not less.

Example

This function expects 2 arguments, and gets 2 arguments:

def my_function(fname, lname):
    print(fname + " " + lname)

my_function("Brian", "lux")

Note:

If you try to call the function with 1 or 3 arguments, you will get an error:

Arbitrary Arguments, *args

If you do not know how many arguments that will be passed into your function, add a * before the parameter name in the function definition.
This way the function will receive a tuple of arguments, and can access the items accordingly:

Example

If the number of arguments is unknown, add a * before the parameter name:

def my_function(*kids):
    print("The youngest child is " + kids[2])

my_function("John", "Andy", "Lin")

Default Parameter Value

The following example shows how to use a default parameter value.
If we call the function without argument, it uses the default value:

Example:

def my_function(country = "Norway"):
    print("I am from " + country)

my_function("Sweden")
my_function("India")
my_function()
my_function("Brazil")

Passing a List as an Argument

You can send any data types of argument to a function (string, number, list, dictionary etc.), and it will be treated as the same data type inside the function.
E.g. if you send a List as an argument, it will still be a List when it reaches the function:

Example:

def my_function(food):
    for x in food:
        print(x)

fruits = ["apple", "banana", "cherry"]
my_function(fruits)

Return Values

To let a function return a value, use the return statement:

Example:

def my_function(x):
    return 5 * x

print(my_function(3))
print(my_function(5))
print(my_function(9))

The pass Statement

function definitions cannot be empty, but if you for some reason have a function definition with no content, put in the pass statement to avoid getting an error.

Example:

def myfunction():
    pass

Recursion

Python also accepts function recursion, which means a defined function can call itself.
Recursion is a common mathematical and programming concept. It means that a function calls itself. This has the benefit of meaning that you can loop through data to reach a result.
The developer should be very careful with recursion as it can be quite easy to slip into writing a function which never terminates, or one that uses excess amounts of memory or processor power. However, when written correctly recursion can be a very efficient and mathematically-elegant approach to programming.
In this example, tri_recursion() is a function that we have defined to call itself ("recurse"). We use the k variable as the data, which decrements (-1) every time we recurse. The recursion ends when the condition is not greater than 0 (i.e. when it is 0).
To a new developer it can take some time to work out how exactly this works, best way to find out is by testing and modifying it.

Recursion Example

def tri_recursion(k):
  if(k > 0):
     result = k + tri_recursion(k - 1)
     print(result)
  else:
    result = 0
  return result

print("\n\nRecursion Example Results")

Return Values

To let a function return a value, use the return statement:

def sum(x):
  return 4 + x

print(sum(1)) # 5
print(sum(3)) # 7
print(sum(10)) # 14

Python Basics, Pythons 101!

Wanjala — Sat, 24 Jul 2021 17:27:00 +0000

What is Python?

Python is an interpreted high-level general-purpose programming language. It's design philosophy emphasizes code readability with its notable use of significant indentation.
Python is dynamically-typed and garbage-collected as it supports multiple programming paradigms, including structured, object-oriented and functional programming.
It was developed by Guido van Rossum who started working on it in 1980 as the ABC successor.

Common Features for Python

Easy to Code: Python is very easy to learn as compared to other languages like C, C#, JavaScript, Java, etc. It is also a developer-friendly language; this is because it has a very simple and defined syntax.
Open Source: A very powerful language and it is free and available for the public to use and alter as needed.
Portability: Python code can be shared across multiple platforms where by, if we have python code for windows and we want to run this code on other platforms such as Linux, Unix, or even Mac then we do not need to change it, we can run this code on any platform.
Being Integrated: Python language makes it easier to use as it can be integrated with other languages to deliver on-point solutions and with ease.
Huge amount of libraries: Python has a large standard library which provides a rich set of module and functions so you do not have to write your own code for every single thing. There are many libraries present in python for such as regular expressions, unit-testing, web browsers, etc.
Object Orientation: One of the key features of python is Object-Oriented programming. Python supports object-oriented language and concepts of classes, objects encapsulation, etc.

Applications of Python Programming Language

AI and Machine learning
Data Analytics
Data Visualization
Web development
Game Development
Fin-Technology
Fields of SEO's

Installation

Download the latest version of Python for your operating system here. You can also read more about setting up a python development and its environment path from this tutorial by pythontutorial.net

Writing your first code

With Python it is hard to imagine complex problems can be solved with fewer lines of code. That can boost your productivity as a programmer, you worry less.

eg. we are going to write our first program.

First, create a new folder where you will be saving you python files, let say FirstApp.
Second, Fire up your VS Code and open the new folder you created, FirstApp.
Third, create a new python file, let say app.py file and enter the following code and save the file:

print('Hello, World!')

To execute our Python program;
To execute the app.py file we created above, you first launch the Command Prompt on Windows or Terminal on macOS or Linux.

Navigate to the folder containing our file, for our case FirstApp.

After that, type the following command to execute the app.py file:

Python3 app.py

If everything is fine (Environment installed and paths clearly defined) , you’ll see the following message on the screen:

Hello, World!

And that is it! Pretty easy right?

What did you learn from the post today?

Feel free to share with me your thoughts and any questions, I'd be glad to answer them all. Plus also, I never stop learning, so am open to new ideas from you, we grow together.

For more information about learning python basics feel free to reach out to amazing resources from Lux

DEV Community: Wanjala

Walmart Store Sales Forecasting -DS Challenge

Business Problem

Business Objectives and Constraints

Data Overview

Dataset Size

Understanding the type of Machine Learning Problem that we are dealing with

Away with the theories lets get staright to the Practical work now.

First I imported the libraries

Then I changed the style that allows me to do the plotting. This is specifically useful when it comes to plotting of data.

Data Cleaning

Plotting the data by Store

Plotting the data by year.

AUTOMATICALLY LOAD ENVIRONMENT VARIABLES IN FLASK

What We'll Build

Set Up The Project

Add FLASK_APP Environment Variable

Flask CLI Options

Adding in .env

Running in Production

Conclusion

What we covered:

Introduction To Python Functions

What is a Python function?

How do we create this functions?

Example

In-built Functions.

Python abs() function.

Python float() function.

Python sum() function.

User Defined Functions.

Arguments

Example

Parameters and Arguments? Let's get started.

An Argument?

Number of Arguments

Example

Note:

Arbitrary Arguments, *args

Example

Default Parameter Value

Passing a List as an Argument

Example:

Return Values

Example:

The pass Statement

Example:

Recursion

Recursion Example

Return Values

Python Basics, Pythons 101!

What is Python?

Common Features for Python

Applications of Python Programming Language

Installation

Writing your first code

What did you learn from the post today?

See you next time on my next post!