DEV Community: Rose Day

What advice would you give someone who wants to mentor younger developers? What are good attributes to focus on as a mentor?

Rose Day — Thu, 09 Jul 2020 19:59:39 +0000

Log SMS Data in Google Sheets using Twilio and Flask

Rose Day — Thu, 30 Apr 2020 01:38:52 +0000

Happy Wednesday evening and last day of the Twilio x Dev Hackathon! I hope you are all doing well. It has been great to spend the month of April working with Twilio and Google sheets on this project in order to capture SMS data from a survey.

For those who may not have seen the other posts in this series, you can view them above. The first post began as the initial start of the project in Tracking Migraines with SMS which is just an introductory post.

The setup of Google Sheets is discussed in Setting up Python to Connect to Google Sheets where I talked about API Errors due to scope along with path issues with PyTest.

My first experience with Flask is talked about in the post What I Learned using Flask for the First Time in which I discussed port issues, data types, casting data types, and Flask sessions. This section also details issues with getting to the next question in the survey.

The final integration of Google Sheets and Twilio is discussed in Integrating Twilio with Google Sheets. This post discusses clearing results in a list, and data import issues with Google Sheets appending single quotes at the start of numbers and dates.

With all of that said, todays post will focus on the wrap up of the Twilio x Dev Hackathon, along with a walk through and discussion of the application flow through the functions in app.py.

Quick Recap

For a quick recap on the project concept, I spend time logging migraines for the doctors office but don't remember the format she wanted, or the details she found important. This application was created as a way to request a survey for a migraine, quickly fill out the details, and log the data for later use at the doctors appointment.

Commonly Asked Questions for Migraines

Below is a list of commonly asked questions relating to migraines:

In a scale of 1-10, how would you rate your migraine?
Where is your migraine located today?
How long did your migraine last (in hours)?
What medication did you take to treat the migraine?
Has anything changed? Is anything out of the ordinary? Do you have any other notes to add?

Category - Engaging Engagements and Interesting Integrations

During this hackathon, I focused on two categories: Engaging Engagements and Interesting Integrations.

Engaging Engagements looked at developing applications that a company could implement to better engage with their customers or to manage their business. An application of this type could be utilized by doctors offices in order to collect data commonly needed during patient visits or to have patients collect their own data for use in their visits.

Interesting Integrations focused on the integration of the Twilio API with at least one other API. This application integrated Twilio with the Google Sheets API in order to log data in an easy to access format.

Filled out Twilio CodeExchange Agreement: ✔️
Agreed to the Competition's Terms: ✔️

Link to Code

To clone the code, please visit the GitHub project Migraine Tracking.

Demo Link

In the repository is also a GIF showing a brief demo of the code. There is also a README available for those who would like to run the code locally and try it out.

Application Flow

The first route that is reached when the application receives an SMS is the /sms route that calls the function sms_survey(). This function looks to see if a question ID comes in and in the session, the function will redirect to the /answer route.

Before a question ID will come in, the function will first default to collecting the Twilio number, the users number, and the date. Note: These were hidden in the demo data output in Google Sheets. These values are appended to the collected data list and then the a welcome message will be displayed to the user through the welcome_user() function. After displaying the welcome message, the user is redirected to the first question in the survey.

welcome_user(), seen below, will return a brief message to the user. Then the user is directed to the /question/<question_id> route.

The /question/<question_id> route takes in the question ID and tells the application which question it should respond back to the user with. Using the question ID, we pass the ID into the parseJson pulls out the first question and its type. The question ID is stored into a session variable called question_id to be used as the application looks for the next question or the end of the survey. Once that has been captured, the sms_twiml() function is called with question and type.

This function responds with the message and type of data to ask the user for. The three types of data asked for in this survey are text, hours, or a numeric number from 1 to 10.

After the user has answered the question, the application will redirect to the /answer/<question_id> route. This route first iterates to the next question ID which is used to grab data collected from the parseJson object. Then, the user entered data for the previous question is extracted and appended to the data_collected list. extract_content() returns either text for text elements or numeric digits for hours and numeric answers. If data is available for the next question, the survey will continue by redirecting to the /question/<question_id> route, otherwise the survey will end and display a goodbye message through the use of the goodbye_twiml() function.

The redirect_twiml() redirects the survey to the /question/<question_id> route to allow the user to view and answer the next question, repeating the process until all questions have been answered.

The goodbye_twiml() function ends the survey, appending all the data collected into the Google Sheet using the spreadsheet object. After a goodbye message is displayed to the user, the data collected is cleared from the data_collected list and the session variable for question_id is cleared.

And that is the overall flow of the code from first question to last.

Development Stack

Thank you for following along this past month as all of the pieces have come together to log the data into the Google sheet from the SMS survey.

Thank you to all those who put together this hackathon and participated in it. Have a Happy May!

Additional Resources

Reference Links

Twilio's Rest API
Automated Survey - Python Flask App
GitHub Sample Automated Survey
Google Sheets and Python Example
Accessing Google Spreadsheets with Python

Integrating Twilio with Google Sheets

Rose Day — Fri, 24 Apr 2020 00:03:29 +0000

Happy Thursday! I hope you are all doing well. It has been interesting to continue working with Twilio and Google sheets on this project for the hackathon. This week I focus on the issues and learnings I had with using Google Sheets with Twilio.

Quick Recap

I spend time logging migraines for the doctors office but don't remember the format she wanted, or the details she found important. This application was created as a way to request a survey for a migraine, quickly fill out the details, and log the data for later use at the doctors appointment.

Link to Code

To clone the code, please visit the GitHub project Migraine Tracking.

Issues and Learnings

After creating functions to work with the Twilio API and Google Sheets, it came time to integrate the two together. This process did not go smoothly at first.

Clearing Results from a List

While running the Flask app, I append the data collected to a list called data_collected and after answering one survey question and starting onto the next, the results would keep appending to the list. This behavior was expected, as, in the end, I wanted to take the answers to all questions and upload to Google sheets. What I did not realize would start to happen was that once a survey ended and the next started, the list continued to append the data.

Having not worked with Flask before, I found this to be an interesting issue that I did not expect to have. I had assumed the data from one survey would be cleared after the survey ended, allowing for the data from the next to start getting added. After realizing this was not the case, I searched on different ways to clear a list of data.

The first thing I had tried was to assign data_collected an empty list after the data was inserted into Google Sheets.

response = MessagingResponse()
sheet.insert_data(data_collected) # Put data into Google sheet
response.message("Thank you for filling out todays migraine... Feel better soon! :)")
data_collected = []

Doing so gave me the following error:

UnboundLocalError: local variable 'data_collected' referenced before assignment

After looking into this issue further, I found other ways to clear a list in Python that allowed me to remove old survey data once it had been stored in Google Sheets. The clear() method can be used with lists to remove all the elements from a list.

data_collected.clear() # Clear the list after it has been saved

Utilizing this method worked as expected and allowed the data to be removed after a survey has completed.

What did I learn here? The variable data_collected would continue to collect data as new surveys came in until the variable data was cleared. Using clear() on a list allows you to remove the data from the list and start fresh, meaning each survey would only collect the data needed for that survey, store it in Google Sheets, and clear the data before beginning the next survey.

Data Import has Single Quotes Appended to Numbers / Dates

The other main issue I had when integrating the two pieces was an interesting issue. When inserting data into Google Sheets, the numbers would be appended with one single quote.

For example:

'2415434
'3.0834536
'1

I found this problem to be interesting due to the fact that when I originally worked with the Google Sheets API and was able to pass data using the functions I made, I did not see this problem surface with any data that was passed.

With this, I spent more time looking into how data was being passed to a sheet and what could cause the extra character to be appended to the beginning of a number. When using Google Sheet's API there is an option to specify the value_input_option.

Looking into this option, I found that this issue could be solved by changing the function I was using to enter data into a sheet. I was utilizing insert_row() to add data into the sheet but the other option append_row() allowed for the insertion of the of the value_input_option.

    def insert_data(self, data: list):
        """
        Insert data provided into the sheet.

        :param data: List of data elements to insert into the sheet.
        :param index: Index to insert data into. Defaults to index 2, assuming the first index has headers.
        """
        try:
            self.sheet.append_row(data, value_input_option='USER_ENTERED')
        except:
            print('Cannot append data to sheet.')

After making the quick switch to the new function, I read further into what value to pass with this parameter. When looking more at the documentation for this option, I selected USER_ENTERED as it allows for the data to be parsed as if it were typed into a sheet, meaning numbers will stay as numbers and strings will be interpreted as either string, numbers, dates, etc.

What did I learn here? The issue of a single quote appending to the front of numbers was not coming from the code I had developed to collect the data but the way in which I was utilizing the functions(s) to append new data into the sheet. A simple switch to append_row() allowed the data to be considered as USER_ENTERED meaning when entering numbers or dates, that they would be interpreted as such in the sheet when uploaded.

Development Stack

Thank you for following along! We have all the pieces together to be able to collect data with a SMS survey using Twilio and append the data into a Google Sheet.

Stay tuned for next weeks post where I discuss the application as a whole and how I tied all the pieces together. I look forward to sharing with you the final issues and learnings I have encountered while participating in this hackathon!

Additional Resources

Reference Links

Twilio's Rest API
Automated Survey - Python Flask App
GitHub Sample Automated Survey
Google Sheets and Python Example
Accessing Google Spreadsheets with Python
Clearing Lists
Google Sheets Value Input Option

What I Learned using Flask for the First Time

Rose Day — Thu, 16 Apr 2020 18:18:17 +0000

Happy Thursday! I hope things are going well. I have enjoyed working through some (ha, many) issues while learning Flask to integrate Python with Twilio's API. This week I will focus on the issues and learnings I had with using both Flask and Twilio for the first time.

Quick Recap

At the moment, I spend time logging migraines for the doctors office but don't remember the format she wanted, or the details she found important. This application is being created as a way to request a survey for a migraine through SMS, quickly fill out the details, and log the data for later use at the doctors appointment.

Link to Code

To clone the code, please visit the GitHub project here. This project is a current work in progress and is not fully functional at this time, the app will walk through a survey but data is not currently saved anywhere.

Issues and Learnings

Port Issues

The first issue I ran into was connecting to a port when starting my application using python3 -m app run. I have run into this issue in the past when I use to work on servers and would get similar messages to the one below.

socket.error: [Errno 48] Address already in use

This just means a process is bound to the port I am trying to use. This is caused by the same Python app being called before and that process still being bound to the port. To fix this, I looked at what Python processes were still running using ps -fA | grep python.

Using this, you can spot the running Python process that is still active. You may want to test if http://localhost:<port>/ still shows a directory listing for local files. The second number is the process number that you will need to stop. This can be done using the command kill as in the example below:

ps -fA | grep python

>>  501 39548 16817   0  9:50PM ttys001    0:00.20 Python -m app run

kill 39548

>> [1]+  Terminated: 15          python3 -m app run

What did I learn here? This is caused by the same Python app being called before and that process still being bound to the port. This has only happened twice so far, and it is a simple and quick fix to just kill the process.

Type and Casting

The next issue was one I didn't realize was happening until I ran into it. When running the application, I would answer the first question sent by the survey and received the response below:

 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [08/Apr/2020 18:09:20] "POST /sms HTTP/1.1" 200 -
[2020-04-08 18:09:20,296] ERROR in app: Exception on /question/1 [GET]

TypeError: cannot unpack non-iterable NoneType object
127.0.0.1 - - [08/Apr/2020 18:09:20] "GET /question/1?

After an initial search into this issue, it was clear that results were all over the place. With that, I looked into the first question and coded up what I expected to be happening. I set the question_id to 1, read in the JSON for the survey, and passed the question ID into the survey to grab the question information for it. After returning the first question and its type, I printed the information.

from parse import parseJson

question_id = 1
surv = parseJson('../survey.json')
question, type = surv.question_metadata(question_id) 
print(question)
print(type)

When running this code, it produced the expected results, therefore something was wrong with the way the question_id was being passed. To recreated the problem outside of the Flask application, I changed the question_id to a string instead of int. Rerunning that code, reproduced the expected error:

TypeError: cannot unpack non-iterable NoneType object

What did I learn here? Therefore, the issue had been resolved. It was a type issue in which I was expecting an integer and receiving a string instead. Don't assume the type of the value being passed in, make it explicit.

Flask Session Secrets

As I had mentioned, I have not used Flask before, so I expected to end up with issues learning how to use it. The main issue I had when learning Flask was how to use session secrets. After resolving the above issues, the next that came up was the about not having a session secret setup.

    "The session is unavailable because no secret "
RuntimeError: The session is unavailable because no secret key was set.  Set the secret_key on the application to something unique and secret.

Sessions allow you to store user data from one request to the next. This was needed in the survey in order to store and track the question that the user was on and iterate to the next question.

What did I learn here? When researching sessions, I learned that the secret key is used in a session to implement on top of the cookies and is used so that the data cannot be modified without the key used for signing. Here is more information on Sessions.

Getting to the Next Question

The last issue I had came about when I was interacting with the code I developed previously to grab the JSON survey and read it. When implementing this code and trying to get to the next question in the survey, I ran into two main problems I had when working with the survey questions:

How to find out if there was another question in the JSON or not based on the currently stored question_id?
What does the app do if there is another question? What does the app do if there is not another question?

With that, let's take a look at question 1, how do you use question_id to determine if another question exists. To start, I looked back at the function I created in parse.py to see how I was gathering the data and was using a filter object which grabbed out the question whose ID matched and would return the body and type of the data.

 try:
            survey_dict = self._json_to_dict()['questions']
            data = list(filter(lambda item: item['id'] == question_id, survey_dict))

            if data:
                return data[0]['body'], data[0]['type']
        except:
            print("The index povided for the question does not appear to exist.")

After some further reading into this section of the code, I changed the line that filters to a way that will return a list instead of filter object. The line filter(function, iterable) is equivalent to [item for item in iterable if function(item)] which means I can change the code to substitute:

data = [item for item in survey_dict if (item['id'] == question_id)]

This is a more Pythonic way to find the next question in the JSON that we read in.

With that, we can get onto question 2, what does the app do now that it has the data for the next question? The data returned for the next question is either None or the next question's information. Using the question_id, the answer() function reads in the question ID given, (example: question 1 returns 1) and then iterates to the next questions ID by 1. Therefore, when question 1 is provided, the next ID would be 2. We use that ID, 2, and feed it into the surv.question_metadata() function which looks to see if the data returned is None or another question.

Before proceeding with that data collected, the answer() function extracts out the content from the previous question and stores it, which will be discussed more next week. After storing the data, the function looks to see if the data for the next question is None, which will end the survey, OR the if the data contains the next question, it will then pass the ID of the question and pass the ID to redirect_twiml() which will handle the response to the user.

@app.route("/answer/<question_id>", methods=["POST"])
def answer(question_id):

    id = int(question_id) + 1 # Cast to an int, comes in as a string
    data = surv.question_metadata(id) # Return first question
    data_collected.append(str(extract_content())) # Append the data collected

    if data:
        return redirect_twiml(id)
    else:
        return goodbye_twiml()

What did I learn here? After working through these two problems, I gained a better understanding in how Flask operates to redirect from one function to the next and how I needed to setup the survey to redirect correctly as the user answered each question. During this time, I began to understand more about Sessions as well, which I talked about earlier. The smallest tidbit of info I took away from this issue was that the line filter(function, iterable) is equivalent to [item for item in iterable if function(item)] which is a more Pythonic way to work with the data.

Development Stack

So far, I have worked on the code needed to interact with Google Sheets API and Twilio API in a Flask application...

Stay tuned for next weeks post where I discuss how I integrate the two API's together in the Python Flask app that generates a survey for migraines. I look forward to sharing with you the issues and learnings I have encountered while learning to use Flask with these API's!

Additional Resources

Reference Links

Twilio's Rest API
Automated Survey - Python Flask App
GitHub Sample Automated Survey
ngrok
How to Receive an SMS in Python with Twilio
SMS Conversation Tracking

Setting up Python to Connect to Google Sheets

Rose Day — Thu, 09 Apr 2020 23:29:33 +0000

Happy Thursday! I hope you are all doing well. It has been exciting to continue working with Twilio and Google sheets on this project. This week I will focus on the issues and learnings I had with using Google's APIs for the first time.

Quick Recap

Link to Code

To clone or view the code, please visit my GitHub project here. This project is a current work in progress and is not fully functional at this time.

Issues and Learnings

API Errors due to Scope

The first issue I ran into when trying to implement code to interact with Google Sheets through API was related to scope issues. I first assumed I had setup the configuration file wrong but later realized it was due to version changes on the API and referencing older tutorials. The issue I was seeing was as follows:

gspread.exceptions.APIError: {'errors': [{'domain': 'global', 'reason': 'insufficientPermissions', 'message': 'Insufficient Permission: Request had insufficient authentication scopes.'}], 'code': 403, 'message': 'Insufficient Permission: Request had insufficient authentication scopes.'}

After spending time researching this error, I came across a post that showed the scopes needed were the two shown below:

# Create a spread client authorizing it using those credentials.
scope = ['https://spreadsheets.google.com/feeds','https://www.googleapis.com/auth/drive']

What did I learn here? The reason this issue was occurring was due to an API update. gspread was upgraded and is based on API v4. The change allowed the API to be faster than previous versions but also required scopes to be updated in order to work properly.

Path Issues with PyTest

The next issue I ran across was when working with PyTest. The constructor of the class I setup to work with spreadsheets requires two inputs (1) filename of the JSON containing the credentials for the API which includes the path in the input, and (2) the name of the sheet to connect to.

When working with this constructor in the executor.py file, I provided the path seen below, which worked as expected. (Note: This code will later move to the app.py file when the Twilio API is introduced.)

sheet1 = spreadsheet('../client_secret.json', "migraine_tracker")

Utilizing the same line, I placed the code into a test file and came back with the warning WARNING: Failed to generate report: No data to report. meaning that no data could be read.

After seeing this, I installed pytest-cov to analyze the test coverage and see what lines were being hit and what lines were being missed. At this point, I realized the test coverage was stopping at an important line in the constructor:

credentials = ServiceAccountCredentials.from_json_keyfile_name(json, scope)

Therefore, the credentials were not being created due to this line failing and the sheet was not able to be called.

When looking back at the test for the line that provides the JSON file, I worked to determine why the path was not working correctly. To look further at the path issue, I created a small test that would assert if the path existed. I ran this test with both paths ../client_secret.json and client_secret.json, at which point I found out the path needed to be the latter to work properly in the test file.

import os.path
import pytest
import pprint

def test_path():
    """
    Determine if the path specified is correct.
    """
    assert path.exists('client_secret.json') # Produces True and pass 
    assert path.exists('../client_secret.json') # Produces False and fail

Therefore, I changed the input into the constructor which in turn fixed my issue and made it so I could run my tests successfully.

sheet1 = spreadsheet('client_secret.json', "migraine_tracker")

What did I learn here? Check the root directory. Pytest was running from rootdir: /../../../twilio_application and not the folder I was expecting it to run from. Seems like a trivial mistake when looking back at it, but it is something that can easily be overlooked.

Development Stack

So far, I have worked on the code needed to interact with Google Sheets API...

Stay tuned for next weeks post where I discuss how I utilize the Twilio API to develop a Python Flask app that generates a survey for migraines. I look forward to sharing with you the issues and learnings I have encountered while learning to use Flask!

Additional Resources

Reference Links

Google Sheets and Python Example
Accessing Google Spreadsheets with Python
Python Check if File or Directory Exists
pytest
pytest-cov

Tracking Migraines with SMS

Rose Day — Fri, 03 Apr 2020 20:00:31 +0000

Happy Friday! I am excited to try something new and begin working with Twilio's APIs.

Category - Engaging Engagements and Interesting Integrations

I am focusing on two categories during this hackathon: Engaging Engagements and Interesting Integrations.

Engaging Engagements looks at developing applications that a company could implement to better engage with their customers or to manage their business and Interesting Integrations focus on the integration of Twilio with at least one other API.

Overview

During this hackathon, I am looking for a simple method to collect data using SMS and Google Sheets. Every month I spend time logging migraines for the doctors office but don't remember the questions she commonly asks, or the details she found important from my last appointment. This application is being created as a way to log migraines in Google Sheets by quickly filling out the details of a survey sent through SMS. The user can then print off the sheet that has been populated and bring it with them to their next doctor's appointment.

Commonly Asked Questions for Migraines

In a scale of 1-10, how would you rate your migraine?
Where is your migraine located today?
How long did your migraine last (in hours)?
What medication did you take to treat the migraine?
Has anything changed? Is anything out of the ordinary?

Development Stack

I am working with the Twilio API to develop a Python Flask app that generates a survey for migraines and saves the data in a Google Sheet.

I look forward to posting each week my progress and learnings as I tackle this problem. Stay tuned for next weeks post as I begin to utilize Googles API to read and write data to a sheet!

Additional Resources

Reference Links

Twilio's Rest API

Software Engineering to Data Science

Rose Day — Sat, 28 Mar 2020 02:15:50 +0000

The other day I saw a question from @yadlra posted:

Hey guys, what skills do I need to be a data scientist? Do the same skills apply as being a software developer?

I posted a lengthy response but felt others would also benefit from a post on the topic.

In 2017, I graduated college as a computer engineer and began working full time before making the transition into data science. I have found the skills learned from software to be greatly transferrable to data science, especially when you are looking to build tools to work with big data.

Currently, I work on a team that focuses on tools and machine learning application, but there are many areas of data science that one could consider. Below, I outline some of the areas, skills, and resources that I looked into during my transition into the field.

Areas of Data Science

There are many fields in which data science can be applied. Recently, I have spoken at two colleges for talks on data science and was asked a series of questions:

Is there a field in which you cannot utilize data science?

My answer to this question was no. I believe that every field could utilize data science if the data is available, or in turn, be able to use it in the future, if the data is made available. I have even seen artists utilize data to create sculptures. Once you have picked a field of interest, you can find data related jobs in data science, data engineering, and more.

Did you study data science in your undergraduate or is it okay to have a Bachelors degree in another field first?

Like mentioned earlier, I did not study data science initially but transitioned to it when I went back for a masters. I have found that with most data scientists I have worked with, they have studied something else first before moving into data science. These individuals specialized in computer science, different areas of engineering, mathematics, and more. It is perfectly fine to specialize in one area and then use the knowledge gained there to further your data science education. You will bring different aspects to data science that others may not see due to your background.

How did you go about searching for jobs?

One thing I have learned while job searching in the field is that not all titles mean the same thing. Titles ranged from data scientist, data analyst, business analyst, data engineer, and more. I stopped searching by title and started searching by keywords as it helped me narrow down job descriptions that fit with what I was looking for. Keywords I used when searching focused on the languages I wanted to program in (Python), the tools I wanted to get exposure to or keep using (Tableau, Spark, Git), and the type of work I was looking for (IoT, Connected Systems, Sensors).

Another method of job searching that can be a valuable tool is networking, both virtually and attending meetups. For example, I have found LinkedIn to be a wonderful tool to network virtually with colleagues in my field, and to connect with them on what their companies are doing and the types of conferences or events they are attending.

What skills have helped you in your career and what tools have you used?

Once I found a field to specialize in, I began focusing on the skills I wanted to learn. The first job I had in data science was a co-op as a data engineer. During this time I worked to learn more about Docker for containerization of data applications, databases, and ETL processes. After this job, I began working as an engineer in applied data science. At this point, I began focusing my skills more on statistics / math, data cleaning, big data handling, data storytelling and visualizations, and tool development. Of these two jobs, tool development and application development was where I was able to leverage more of my software background, while more data analysis and visualizations were where I was able to use more of my data science background. During both jobs, I have found programming to be a valuable skill, especially object oriented programming and writing clean code. Skills can vary job by job but finding out what you enjoy doing will aid you not just in finding a job but in enjoying it too!

If you are looking for tools and frameworks to learn, here are just a few to consider:

Jupyter Notebooks for prototyping code
Git for version controlling notebooks and other code
Spark and Hadoop for big data processing
Databases and data storage for different types of data, understanding when to use what type of data store will be helpful
Tableau and other BI tools for dashboarding and visualizations
Python visualization libraries: Bokeh, Matplotlib, etc.
R and its respective libraries / R Shiny

What resources have you found helpful to continue learning outside of going to college?

A good book to look at when starting in data science is An Introduction to Statistical Learning: With Applications in R. This was a great read and very helpful when I was starting out. I still keep a copy at work as a reference for when I am working on something.

If you are looking for another good read that is more software focused, Clean Code by Robert Cecil Martin is a great book focused on how to write well structured code that is easy to read. The book focuses on Java to teach the skills but I apply the same concepts to Python.

Another good resource for books is the publisher O'Rielly. They have tons of good books on data science and Python. I have found their books to be helpful and keep some of them as reference at work as well.

I also recommend looking into open source data sets available to see what flavors of data exists and to determine your preference on what kind of data you want to work with. Some types of data include spacial data, image processing, structured or unstructured documents, etc.

There is so much you can look into with data science, just pick somewhere to start and run with it!

From Interviewee to Interviewer to Interviewee

Rose Day — Thu, 11 Jul 2019 17:57:36 +0000

Back when I began interviewing for internships, it seemed like questions were asked at random with no real goal in mind. Some questions would be more open ended, some looking for a specific answer, and others technical in nature. As I have continued in my career, I have not only had many interviews but I have now been on the other side of the table interviewing people for open positions, and it brings a whole new light into how and why certain questions are asked. Over the past two years or so I have had many opportunities to interview and be interviewed. Through which, I have accumulated notes on how people interview, what types of questions they ask and why. So, here is my experience with it:

Behavioral Questions

The questions that I have always found interesting are the behavioral questions. In asking different people why they ask these types of questions during interviews, most have told me it was to evaluate how I would react in different situations and to determine if I would be a good cultural fit for the team. Here are a ew behavioral questions that I have always loved to be asked and to ask:

If you had to be a fruit/dessert, what would you be and why?
Explain a time when you had to be a leader on a project or at work. How did you take on this role? What did you learn?
Explain a time when you had to educate others on a technical topic? How did you approach this? How did it go?
Discuss your experience with public speaking.
Favorite type of music / instrument / band and why?
You are on a sky lift that has a tight turn radius and 24 seats. You are at the bottom getting on the chair. There is a chair at the top coming down. How many chairs do you pass on the way up to the top?
Explain your volunteer work. What do you do outside of work?

Technical Questions

The next set of questions come from the technical side but range in what is asked and how. From my experience, these are the types of technical interviews I have had and what I learned from each:

Online Code Test

(Time: 1-2 hours, Questions: 3-4)

The online code tests I have taken are the standard tests that ask a handful of questions (typically 3 to 4) in a coding language. Depending on the company, some require a specific language to be used while other don't. Most of these resemble the algorithmic questions you would get in a college coding class. They expect a specific answer to a problem. Some companies will send out the code test to be completed while others will have you complete the code test while being monitored. I have found that the tests in which I am monitored, it is best to take many notes and have an open dialog during the process to show how you're thinking through the problem.

Take-home Code Test

(Time: 1 week, Questions: 1 mini project)

Code tests that I have taken in this format are much more open ended. They provide a problem and desired output, and it is up to the coder to develop a function or set of functions to complete the task. When I take these types of code tests, I read through everything carefully and build up a set of questions / assumptions I have about the problem and verify that what is expected on their end is the same as what I expect. I find it is best to have good communication in these types of code tests, to clear up any misconceptions that could occur while developing the code.

Open Ended Questions

(Time: depends on interview length)

The next type of questions I have seen in an interview is more open ended questions during a 30 minute talk with a team member. These questions have ranged anywhere from basic problems and how you would solve them, to explaining key features in an application. I have found these problems to be the most interesting of the coding questions as it allows for an open dialog with both the interviewer and interviewee. You can bounce ideas off of one another, openly discuss solutions, and explain your thinking as you go. These can also lead into more questions or further discussion.

Discussion on Experience

(Time: depends on interview length)

The last type of technical interview I have been in was very open and more focused on my own experience. When placed in these situations, I have had one or more team members in a room asking about personal experiences, basing questions off the discussion, and diving deeper into the projects or work discussed. I have found that an interviewer may push further into specifics on a project or work experience if they want to test your knowledge or understanding of it. I try to include as much detail as I can on these discussions and allow room for further discussion and ideas to flourish. Sometimes many technical minds in a room can give light to new ideas as you discuss projects you are tackling!

Questions for the Company

After all of their questions have been answered, it is time to begin to ask your own. I find this is a good place to have researched the company and the product(s) that they specialize in, in order to go in with a good set of questions to ask. If it is a multistep interviewing process, I make sure to review all notes and previous questions asked in order to make a new list of questions for each person or interview I walk into.

A few questions I love to ask no matter the place include:

Why do you love to work at you job? What keeps you motivated to stay here?
What do you believe to be a key feature or asset of the product / company that sets you apart on the market?
Ask in-depth questions on the product, how it operates, and what functionality may benefit the company moving forward.

I love to ask the first question of almost everyone I interview with. I began asking this question this past year or so because I wanted to find a place people enjoyed working for and a team that enjoys each other. Making people think about their team and company can tell you a lot about the environment they work in. This has helped me filter out companies that I feel do not have a great culture fit with what I am looking for.

The next question helps me determine what a company is really striving for and where they are headed in the future. I ask this question of different people to get a better understanding of how they view the company versus what is posted online about the company or on the company site. If it is a startup, I also like to ask what began the company? What was the motivation behind building this product in the first place? Sometimes the answers may surprise you.

Lastly, I like to dive deeper into the product itself to understand where the company is now, where they are looking to head, and how the position I am applying for helps reach those goals. If there is not a clear understanding of where the role fits into this development, I take that as a warning. I look for positions that offer growth, learning, and development, not only as an individual but as a company.

What types of questions do you face? What has been your experience with technical interviews? And lastly, what are your favorite questions to ask?

What is your one piece of advice or best tip for getting better in Python programming?

Rose Day — Wed, 24 Apr 2019 14:23:00 +0000

Learning the Differences between Softmax and Sigmoid for Image Classification

Rose Day — Thu, 22 Nov 2018 04:04:36 +0000

Liquid syntax error: Unknown tag '20'

Parsing CSV Rows into Separate Text Files

Rose Day — Tue, 13 Nov 2018 03:42:24 +0000

The #100DaysofCode challenge is an initiative by Alexander Kallaway designed to help people learning how to code to be more consistent. The goal of this challenge was to code for at least an hour everyday for 100 days.

Going off this premise, I decided to work at least an hour every day on just research and coding for image classification. Today marks the end of the first week, in which I researched and worked on the project Analyzing Deforestation and Urbanization Using Intel AI Technologies. During this project, I have been working with Intel Optimized TensorFlow for image classification of satellite imagery. In this first week I have learned many interesting things about image classification and labeling. The one I found most interesting to share would be how to parse labels from a CSV file and transform the labels into the format that I wanted.

For this challenge I faced, let's look at the following example CSV file. This table has two columns, the first column being image name, and the second column consisting of an array of labels. This was the design of the original CSV file that I was working with.

Image Name	Labels
Image_001	haze city farm
Image_002	city
Image_003	water farm

The desired format I needed for labels was to take the image name and create a text file based on that name. For the first row, the file would be called Image_001.jpg.txt and contain all the labels for that image with each label on a separate line. Coding in Python 3, I worked to create a function that would transform the original CSV into the desired text files. As can be seen below, the file Image_001.jpg.txt would be populated with data like so:

haze 
city 
farm

This function created to make this transformation first reads in the data path and filename. Data path refers to the path in which the original CSV is stored and filename refers to the name of the CSV file. This data is stored in a dataframe that can be then traversed to gather data. Each row in the CSV is read as the variable row. Once in row, the first column row[0] is used to create the name of the new text file. Then, the second column, row[1] is split based on spaces. This list of labels, called splitrow, is then iterated over to append each label on a new line of the text file. Once completed, the file is closed and a new file can be created for the next image. This will create one label text file for each image.

def _parse_labels(self):
df_train_tags = pd.read_csv(self.data_path + self.filename)
i = 0
for index, row in df_train_tags.iterrows():
    if i > len(df_train_tags):
        print("Error: Cannot read images, file length too short...")
        break
    else:
        print("Processing: Reading image " + str(row[0]) + "...")
        f = open(self.labels_path + str(row[0])+'.jpg.txt', 'w')
        splitrow = row[1].split(" ")
        for item in splitrow: 
            f.write(item + "\n")
        f.close()
        i+=1
print("Completed: Finished reading labels dataset...")

An interesting note to make here that I didn't realize at first was why the index was required in the for loop. When thinking of a for loop, I refer to for each item in dataset then loop where each item is a row of the dataframe. This logic makes sense but what is happening here is iterrows gives a tuple value rather than just the rows. This means we have to call (index, row) for the tuple in order to access the columns.

After working this past week on research and coding, this was one of the more interesting problems I faced as it involved splitting and transforming one singular column into a text file of many lines. This function also helped give a good overview of basic dataframe and array manipulation in Python.

What interesting problems have you faced this week?

References:

Cover image sourced from Alpha Coders
Intel Optimized TensorFlow
TensorBoard
TensorFlow Guide
Intel DevMesh
Intel AI Academy
pandas.DataFrame.iterrows

Data Cleaning and Mapping Functions - Learning AI with Intel AI Technology

Rose Day — Fri, 02 Nov 2018 21:12:22 +0000

I started on my journey to learn more about data science and AI a little over a year ago as I began my masters and then quit my job. I enrolled full time as a data science student and have enjoyed the decision ever since. When I began my data science classes, most focused on the fundamentals of data storage, management, and analysis. So in my spare time, I have been trying to learn more about AI. One particular thing I find of interest in AI is image recognition and classification. Through this, I have dabbled in different projects to look at live image recognition for movement and person detection. Moving on from this, I have found an interesting in analyzing satellite imagry to gather information from those images. One particular project I have been working on is my proposal for Analyzing Deforestation and Urbanization Using Intel AI Technologies. Through these projects, I have learned new things in data import, setup and pipelines that I thought would be of interest to share.

Cleaning Images

One of the first things I have been learning while working with images for AI was that image import and setup was not as simple as sending an image to a model. When working with Intel Optimized TensorFlow to train a neural network to recognize images, an image dataset is needed with categories. The first thing I read that struck me as odd at first was that all images should be of the same size, for example (28 by 28 pixels). This is common on real-world image datasets that are being used to train neural networks as the images often come in different sizes. Based on this information, they should be batched into a fixed size. This information was well explained in the TensorFlow Guide for Importing Data, as seen below.

# Reads an image from a file, decodes it into a dense tensor, and resizes it
# to a fixed shape.
def _parse_function(filename, label):
  image_string = tf.read_file(filename)
  image_decoded = tf.image.decode_jpeg(image_string)
  image_resized = tf.image.resize_images(image_decoded, [28, 28])
  return image_resized, label

From this, I was able to understand just how images were preprocessed before placing them into the model. As I continued to research just how images were processed, I also found it interesting to reexamine pixel format. The most common pixel format is a byte image which means each pixel in the 28 by 28 pixels image would range from 0 to 255 where 0 is black and 255 is white, and all values in between range in gray. This means that all images processed are analyzed in grayscale instead of color. More on image transformation can be seen at TensorFlow Images documentation.

Data Pipelines and Decoding Images

After understanding the image cleaning, I looked into data pipelines and decoding images. Moving from the code snippet above, it can be seen that the function reads an image from a file, then decodes that image into a dense tensor before resizing it into the determined 28 by 28 pixels.

# A vector of filenames.
filenames = tf.constant(["/var/data/image1.jpg", "/var/data/image2.jpg", ...])

# `labels[i]` is the label for the image in `filenames[i].
labels = tf.constant([0, 37, ...])

This function, as can be seen, takes in a set of images and labels. These images and labels can be saved in tf.constants as below. At first this seemed odd, but through further reading I realized this action of tf.constant(value) was saving both the images and the labels into two constant tensors called filenames and labels.

TensorFlow Mapping Functions

Through all of the data cleaning, decoding, and pipelining it finally came down to the last two steps of the code which create the TensorFlow dataset from the filenames and labels before mapping the final function. I found this line, tf.data.Dataset.from_tensor_slices, to be particularly confusing at first glance. Through research, I was able to discern that this line refers to the making of a TensorFlow dataset from the input constant vectors created above. This allows for the division of image files and labels in the dataset once created.

dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))
dataset = dataset.map(_parse_function)

Once created, the dataset.map function can be called with the parse function created earlier as a parameter. Map allows a transformation across all elements of the dataset and then returns a new dataset containing all transformed items. These items will appear in the same order at the original input. Therefore, when this map function is called, it calls upon the parse function created originally. This parse function was what had done all the image transformation and cleaning for us earlier.

Writing my proposal for Analyzing Deforestation and Urbanization Using Intel AI Technologies many months ago seemed like a small task, but now it has allowed me to push further into AI and image classification. I hope what I have learned thus far has helped you as well!

References

Cover image sourced from Vexel's Blog
Analyzing Deforestation and Urbanization Using Intel AI Technologies DevMesh
Intel Optimized TensorFlow
TensorFlow Importing Data
TensorFlow Basic Classification
Building an Image Data Pipeline
tf.constant
TensorFlow Images
TensorFlow Dataset Map