DEV Community: Kazi Priom

Alternatives - low_cal_alt_update-10

Kazi Priom — Fri, 14 Feb 2025 18:51:48 +0000

I have not imported all the necessary data from the US Food DB into my PostgreSQL server. However, while I was looking into this, I found a DB called OpenFoodFacts. From my quick search, using OpenFoodFacts as my primary DB has certain benefits and drawbacks:

Pros of Using OpenFoodFacts:

The information is more user-friendly. It has pictures, barcodes, and names of the food item. This is absent in the US Food DB, and why I wanted to use Nutrionix at the beginning
The data is easier to handle as it's all in long table

Drawbacks of using OpenFoodFacts:

The data comes from user contributions, allowing for many more mistakes. In fact a lot of the information is different from the US Food DB.
Lots of data is missing

With these issues, I don't think I will use OpenFoodFacts as my primary DB. However, I might use it near the end of the workflow if I want to provide the user pictures, barcodes and names of the products.

Alternatives - low_cal_alt_update-10

Kazi Priom — Fri, 14 Feb 2025 18:51:48 +0000

Pros of Using OpenFoodFacts:

The information is more user-friendly. It has pictures, barcodes, and names of the food item. This is absent in the US Food DB, and why I wanted to use Nutrionix at the beginning
The data is easier to handle as it's all in long table

Drawbacks of using OpenFoodFacts:

The data comes from user contributions, allowing for many more mistakes. In fact a lot of the information is different from the US Food DB.
Lots of data is missing

Progress - lowCal_update 9

Kazi Priom — Mon, 03 Feb 2025 19:54:50 +0000

Hello everyone. During the time between the last post and this one, I have made significant progress. I met with a NutrionIX employee, and I figured out that database licensing costs are above my budget for this project. However, they pointed me back in the direction of the USDA Food Database for the scope of my project. I've had quite a lot of navigating the database, so I was hesitant. But, I've started to figure out how the database works. I'm currently importing the nutrients of the foods into my PostgreSQL server. I'll let you know on the progress.

I was also playing around with Tiny-Bert, an AI model to help with locating low-calorie alternatives to foods. I plan to use semantic similarity to aid this goal. I'll try to figure it out in the next stage of this project.

The Popcorn Problem - lowCal_update 8

Kazi Priom — Tue, 31 Dec 2024 00:10:05 +0000

I'm trying to find a way to use the Nutrionix API to work with my model. I don't have the entire database, which is problematic, but perhaps running a script will give me enough data for the model to learn. The worst case is that I have to get a license for the DB with money.

I will have to calculate the calorie density of the food and use that as the metric. I will try with calories/gram, however, this might be too simplistic in the vast data of food. One simple example of a food with a lot of density with an average calorie/gram ratio is popcorn. This is due to the large amount of air in popcorn. This causes an issue in using the calories/gram metric as air doesn't have weight. Perhaps another metric can be used: calories/serving. However, this causes the issue of the serving size being chosen by the manufacturer. Many foods have very small serving sizes, leading the consumer to think that a food has 200 calories, while in reality it has 400 as it has two serving sizes. But a potentially perfect metric might be calorie/cup. Cup is a volume metric, and is perfect for my use case. Work needs to be done in this area.

Big Discovery - lowCalAlt_update 7

Kazi Priom — Mon, 23 Dec 2024 20:29:17 +0000

I figured out something important yesterday. I was trying to figure out the issue of the missing columns, but I couldn't find the bulk data in the Nutrionix Bulk data. I kept looking around the documentation and I couldn't find anything. Then, something the customer service department of Nutrionix asked me crossed my mind. They asked me, "Where did you get that data from?" At the time, I thought I found it on their website, but I took a closer look yesterday. And I found out it was the US FoodData Central's database. I'm still not sure what this means for me. I would still like to use Nutrionix's DB, so I'll need to figure out how to do that.

lowCalAlt_update 6

Kazi Priom — Sun, 22 Dec 2024 22:04:10 +0000

I'm sorry for being away for some time. I was busy. However, I am back, and I have good and bad news.
Bad News:
Missing data in csv files still needs to be resolved. The company behind the food DB does not email with users who don't pay for their paid plans. Work needs to be done in this area

Good News:
I figured out that you can insert into PostgreSQL much quicker with the "COPY table_name (column 1, column 2, column 3) ..." instead of adding one by one. Was able to insert 2 million items in around 30 seconds, which is much faster than the 5 or so hours needed before.

Updates should be more consistent now.

Adding new columns - lowCalAlt_update5

Kazi Priom — Fri, 29 Nov 2024 06:37:51 +0000

While I still needed information about the missing columns in the CSV files found in the NutritionIX DB, I went ahead and added the columns I had access to. These included brand_name, ss_metric_qty, ss_metric_unit, item_id. However, the insertion process took a lot longer than it should have. I have some theories:

The insertion is being done in a Python for-loop. This might be causing a lot of overhead.
I am adding one row at a time instead of all at once
Indexing in PostgreSQL

Please let me know what you think. I will be doing my own research with another python script.

Missing columns - lowCalAlt_update4

Kazi Priom — Thu, 28 Nov 2024 00:36:14 +0000

I've been working with CSV files from a bulk database download of the Nutrionix DB. However, a lot of important columns are missing. I'm not sure why this is, and I don't know how else to find the vital data. I'm contacting the support service for NutritionIX for information. However, I am working with what I have right now.

PostgreSQL CSV Errors-lowCalAlt_update3

Kazi Priom — Tue, 26 Nov 2024 04:44:03 +0000

I've had a hard time recently working with the CSV files that Nutrionix published for their database. They are quite large, and that makes the files hard to open. However, the main issue was deleting the correct columns and having correct punctuation around items. This caused PostgreSQL to give me various errors. Here are the main ones.

The column names in the CSV were not the same as database
I had an empty column which was represented by a ','
I didn't have quotations around items
single quotations (') needed another single quotation, (').

I initially tried solving these issues using Excel, but Excel has a problem of deleting quotation marks in the file. I discovered that is a common issue for individuals, so I used pandas. Here is the Python script I used for one of the CSV files:


import pandas as pd

#read
file = pd.read_csv('food_insertion3(Done).csv')
#rename columns
file.rename(columns={'fdc_id' :
     'item_id', "description": "item_description", "food_category_id" :
    "food_category"}, inplace= True)
#drop unneeded columns
file.drop(columns='data_type', inplace=True)
#apply quotations
file = file.applymap(lambda x: f'"{x}"' if isinstance(x,str) else x)



file.to_csv('food_insertion3(Done).csv', index= False)
#deal with single quotes
import csv

with open('food_insertion3(Done).csv', 'r', newline='', encoding='utf-8') as infile:
    reader = csv.reader(infile)
    rows = []

    for row in reader:
        cor_row = []
        for field in row:
            cor_field = field.replace("'", "''")
            cor_row.append(cor_field)
        rows.append(cor_row)

with open('food_insertion3(Done).csv', 'w', newline='', encoding='utf-8') as outfile:
    writer = csv.writer(outfile)
    writer.writerows(rows)

PostgreSQL - lowCalAlt_update2

Kazi Priom — Sat, 23 Nov 2024 05:14:50 +0000

I decided to upload the Nutrionix db into PostgreSQL. I could have queried directly using my API key, but there is a limited number of queries I can use. I also find it easier to work it.

The DB will be simple for now. Until optimizations are needed, this should be good enough. Here is a general ER diagram representing it. It's not perfect. I still need to make some decisions.

lowCalAlt_update1

Kazi Priom — Fri, 22 Nov 2024 01:05:08 +0000

So far, my concern has been to look for databases for common and branded foods. I found out a few things:

The US Department of Agriculture has a database of foods called: FoodData Central. Issue: the information is not stored as a relational database. This makes querying very difficult if you want a specific name, nutrient, etc.
Nutritionix has a relational database of food, and this will probably be my solution.

I'm currently trying input all the necessary Nutritionix information into my PostgreSQL server. Information is spread across different CSV files, so its been a little difficult.

Hey, welcome to my blog

Kazi Priom — Thu, 21 Nov 2024 17:31:45 +0000

Hey, welcome. I plan to document my experiences in creating front-end, back-end, and full stack projects. If anyone wants to discuss I am willing to.

The project I am currently working on is an application that finds low-calorie alternatives to food. This stems from my personal experience in weight loss. I've lost around 30 pounds over the last year. While you might think this was the result of following a strict diet, this was not the case. I ate ice cream, burritos, fried chicken, pancakes, and more just by finding low-calorie alternatives (For example, instead of fried chicken, you can use air-fried chicken breast.) So I want to develop an app that can help others in making right food choices.

I plan to use machine learning to build a model that can find low-calorie alternatives to food. I am very new at this, so please let me know if you can help.

Thanks.