DEV Community

Cover image for Text Analyzer | Python | Day 1 | Week 1| 10WeekChallenge
Nattar Kani Murugan
Nattar Kani Murugan

Posted on

Text Analyzer | Python | Day 1 | Week 1| 10WeekChallenge

#ai #python #beginners #learning

Okay, Nattar.

You said you wanted to get better at Python.

Not just watch another tutorial. Not just save another course. Not just tell yourself, “I'll start tomorrow.”

So... let's actually build something.

This is Week 1 of my 10 Week AI Challenge, and I decided to start with something simple: a Text Analyzer.

Nothing fancy. No Python libary. No AI model. No API. No LangChain. No vector database.

Just Python. And honestly, that is exactly what I needed.

So, what does this thing actually do?

I wanted to take some text and make Python answer a few basic questions about it.

Things like:

  • How many sentences are there?
  • How many words?
  • Which words appear the most?
  • What are the most common word combinations?
  • How readable is the text?

So I broke the problem into smaller pieces.

The final analyzer has a few steps:

  1. Take text as input
  2. Validate the input
  3. Count sentences
  4. Normalize and tokenize the text
  5. Calculate word frequency
  6. Find the top N words
  7. Find common n-grams
  8. Calculate a reading score

Sounds simple. And that's the point.

Step 1: Okay, give me some text

Python needs something to work with. So I started with basic input handling.

text = input('Enter your text: ')
Enter fullscreen mode Exit fullscreen mode

Step 2: But what if I give it... nothing?

"What happens if I just press Enter?"

So I added input validation.

if not text.strip():
    print("No input provided")
    exit()
Enter fullscreen mode Exit fullscreen mode

This is a small thing, but it made me think differently about writing programs. It's not enough to make something work when everything goes perfectly.

I also need to think about: What if the user does something unexpected?
That's a useful habit to build early.

Step 3: How many sentences did I write?

For a simple version, I looked at sentence-ending punctuation.

sentence_count = text.count(".") + text.count("?") + text.count("!")
if(sentence_count) == 0:
    sentence_count = 1
Enter fullscreen mode Exit fullscreen mode

Perfect? I don't need to build the world's most sophisticated NLP system for Week 1. I need to understand the problem, build something that works, and improve it as I learn.

Step 4: Time to clean things up

Shawarma
shawarma
shaWARma!
shawarmA?

Python sees these as different strings. But I don't want that. So I started *normalizing * the text. We're going to lowercase everything and remove the unnecessary punctuation. And also I need individual words. That's where *tokenization * comes in. For this simple project, splitting the text was enough.

clean_text = text


punctuation = ".,:;?!@#$%^&*()_+-="
for i in punctuation:
    clean_text = clean_text.replace(i,"")

words_list = clean_text.lower().split()
Enter fullscreen mode Exit fullscreen mode

Step 5: Which words do I keep repeating?

I decided to use a dictionary. Then I loop through every word. So, just a dictionary, loop and a condition.

for word in words_list:
    if word in freq:
        freq[word] = freq[word] + 1
    else:
        freq[word] = 1
Enter fullscreen mode Exit fullscreen mode

Now I have the counts. But they're not necessarily in the order I want. So I needed to sort them based on their frequency.

sorted_freq = sorted(freq.items(), key=lambda item: item[1], reverse=True)
Enter fullscreen mode Exit fullscreen mode

Step 6: Give me the top N words

I didn't want to hard-code the number of words. So I added another input and formatted the output.

top_n = int(input("How many top words do you need? "))


print(f"Top {top_n} frequently repeated words:")

for word, count in sorted_freq[:top_n]:
    print(f"{word} -> {count}")
Enter fullscreen mode Exit fullscreen mode

I have already got the top n value and I am going to have the for loop iterate in the sorted list of words. The end of the range parameter is the input received. So, if I got top 3, then the range goes from 0 to 2 indexes and shows the top 3 words. This is where sorting becomes handy!

Step 7: Give me the top N-grams

What I mean by this is..

These combinations can tell us something different from individual word frequency. That is n-grams, my Lady.

n_size = int(input("How many n-grams do you need? "))

n_gram_freq = {}


for i in range(0,len(words_list)-n_size+1):
    n_gram = words_list[i:i+n_size]
    gram_text = " ".join(n_gram)

    if gram_text in n_gram_freq:
        n_gram_freq[gram_text] += 1
    else:
        n_gram_freq[gram_text] = 1

sorted_ngram = sorted(n_gram_freq.items(), key=lambda item: item[1], reverse=True)
Enter fullscreen mode Exit fullscreen mode

Let me quickly, break this down, so I don't get confused when I look at the program after n days.

How did I come up with this for i in range(0,len(words_list)-n_size+1): and importantly why?

So, if the input is: "I want shawarma with garlic sauce" and n-grams is 2 (that is bigrams). Then I need something like this:

  1. i want
  2. want shawarma
  3. shawarma with
  4. with garlic
  5. garlic sauce

So the formula is: Total words - n-grams + 1

6 - 2 + 1

Then,

n_gram = words_list[i:i+n_size]
gram_text = " ".join(n_gram)

which is going to take my words out from the list (i.e.) I have ['i','want'] but i need 'i want'

Now, I need a count of these grams and going to make use of n_gram_freq dict to get that.

if gram_text in n_gram_freq:
n_gram_freq[gram_text] += 1
else:
n_gram_freq[gram_text] = 1

Of course, if I'm counting them, I want to know which ones appear most frequently.

sorted_ngram = sorted(n_gram_freq.items(), key=lambda item: item[1], reverse=True)

print(f"Top {top_n} frequently repeated words:")
print(f"{n_size}-grams: ")

for word, count in sorted_ngram[:top_n]:
    print(f"{word} -> {count}")
Enter fullscreen mode Exit fullscreen mode

Step 8: Counting syllables

Remember the rule? No external libraries. So I had to come up with a simple approach myself. I started with the vowels:

vowels = "aeiou"
syll = 0

Then I went through each word and counted vowel groups. The basic idea is, if I see a vowel and the previous character wasn't a vowel, I count a syllable.

But, word like 'beautiful' has vowel groups like:
eau
i
u

The logic isn't a perfect English syllable detector. And I know that.

But for this project, I wanted to understand the logic instead of reaching immediately for a package.

vowels = "aeiou"
syll = 0

for word in words_list:
    prev_vowel = False
    word_syll = 0

    for letter in word:
        if letter in vowels and prev_vowel == False:
            word_syll += 1

        prev_vowel = letter in vowels

    if word_syll==0:
        word_syll = 1   

    syll += word_syll
Enter fullscreen mode Exit fullscreen mode

Step 8: Readability score

I wanted to calculate a readability score. I used the Flesch Reading Ease formula.

The formula uses:

  • Number of words
  • Number of sentences
  • Number of syllables

The formula is:

206.835 - 1.015 × (words / sentences) - 84.6 × (syllables / words)

score = 206.835 - (1.015*(word_count/sentence_count)) - (84.6*(syll/word_count))
print(f"Readability score: {score:.2f}")
Enter fullscreen mode Exit fullscreen mode

A simple Python script has gone from:

"Give me some text."

to:

"Here's what I can tell you about that text."

The project itself is small. But I ended up touching quite a few Python concepts:

  • User input
  • Input validation
  • Strings
  • String methods
  • Lists
  • Dictionaries
  • Loops
  • Conditions
  • Slicing
  • sorted()
  • lambda
  • List processing
  • Word frequency
  • N-grams
  • Basic text processing
  • Readability calculations

Is this code perfect? No. And that's okay.
I don't want to spend Week 1 trying to build a production-ready NLP library.

I'm starting with Python fundamentals because I know I need a stronger foundation before I start building more serious AI and GenAI projects.

And I want to remind myself of something:

I don't need to know everything before I start. I just need to start.

So, Nattar... Come back tomorrow again!

You can find the complete project on Github:
👉 Week 1 - Day 1 - Text Analyzer in Python

Top comments (0)