DEV Community: Anindita Basu

Dictionaries and learning

Anindita Basu — Sun, 11 Oct 2020 07:51:41 +0000

Python 🐍 people, I need help 😊

If you have the time to spare, help me make Mercury better. Mercury 🌑 is a customisable translation app, meant for English learners.

The issues (all #hacktoberfest eligible) are in this repo 👇🏽

AninditaBasu / mercury

Customisable web app to translate words to English. Uses the Oxford Dictionaries API.

Mercury

Mercury is a customisable web app that uses the Oxford Dictionaries API to translate words to English. You enter a word in your language, and are shown a list of their English counterparts and how to pronounce them, together with examples to show how to use the words in English sentences.

👉 A customised version of the app, which translates Urdu words to English, is live at Tarjuma. 👈

Customisation instructions are in the Mercury documentation, which also contains a demo.

Disclaimer: This app is neither developed nor endorsed by Oxford Dictionaries. The code is available as-is, on an MIT license.

How to use Mercury

The app runs on Python, but you don't need to know Python to use the app or to customise it.

Prerequisites

You need a paid API key from Oxford Dictionaries API. The free plan is adequate for testing purposes but only…

View on GitHub

(sits back and waits for #community magic to float in 🧚🏽‍♀️🧚🏽‍♂️✨)

Designing a customisable translation app

Anindita Basu — Wed, 13 May 2020 15:54:38 +0000

There already are so many translation apps out there; why another one?

Well, because the Oxford Dictionaries API is there 🤷‍♀️

And, therefore, Mercury was born, targetted towards English-language learners. Mercury translates words to English, shows example usage, and includes a click-to-play pronunciation file.

Design considerations
Coding
Call for action

Design considerations

Do not overwhelm a language learner with info. Include barest minimum info, and give links for more (target audience: those who've just begun to learn English).
Make a customisable app, so people can specify which language they want and what text they see on the UI (target audience: people for whom the English language is very very foreign and who, therefore, are more comfortable with a UI in their own language).
Make the customisation so easy that people who don't know how to code can also download and run the app (target audience: schools and institutes that do not have coders on their rolls but are teaching English as a foreign language).
Make the app code lightweight (reason: internet connectivity is poor in too many places in the world).

Coding

In Python 🙂 because that's the only language I know.

AninditaBasu / mercury

Customisable web app to translate words to English. Uses the Oxford Dictionaries API.

Mercury

👉 A customised version of the app, which translates Urdu words to English, is live at Tarjuma. 👈

Customisation instructions are in the Mercury documentation, which also contains a demo.

Disclaimer: This app is neither developed nor endorsed by Oxford Dictionaries. The code is available as-is, on an MIT license.

How to use Mercury

The app runs on Python, but you don't need to know Python to use the app or to customise it.

Prerequisites

You need a paid API key from Oxford Dictionaries API. The free plan is adequate for testing purposes but only…

View on GitHub

Call for action

Please help me make this app better? Your feedback, suggestions, and comments are most welcome, as are PRs. Let's together 🌈 bridge languages.

Pushing out an MVP

Anindita Basu — Fri, 24 Apr 2020 07:32:53 +0000

I want to build a chatbot that answers questions about the Mahabharat.

When I started out, my intention was to build the chatbot (and, subsequently, an API) on a MySQL database with Python. Things didn't work out as planned, however 😄. For starters, my RaspberryPi refused to play nice with MySQL through Python. Yes, I googled. No, nothing worked (yet).

Meanwhile, I was getting frustrated. Here I was, with tons of data to share with the world, and some stupid technical non-knowledge was standing in my way.

So, I decided to look for a simpler tech stack and push out an MVP. Doing so will help me in two ways:

Gather feedback on the usefulness of my offering
Find simpler solutions to technical challenges (did I mention I am not a coder?)

So, here's my MVP: https://mb-verse.herokuapp.com. Its tech-stack uses:

json for storing data
python for processing data
heroku for hosting (The code is in a private GitHub repo at the moment because I am not comfortable yet sharing the json files.)

Please let me know your thoughts regarding the MVP. I am looking for feedback on usability and usefulness. I started a public GitHub repo to gather issues: https://github.com/AninditaBasu/calm-wildwood

(And, if you don't know about the Mahabharat, you might want to read this summary: https://www.ancient.eu/Mahabharata/. But even if you don't know the story [and especially if you dont know the story], your feedback is very welcome because you represent a good user persona: someone who wants to know about a story - any story - that's much loved by some other group. For example, if I haven't seen any Star Wars and there was this chatbot...)

Let me know what you think 🙂

How to build a basic website on GitHub with Liquid

Anindita Basu — Thu, 05 Mar 2020 13:29:26 +0000

Why this post?

Because I could not find any resource that tells me how to build a basic website with Liquid tags on GitHub. Through the GitHub web interface, not through any CLI, whether Jekyll or Git. Without needing to install Jekyll (which isn't officially supported for my Windows machine anyway).

Also because the official Jekyll documentation (and the official Jekyll tutorial) assumes a level of knowledge that I do not have. Also because all the so-called starter packs I found on GitHub (and through Google) have way too many files to be useful to me as a beginner.

So, this post.

This post is for people like me who get frustrated when they encounter something like this:

This post assumes you know the basics of HTML and CSS.

GitHub is a source-code repository; it also gives you a webpage for each of your repositories. The intention of this post is to show you how to make use of GitHub's hosting facility to build your website.

GitHub uses Jekyll to render the pages on your website.

This means, GitHub's publishing engine expects the same directory structure that Jekyll's publishing engine expects. Which is something like this, if you start off with the barest minimum files:


- _layouts        # folder to contain HTML templates
  -- default.html
- css             # folder to contain stylesheets
  -- main.css
- topics          # folder to contain pages of the website
  -- page1.html
  -- page2.html
  -- page3.html
- _config.yml    # basic configuration info
- index.html     # landing page of the website

Jekyll uses the Liquid template engine to create webpages. This means, you can use Liquid variables in your HTML pages, and during build time, the value of these variables will be inserted onto the page. So, for example, if you have something like this:

<p>I graduated in {{ grad_year }} and have since worked with {{ company_1 }} ({{ comp1_start_year }} - {{ comp1_end_year }}), {{ company_2 }} ({{ comp2_start_year }} - {{ comp2_end_year }}), and {{ company_3 }} ({{ comp3_start_year }} - {{ comp3_end_year }}).</p>

it'll be rendered on the page like this:

I graduated in 2020 and have since worked with WB (2020 - 2025), YOLO (2025 - 2074), and FIK (2134 - 2135).</p>

You can immediately see the possibilities that Liquid gives you, right? You can almost build dynamic webpages with a static site generator. Almost. Without needing to install or configure anything.

So go ahead, create that GitHub repo and build a website. Once you're comfortable with this basic setup, you can explore the full power of the objects, tags, and filters of Liquid.

If you need a template to take off running, I've created one for the files you'd need: jekyll-github-starter-pack. It also has a README to help you get started.

Was this useful?

How to move your Flask app from the local machine to the Heroku cloud

Anindita Basu — Thu, 21 Nov 2019 11:40:55 +0000

Why this post?

Because I could not find any resource that tells me how to deploy a Flask app to Heroku through the Heroku web interface. Through the web interface, not through the CLI 😒

Also because the official Heroku starter pack for Flask has way too many files; it intimidated me no end. Are so many files really needed, I asked myself 😓 And the Heroku documentation for that starter pack (and the documentation otherwise) assumes a level of knowledge that I do not have.

Also because the Flask app that ran perfectly on my laptop would not work on Heroku and none of the resources Google threw at me could tell me how to solve it. I wanted something beyond the hello world apps, something that took in a user input and returned a result. I went on to the 5th and 6th page of the search results, looking... 😞 Why isn't this part clearly documented anywhere, I asked myself.

So, this post.

This post is for people like me, people who are just learning to code, are not professional coders, and will not use the CLI until you make them an offer they can't refuse.

This post is for people who want minimal stuff on their laptops, and prefer to do things on the cloud through graphical user interfaces. UIs for the win 🎉

This post assumes you know the basics of Python (and Flask). That you already know how to get a Hello world app in Flask running on your local machine.

TL;DR

(i) In your Flask code, replace app.run() with host and port instructions.
(ii) In your repo, have a pip file to pip install the required packages at the Heroku end.
(iii) In your repo, also have Procfile and requirements.txt to specify a web server for the app.

What does the app do?

It takes a user input, processes it, and returns the results of that process. To do so, it uses the Flask framework of Python.

To test a Flask app locally, you use the app.run() command. By default, Flask starts a web server on the local machine and uses port 5000 to deploy the app.

To run the app on Heroku, three additional steps are needed:

Heroku does not have a web server for Flask, so you need to tell it how to start a web server. This app uses gunicorn as HTTP server. So, for the app to run on Heroku, it needs a Procfile that contains the following command: web: gunicorn flaskStarter:app where web tells Heroku this is a web app, gunicorn tells it to use that server for app deployment, flaskStarter is the name of the Python program that contains the app code, and app is the name of the app routine inside the code.
Heroku needs to install gunicorn before it compiles the app, and you tell it to do so through a file called Pipfile.
Then, you need to modify your Python file to include the host and port info for app deployment. You do so by replacing app.run() with the following lines of code:

        port = int(os.environ.get("PORT", 5000))
        app.run(host='0.0.0.0', port=port)

Both Procfile and Pipfile must be named exactly so, and not have any file extension.

That's it. Now the app, which ran so beautifully on the local machine, will also run on Heroku.

🚶 Armed with this knowledge, I'll now fine-tune an app that I am building around an API and deploy it to Heroku for the world to use ...

If you'd like to get started with Heroku, I created a GitHub repo for the files you'd need: flask-heroku-starter-pack. And, here's the Heroku app.

Designing the database

Anindita Basu — Wed, 14 Aug 2019 08:57:35 +0000

My chatbot is going to answer queries on the Mahabharat, so my database will be a Mahabharat¹ database. Let's design one 🚧

I should be able to apply the design principles in this post to any story corpus, epic, or novel (and not just the Mahabharat). The Bible, for example, or the Harry Potter series. 🌟⚡️

Who am I designing the database for?

A chabot.

Who will use the chatbot?

Human beings.

Why will they use the chatbot?

To get quick info on the story.

What kind of info?

Info about the people in the epic, what they do, how they live day to day, how they die.

I know that my chatbot is not a story-telling bot; it is an info-supplying bot. How do human beings look for info? They do so by asking questions.

Why is the sky blue?

How do I dance the foxtrot?

When is that moron going to resign?

Where were they when the lights went out?

Who framed Peter Rabbit?

Applying these questions to my project, I can see two things:

That I need to group the information into categories
That a category should answer one - and only one - type of question
That an answer to a question in one category can lead to further questions that can belong to either the same category or to another

I believe, in geek parlance these are known as taxonomy, ontology, and normalisation?

So, categories will be my database tables. The rows in the tables will be the Who, What, Where of that category. The columns will contain the information about one specific Who, What, Where. The info in one cell of one table can be exactly equal to the info in a cell of one or more tables. 💥

My Mahabharat database is going to have the following categories (tables):

persons
families
clans
weapons
killings
places

A character in the story (a person) can belong to persons, families, clans, weapons, and killings. In one table, a person can appear in one - and only one - row. In all other tables, that person can appear at several rows, but only once in each such row.

I believe, in geek parlance this is known as primary key and foreign key?

Now that I have an idea of what my database will look like, I should write it all down. So, here we go:

To sum up, for designing my database, I:

Ask what kind of questions my database will answer.
Group the questions into categories.
Create a table for each category.
Turn the sub-groups of these categories into the table columns.
Link one table to another through a common column.

Next, I'll create these tables in MySQL on my RaspberryPi and populate them with the data.

Footnote:
1. The Mahabharat is one of the two epics of ancient India, is the longest epic in the world, and is the epic that contains the Bhagvad Gita (a mini-book that's sacred to the Hindus).

Preparing my Raspberry Pi

Anindita Basu — Mon, 12 Aug 2019 09:45:11 +0000

What is a RaMP stack, you might ask? It's my parody of the LAMP stack, where RaMP = RaspberryPi, MySQL, Python.

Why a RaMP stack? Because I am looking for a low-cost, low-space, low-weight alternative for my personal laptop, and because I want to keep the bot experiments completely separate from the stuff I have on either my work laptop or my personal laptop, and because, at some point of time, I dream of building my own Alexa-like voice bot...Also because I know basic Python and can pick up SQL.

Here's a picture to show you how tiny a barebones RaspberryPi can be.

These are my tentative steps:

Set up my RaspberryPi.
Design and create the databases.
Design and create the bot app.

I am neither a geek nor a coder; this journey will not be easy. This post is the first of a series where I share my learning, failures, and successes with you all. In these posts, I plan to focus more on the design considerations (the WHYs) of the chatbot rather than the code aspects (the HOWs).

This post is about making a RaspberryPi ready for my experiments.

A RaspberryPi is just a CPU (a motherboard). To work with it, one needs a power source (a power cable), an input device (keyboard), and an output device (display monitor? TV?). One also needs to load the operating system on to the CPU, so one needs an external storage device (a microSD card) that contains the OS.

I bought a starter kit, followed a YouTube video to guide me on how to put the RaspberryPi into its official case, and then looked around to plug stuff into the ports and get started.

Rookie mistake.

The starter kit did not have a VGA to HDMI adapter (and my monitors don't have HDMI ports, only VGA).

Starter kits such as the one I bought are well-meaning but can contain stuff I don't need (and not contain stuff I need). Googling tells me it's cheaper to go for combo-boxes or starter kits than buying individual components, but I disagree. The very minimum things that I needed were these: the motherboard, a power source, and a microSD card loaded with the Raspbian OS. Had I bought just these, and a VGA-to-HDMI connector, I would've saved about eight hundred rupees.

So, here's a buying guide for you (I'm assuming you already have a monitor, a USB keyboard, and a USB mouse).

Buy these 2 items: RaspberryPi 3B+ motherboard and a 2.4Amp power adapter with microUSB plug. I bought model 3 B+; there's a model 4 out too if you want that.

If you already have a spare microSD card that you can dedicate to your RaspberryPi, go to the next step. If not, buy one, preferably one pre-loaded with the Raspbian OS. Or, you can load the card with the Raspbian OS from raspberrypi.org.

See what port your TV or monitor has, and buy either a VGA-to-HDMI connector or an HDMI-to-HDMI connector (RaspberryPi has an HDMI port).

I bought the correct display connector and plugged in the keyboard, mouse, and display. Then, I looked around for a port to plug in the microSD card.

Another rookie mistake.

The microSD port is on the reverse side of the motherboard; to get to it, one needs to turn the motherboard over.
Which means, I had to unplug all of what I had earlier plugged in (because, I was scared of snagging one of those pins with a wire or something).

So, here's the correct order of plugging the components in:

Turn the motherboard over and plug in the microSD card very slowly into the slot.

Plug the keyboard and the mouse into USB ports.

Connect the display to the HDMI port.

If you're going to use wired internet, plug the LAN cable into the ethernet port (I was going to use WiFi).

Plug the power adapter into the microUSB port.

Switch the display on. Then, switch the power adapter on (googling told me the power adapter should be plugged in last and switched on last).

And voila! My tiny nano micro computer blinked red and started up. On the screen, I selected the Raspbian OS, clicked Install, followed the onscreen instructions, and then waited for the installation to be over (it took almost 15 minutes).

Googling also told me to update the packages after they're installed, so after the installation was over, I ran the following commands:
sudo apt-get update, which took about 10 minutes to complete.
sudo apt-get upgrade, which took about 15 minutes to complete.

Looks like I'm done setting up the computer. Next post, I'll talk about the chatbot back-end (which will be a MySQL database).

Meanwhile, since this is supposed to be my ultra-portable computer, I'll buy it an LCD touchscreen display.

Also, since the RaspberryPi official case cuts off all access to the GPIO connector pins, I'll buy it a case that keeps the pins accessible. The pins are needed to plug the LCD display in, and also for all of the IoT stuff (assuming I go that way later).

And, since I don't like the idea of having to turn the motherboard over every time I need to use a microSD card, I'll also buy it a USB microSD card reader.

How a non-coder created an API (and some questions)

Anindita Basu — Wed, 15 Aug 2018 08:35:37 +0000

APIs fascinate me. That you can ping an unknown system and get it to share its treasure-trove with you, for you to play around with, is something that I find utterly fascinating. The first APIs that I ever used were the Twitter APIs and the IBM Watson ones; I totally loved (love?) them.

The other thing that I am fascinated by is Indology. However, APIs and Indology do not meet. While there are plenty of digital resources, freely available, on ancient India, none of these resources make available the data (treasure?) held in them in a manner that can be consumed by apps and programs.

I had the books, I had the data. But I had no idea about how to make APIs (I'd only ever used them to make bots and apps). So, I hunted around for an easy solution and found sheetlabs.com. Thus was born the vedic APIs.

That's the non-coder way to go about making APIs, I guess. In this post, I'd like to ask you coders a couple of questions (and also to shout out about my APIs because I am so thrilled about them :) ).

Question 1

Are nested arrays and lists a good idea, or no?
My APIs are designed to return JSON data like this, for example:

[
    {
        "mandal": 1,
        "sukta": 2,
        "sungby": "poet_name_1",
        "sungfor": "god_name_1",
        "meter": "meter_name_1"
    },
    {
        "mandal": 1,
        "sukta": 2,
        "sungby": "poet_name_2",
        "sungfor": "god_name_1",
        "meter": "meter_name_1"
    },
    {
        "mandal": 1,
        "sukta": 2,
        "sungby": "poet_name_2",
        "sungfor": "god_name_2",
        "meter": "meter_name_1"
    },
    {
        "mandal": 1,
        "sukta": 2,
        "sungby": "poet_name_2",
        "sungfor": "god_name_3",
        "meter": "meter_name_2"
    }
]

Is it better if they're like this? And, why so?

[
    {
        "mandal": 1,
        "sukta": 2,
        "sungby": ["poet_name_1", "poet_name2"],
        "sungfor": ["god_name_1", "god_name2", "god_name_3"],
        "meter": ["meter_name_1", "meter_name_2", "meter_name_3", "meter_name_4"]
    }
]

Question 2

What could be a low-cost setup to host my own database server that'll have the data tables, and then making them available through web calls? One option that I had thought about was getting a Raspberry Pi to host a PostgreSQL server, and throwing the database open for API calls. Advisable? Other options?

Question 3

What are the steps to convert a database table into an API? Dumb question?

I have tons of data on the Mahabharat that I want to turn into APIs, but the spreadsheets will go beyond the free limit of sheetlabs.com. Plus, some of the columns in one sheet could be linked to columns in another sheet. And I am looking for a more permanent solution anyway, so...thoughts?

How bots tweet Urdu word meanings in English

Anindita Basu — Thu, 01 Jun 2017 06:14:53 +0000

In this post, I'll talk about how a bot tweets translations of Urdu words to English.

TL; DR:
1. Use the JSON data returned freely by the well-documented Oxford Dictionaries API.
2. Tweet like @_alfaaz_¹

Why this bot
How the bot works
- How the Oxford Dictionaries API works
- The difficulties in using the API
- The workarounds
- How do the tweets happen
Where's the code
Why this bot, again

Why this bot

I know Urdu. At least, I think I know Urdu, chiefly because I know the lyrics of practically every popular Hindi film song ever made. But film songs are ... poems, and there's this thing called poetic license, and then, not all words always mean what I might think they mean.

And I am not alone. Dozens of people around me, growing up following the major religion in India (which is 'Films') know and quote lyrics off the cuff, but don't always know the exact meaning of certain words. For example, I (and hundreds of others) went through teenage and the rest of our lives thinking jigar means heart. It doesn't. Nope.

So, when I chanced upon the Oxford Dictionaries API, which includes an Urdu dictionaries endpoint, I thought, "Why not make a tweet bot?"

How the bot works

It picks a Urdu words one by one from a list I give it, supplies those words to the Oxford Dictionary API endpoint, gets the JSON responses, parses the responses to pick the translations, and tweets the Urdu words and their English translations.

Its requirements are as follows:

A computer connected to the internet
A token, token secret, consumer key, and consumer secret from Twitter
An app ID and app key from Oxford Dictionaries
A scripting environment to write and run the bot code in

How the Oxford Dictionaries API works

It serves a JSON response through REST endpoints that are well-documented².

The actual translations of a word are contained in a list of dictionary objects; the list is called results.

Each dictionary object in this results list has a key called lexicalEntries. If you drill down from there, you finally arrive at a list called translations where a dictionary key called text contains the actual translation. The following image shows the complete sequence, with the translation highlighted in a red box :

A word can have several meanings and, therefore, several translations. How would one know how many meanings exist? Well, one looks at the number of items in the senses list for every dictionary object in the results list. Here's the hierarchy of senses:
['results'] > ['lexicalEntries'] > ['entries'] > ['senses']

The difficulties in using this API

The script. The difficulty is the script.

Urdu, as spoken in its daily form in India, is indistinguishable from Hindi. Urdu, when written, is written in a script called Nastaliq. I don't know how to read Nastaliq³. The Oxford Dictionaries API takes input only in the Nastaliq script. Uh oh!

The workarounds

Learn how to read Nastaliq.[But] Yeah, right, on top of everything else. It's because I don't know that script that I've been unable to look up a dictionary thus far but, surely, in this day and age of OneWorld, there is an easier way? After all, I can make out the letters with huge effort, surely that's sufficient?
Use a transliteration service.[But] I searched, but found nothing. The GoogleTransliterate API has been deprecated. Python (the language I can code in) has a polyglot library but it refused to pip install on my machine. I did not find a single resource that could take in a word written in Devanagari and give out that word in Nastaliq programmatically.
Extract the Urdu word from one of the online dictionary resources that contain both the Urdu word and its transliterated English form, like the Platt's dictionary⁴ does: Ø¬Ú¯Ø± jigar.[But] None of these web resources are in any structured format such as XML or JSON. Not one.
Build my own transliterator.[But] Heh, like I can even write two lines of code without looking up a textbook... But wait. I found a Nastaliq-to-Roman mapping table on Medium⁵. Maybe I could do something similar for tweeting, I thought, and parked away this idea for later. But this doesn't help me get spelling-perfect transliterations that a language dictionary can recognise.
Use a readymade word-list.[But]Oxford Dictionaries supplies ready-made wordlists for English. But not for Urdu.

Because none of these workarounds were feasible, I had to generate a word-list manually. I used Platt's dictionary to make my word-list. I would type the Urdu word in Roman script, and copy the Nastalikh version of it in my word-list.

How do the tweets happen

The bot uses tokens to authenticate itself with Twitter and with Oxford Dictionaries.
It picks a word from the word list and gets the translations from Oxford Dictionaries.
It puts the translations on to a tweetable image, together with all of the letters in that word broken up and mapped to their Devanagari counterparts (see #4 under Workarounds). It then tweets this image.
[Enhancement, added later] It picks up the Devanagari version of the word (present in the same word-list), gets the Hindi meanings from Oxford Dictionaries, and tweets these meanings.
It also feeds the word to Platt's dictionary, gets the URL of the search result, and tweets that URL.
It then sleeps for an hour, and repeats steps 2 through 5 for as many times as there are words in the word-list.

Where's the code

Why, on GitHub of course. Where else?

Why this bot again?

Because other resources - bot tweets, emailed WOTDs, non-bot tweets - are either boring or overwhelming. The design goals for this bot were:

do not overwhelm the noob
tie a word to a fun context that every noob follower will likely know.

So, that was the story about my first tweet bot. What do you think?

Note 1.

alfaaz is an Urdu word that means 'words'. (Urdu has a different word for the singular word 'word'; it is lafz.)

Note 2.

Oxford Dictionaries API documentation

Note 3.

In its spoken form, Urdu is practically indistinguishable from Hindi. However, for writing, the two languages use different scripts. Hindi uses Devanagari; Urdu uses Nastaliq. The sentences in each language would be exactly the same, but written in two totally different writing styles, whose major differences are these:

Nastaliq is written right-to-left; Devanagari, left-to-right.
Nastaliq often does not use vowel markers; Devanagari is invariably phonetic and always with vowel markers.

Note 4.

John Platt's Dictionary of Urdu, Classical Hindi, and English

Note 5.

Transliterate Urdu to Roman Urdu In Python