DEV Community: Tanmay Chakrabarty

Making sense of the news with GPT-3, NLTK, Python and Streamlit

Tanmay Chakrabarty — Tue, 18 Jul 2023 14:10:20 +0000

ChatGPT buzz is all around and I was keen to use the GPT-3 API in a project. In this post I will walk through the process of creating a Python web app using Streamlit, called "News 📰 Sense." The app fetches top news articles from the News API, generate summaries of each article using the OpenAI GPT-3 model, analyze the sentiment of the article using the Natural Language Toolkit (NLTK) library, and classifies the articles into different categories based on keywords.

Understanding Sentiment Analysis

Sentiment analysis is a natural language processing (NLP) technique used to determine the sentiment or emotion expressed in a piece of text. It aims to understand whether the text conveys a positive, negative, or neutral sentiment. For my project I used the VADER (Valence Aware Dictionary and sEntiment Reasoner) lexicon from the NLTK library.

The VADER lexicon contains a list of words and their associated sentiment scores. It takes into account the intensity of the sentiment words and the context in which they appear to provide a compound sentiment score. A positive (pos) score indicates a positive sentiment, a negative (neg) score indicates a negative sentiment, and a score close to zero suggests a neutral (neu) sentiment.

Categorizing News Articles

Categorizing news articles helps in organizing and presenting information effectively. In my app, I classify each article into predefined categories, such as Business, Technology, Sports, Entertainment, Politics, and Society. To achieve this, I created a list of keywords associated with each category. In the code I checked if any of these keywords are present in the article content, and if so, assigned the article to the corresponding category.

Summarizing News Articles with GPT-3

Summarization is the process of condensing a longer piece of text into a shorter, concise version that captures the essential information. For summarizing news articles, I used the OpenAI GPT-3 DaVinci, a powerful language generation model that can generate human-like text. Shameless admission: Using GPT was the main draw for this project :)

GPT-3 works by taking a prompt (the news article content in our case) and generating a continuation or summary based on that prompt. We specify the maximum number of tokens for the summary to control its length. The GPT-3 model is fine-tuned to understand context and generate coherent and contextually relevant summaries.

The application

You can find the code for News 📰 Sense and instructions to use it at https://github.com/tanmaychk/News-Sense. Before starting, make sure you have the following:

Python installed on your system
An API key from the News API. You can sign up and get your API key from here.
An API key from OpenAI GPT-3. You can obtain your API key from OpenAI.

What have been the Learnings

As a developer, this project allowed me to apply various NLP concepts and technologies in a practical and meaningful way:

API Integration: how to integrate external APIs like the News API and OpenAI GPT-3 API into Python code to access real-time news data and perform advanced language generation.
Natural Language Processing (NLP): By utilizing the NLTK library, gained insights into how NLP techniques like sentiment analysis and tokenization work. I also understood how the VADER lexicon is used to determine the sentiment of text.
Machine Learning for Language Generation: experimented with OpenAI GPT-3, a state-of-the-art language model capable of generating human-like text. I utilized it to summarize news articles, demonstrating the power of machine learning in language tasks.
Text Classification: applied text classification to categorize news articles based on specific keywords. This demonstrated how NLP techniques can be used for content organization and user-friendly presentation.
Streamlit Web App Development: I had used Streamlit earlier as well, in another project. Streamlit's simplicity allows us to focus on functionality and visualization rather than dealing with web development complexities.

Further Steps

The News Sense app can perhaps be further enhanced by incorporating user input for customized news filtering, improving summarization techniques, or integrating more advanced language models. I will implement them, when I get time.

It was a joy working on the project, hope you liked it as well.

Tanmay ChakrabartyFollow

A graduate student in Computer Science passionate about applying technical skills to solve complex problems. Areas of interest: Data analysis, ML, Fintech. Open for internship roles.

How I created my Portfolio website using Hugo and GitHub pages?

Tanmay Chakrabarty — Sun, 19 Jun 2022 08:06:17 +0000

Honestly, you might be coming across umpteenth article on this topic, but I had to jot down my experiments with static website setup. For the uninitiated, apart from being the name of a famous 2011 Martin Scorsese movie, Hugo is also an extremely popular static website generator. Hugo uses data files, configuration, layout templates, static files, and content written in Markdown to render a static website, which means no database, blazing fast site speed and no danger of cross-site scripting and SQL injection attacks.

Installation & Prerequisites

To start with I installed Hugo on my Windows 10 laptop (instructions here), I already had Git Bash installed, but it’s easy to install it from here. You may also choose to use a Git GUI tool like Tortoise Git.

Next, I created a new Git repository at my GitHub account with the name [my-github-username].github.io containing just a blank README.md file (note that its necessary to name the repo this way, in order to leverage the GitHub pages feature).

While GitHub will create a “main” code branch by default, at this time I also created an empty new branch and named it “gh-pages” (following sections will talk about the significance of creating this other code branch).

Next, I need to tell GitHub that my repository will use GitHub Pages. To do that, click the "Gear Icon > Settings" for the repo, and then click "Pages" on the left menu. Then, under "Source" on the right pane, select the "Branch" as "gh-pages". This tells GitHub that the website would be rendered using the content in the gh-pages branch (and not the main branch). You might also configure a custom domain for the website, which is not something I did.

All set? Let’s move on.

Hugo Site and Theme

Every Hugo site has an associated theme to apply the site style and look. I decided to use the freely available Academia Hugo theme, and since I might need to make changes to the theme files I decided to fork their Git repo to my GitHub account.

To begin building my site, I first clone my GitHub pages repo locally, as follows:

git clone https://github.com/tanmaychk/tanmaychk.github.io
cd tanmaychk.github.io

I will add the Hugo theme to my site as a Git submodule, this will help me pull-in the theme (which is a separate Git repo, as mentioned above) changes easily. This is especially useful if I were using a theme from a Git repo I don't own.

git submodule add https://github.com/tanmaychk/academia-hugo.git themes/academia-hugo

The Academia Hugo theme thankfully comes with a core set of files needed to create a Hugo site from the theme in a folder "exampleSite", so I recursively copy this folder's entire content to my Hugo site root, as follows:

 cp -r themes/academia-hugo/exampleSite/* .

Now the basic structure of my website is ready, and I can immediately test it using the local Hugo server (yes, without even deploying my site yet), as follows:

hugo server -b http://localhost:1313

The site will now be accessible at http://localhost:1313 (your port could vary if you supplied it differently). The parameter -b is to specify the baseURL for the site because as of now I haven't touched the Hugo config.toml file, which contains all the website configurations. I could have also added the baseURL to config/_default/config.toml file directly, in lieu of specifying this parameter.

Academia theme has in-built support to add details about Courses, Projects, Skills, and Blogs. I changed the content (text as well as images) according to my needs and kept checking it locally. Images were added under the static/img folder. I also added my resume PDF under static/files folder and added the link to the PDF via the config/_default/menus.toml config file.

I created the PDF version of my resume using LaTeX and Overleaf. The GitHub repo is here if you are interested.

Generating the website

Hugo can help generate the HTML and static files that we would deploy to our website (remember raw HTML are the crux if the static website thingy). You can deploy them using online CMS tools like Netlify and Forestry.io, after generating the public website through the following simple command:

hugo

I see a new folder public now in my Hugo site folder that contains all the generated HTML files and static assets which can be straightway deployed to any hosting platform.

Since this article is about deploying the site to GitHub pages, I will now focus on that ask. At this point I can safely delete the complete public folder.

Deploying to GitHub pages

Before I start, a quick recall about the two code branches I created when setting up my repo, the default "main" and a "gh-pages" branch. I will now explain why I did that. While the "main" code branch will store all our core "source" files (that I created in above-said steps while creating my Hugo site), the "gh-pages" will contain the content that the public folder in the previous paragraph contained, i.e. the entire generated site. But I will cleverly hand over the job of creating this generated website to GitHub via Actions.

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows us to automate our build, test, and deployment pipeline.

In my case I create a workflow that will

get triggered on code check-in to the "main" branch
check-out our "main" branch code, including the theme submodule
perform a setup of Hugo (latest version)
generate the static website in "public" folder, using the "hugo" command
check-in the content of the "public" folder to the "gh-pages" branch of the repository

To create the workflow, click "Actions" link on the repository page, then click "New Workflow" and paste the YAML content provided below. Then save the workflow.

Interesting to note that GitHub stores this workflow in the "main" branch under .github/workflows folder. For more details on this Action please refer the documentation and code at https://github.com/peaceiris/actions-gh-pages.

name: GitHub Pages

on:
  push:
    branches:
      - main  # Set a branch name to trigger deployment
  pull_request:

jobs:
  deploy:
    runs-on: ubuntu-20.04
    permissions:
      contents: write
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
    steps:
      - uses: actions/checkout@v3
        with:
          submodules: true  # Fetch Hugo themes (true OR recursive)
          fetch-depth: 0    # Fetch all history for .GitInfo and .Lastmod

      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v2
        with:
          hugo-version: 'latest'
          extended: true

      - name: Build
        run: hugo --minify

      - name: Deploy
        uses: peaceiris/actions-gh-pages@v3
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_branch: gh-pages
          keep_files: true
          publish_dir: ./public

Now it's time to check-in the code from our local laptop to the GitHub pages repo "tanmaychk.github.io". As soon as the code is pushed to GitHub, the "GitHub Pages" Actions workflow gets triggered and in couple of minutes my website is published to the gh-pages branch. My site is now ready to be viewed at https://tanmaychk.github.io/.

Hope this helps to create a resume/portfolio website of your own.

What impact does Web3 have on your developer life?

Tanmay Chakrabarty — Fri, 03 Jun 2022 04:52:27 +0000

Web3, blockchain, and cryptocurrency have all become broad words that have captivated the world's attention due to their incredible benefits and rewards. This popularity has resulted in a large number of employment openings in these disciplines.

Market Development

Web3, blockchain, and cryptocurrency are predicted to grow and spread in the face of a rapidly growing tech market. According to projections, the value of web 3.0 blockchain is expected to reach $6,200 by 2023. Between 2023 and 2030, this market is predicted to develop at a 44.6percent compound annual growth rate (CAGR). While the web 3.0 blockchain business performs admirably, data also show that the crypto market is growing. From 2021 to 2030, the cryptocurrency market, which was worth $1.49 billion in 2020, is predicted to increase at a CAGR of 12.8 percent, reaching a value of roughly $5 billion. The worldwide blockchain market is also booming, with a CAGR of 8.5-9 percent predicted between 2020 and 2030, up from$5.92 billion in 2021

Web3

Web 3.0 is based on decentralised apps that allow everyone to participate without allowing them to monetize their personal information. Web 3.0 is verifiable, self-contained, permissionless, stateful, and includes native payments. Web 3.0 differs from web 2.0 in that it is based on blockchains and relies on a large number of networks and servers.

Blockchain

Blockchain is a distributed database that distributes data among computer network nodes and keeps it in electronic form, or digital format. The information in blockchain is collected in blocks, which are referred to as groups. Blockchain attempts to make it possible to capture and transmit digital information, but it does not allow for data editing. Blockchains have laid the groundwork for immutable ledgers, or records that can't be modified

Crypto

Cryptography refers to a variety of encryption algorithms that use cryptographic approaches to protect them, such as public-private key pairs, elliptical curve encryption, and so on. Cryptocurrencies are one of the most popular trading tools since they can be bought or mined via cryptocurrency exchanges. Cryptocurrencies have grown in prominence because they provide a means for speedier money transactions at a lower cost than any other service, and they are based on decentralised systems that do not shut down at a single point of failure or disconnection.

Job Market is Growing

As the success rate of cryptocurrencies and blockchain grows year after year, the number of job openings in the blockchain and cryptocurrency business grows. Jobs in blockchain are rising at a rate of 2,000-6,000 percent each year, with pay that are 50-100 percent greater than those of traditional developers. A blockchain developer is a person who works in the field of blockchain technology.

Block chain Developer: This is a highly marketable professional path in which one assists a company in marketing the blockchain platform and creating a high demand for it, which is becoming much easier now that many people are interested in the benefits block chain has to offer. To pursue this professional path, one must have a few years of experience in fields such as C++, JavaScript, and Python.
Blockchain Project Manager: This position requires the development of blockchain solutions as well as cloud project management and basic technical abilities. Work as a solution architect, UX designer, and many other positions are available. Crypto-related job postings increased by 395 percent in 2021 alone. Chief artificial intelligence engineer is one of the most in-demand positions.
Chief Artificial Intelligence Engineer:The engineer is in charge of programming and developing algorithms that aid in the development of AI and improve cybersecurity for safe and secure crypto exchanges.

As we swiftly transition into a new era with new specialities and technology, many chances in organisations and dApps (decentralised applications) that seek to solve difficult challenges are becoming available. You'll also have the opportunity to collaborate with some of the industry's brightest minds.