DEV Community: Ayush Saran

Building a Trend Analysis Tool with the FeedRika API - Part I - Setup

Ayush Saran — Fri, 19 Jul 2024 14:21:00 +0000

Building a Trend Analysis Tool with the FeedRika API

I recently came across this cool News API service called FeedRika that gives you the latest world news along with a sentiment score and relevant categories. It has a Free usage tier so I thought of trying it out and seeing what I can build with it.

One of my ideas was so build a tool to see how a company or topic has fared in the news.

You can see a chart from Google Trends that shows you how popular a term is in the public space but that only reflects the search volume. It does not give you an idea of whether the sentiment around it is positive or negative. So let's build a tool that scours the news to see if the topic is being written about favorably or not and display a similar graph.

Here are the broad steps we will take to build this tool:

Collect the topic to search for, from the user
Fetch news articles from Feedrika that match the topic
Loop through the returned articles and extract the sentiment score for each article
Plot these scores to a chart to display visually
Do some math to generate additional stats for the topic, such as average sentiment, total positive/negative, etc...
Show the source news articles to the user so they can explore the topic in further detail.

Before We Start

Let's get an API key from the Feedrika website so we can fetch news articles to work with.
Head over to feedrika.com and sign up for an account.

Once you sign up you will find your API key on your profile page feedrika.com/profile along with your credit balance and a request log showing what requests you have made.

Choosing the platform

We could build this tool in just HTML, CSS and Javascript but it involves using a private API key and it's not a good idea to transmit that openly over the internet so let's use node and express to hide the API key on the server side as an environment variable and keep it private.

I'm going to tailor this tutorial to absolute beginners so if you're already familiar with node and express feel free to skip ahead to the more interesting parts.

Setup:

1. Node and Express

Make sure you have the Node runtime environment installed. If not you can get it here.

Create a directory for this project on your local machine and navigate inside it.

Run : npm init -y in the terminal to initialize a node project with the defaults.

Run: npm i express to install the express framework.
Express is a simple webserver that will allow us to serve the pages and api routes within our application. It's easy to setup and widely used so finding help online and troubleshooting is easy.

Open up the folder in VSCode or your favorite IDE and look inside.

You should have a node_modules folder, a package.json file and a package-lock.json file.

2. Creating our first Route

Let's make an index page that welcomes users to our app
Make a new file 'welcome.html in' the root of your project. Fill it with just the basic information to get started

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Welcome</title>
</head>
<body>
    <h1>This is my news trends app!</h1>
</body>
</html>

Let's setup our first route and return this welcome.html page when someone opens the app

Create an 'index.js' file in the root of your app and import the express framework.

// Import the express framework
express = require("express");

// tell node that we are creating an express app
const app = express();

// define the port that the server will run on
const port = 3000;

// define the route for the index page
app.get("/", (req, res) => {
 res.sendFile(__dirname + "/welcome.html");
});

// Start the server and tell the app to listen for incoming connections on the port
app.listen(port, () => {
  console.log(`Server running at http://localhost:${port}`);
});

Let's test our progress.
From the terminal run node index.js. You should see a confirmation message saying the server is running

Click the link in the terminal or paste it into the browser to confirm that you can see the welcome page

3. Environment Variables

Let's setup an environment variable to save our API key.
Create a new file '.env' in the root of your project.
Copy and paste your API key from the Feedrika profile page here

Let's also add a '.gitignore' file so we don't accidently upload this private key to the web

Now for some housekeeping

We don't want to start and stop the server from the terminal every time we make an edit to the app so let's setup auto reloading.

Open your package.json file and add these lines to the script object

"start": "node index.js",
"dev": "nodemon index.js -w"

We are using nodemon with the '-w' flag to watch for changes in our root folder and restart the server.

Now we can start our server with the npm run dev command and it will automatically watch for changes and restart the server for us.

If you get an error about not recognizing nodemon run this to install it globally and try again:
npm i nodemon -g

Okay that completes the setup, lets move on to building out our App!

Let's update the welcome page and add a search box to ask for topics

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Welcome</title>
    <link rel="stylesheet" href="styles.css">
</head>

<body>
    <div id="container">
        <h1>News trends</h1>
        <h3>Search for a topic to get started</h3>
        <form class="search-form" action="/search" method="get">
            <input type="text" name="topic" placeholder="Search for a topic">
            <button type="submit">Search</button>
        </form>
    </div>
</body>

</html>

Setup Stylesheets

Create a 'public' folder in the root of your project that will host our client side javascript, css and image files.
Add a 'styles.css' file to the public folder and add some basic styles for the welcome page

styles.css:

/* Import the Inter font */
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;700&display=swap');

body {
    margin: 0;
    padding: 0;
    font-family: 'Inter', sans-serif;
}

#container {
    width: 100%;
    height: 100vh;
    display: flex;
    flex-direction: column;
    justify-content: center;
    align-items: center;
}

/* SEARCH FORM */

.search-form input {
    padding: 1em;
    border: 1px solid #ccc;
    border-radius: 8px;
}

.search-form button {
    padding: 1em;
    border: 1px solid #ccc;
    border-radius: 8px;
    background-color: #313131;
    cursor: pointer;
    color: #fff;
}

Now we need to tell express how to serve these static files so open 'index.js' and add this line:
app.use(express.static("public"));

You should be able to see the changes reflected right away, refresh the page in your browser and confirm

Great! Let's now tell express how to handle this form submission

If you notice the form it submits to a '/search' endpoint so let's setup this route and handle the form submission

Open up your 'index.js' file and add these lines

// define the route for the /search endpoint
app.get("/search", (req, res) => {
  // get the query string from the request
  let query = req.query.topic;
  // send the query string back as the response
  res.send(query);
});

Let's test it out, go to your browser and enter a search term in the box and click submit
You should see a response from the server which shows your search term, like this

Good Job!

Now that we have a search route working let's plug-in the FeedRika API and fetch news for the topic.

Coming soon Part II - Fetching Data

Dark Interfaces. The Why?

Ayush Saran — Mon, 15 Jul 2024 17:11:22 +0000

Continuing in my experiments in simplicity I decided to see what the Matomo analytics dashboard would look like in dark mode.

Here is the original dashboard, in it's dated color scheme:

And here is a quick draft of the dashboard in a modern dark theme:

What Works?

Every visual element requires processing and removing the shadows and colors allows for the eyes to quickly find information because of the lack of competing elements to separate out.

Reducing the color scheme to only the essentials increases the contrast and makes the page look less daunting at first glance.

The higher contrast of white text on a black background makes it easy to direct focus to the data instead of the box-backgrounds and shadows which are no longer competing against the text for attention.

What doesn't ?

Some of the branding of the original interface is lost in dark mode, it now looks like every other dark-ui dashboard.

This may not be a huge issue as the decision to prioritize readability, legibility and information clarity seems like the right trade-off in this case.

Overall I'm happy with the direction this is taking and I will continue to refine this draft and see if I can get to a finished design for the other screens also, then compare the whole set vs the original to see if it aids usability..

A Simple One-Page Portfolio Template

Ayush Saran — Mon, 15 Jul 2024 17:00:21 +0000

This week I worked on simplifying my personal website. Although I loved the character of the old design I wanted to boil the page down to it's basics to make the vital information easier to scan and process.

It's tempting to get caught up in visual elements when you're designing something but it's important to question the why? Like who is this page for? What are they looking for? How can I get them to the information they want faster and easier.

Here is my new minimal one-page template:
www.ayushsaran.info

I decided to release a freebie version on GitHub so anyone can use this for their own portfolio or personal one-pager.

It's just a single .html file with CSS included so you can host it anywhere and if you host it on github pages then your hosting and CI/CD is taken care of automatically for free.

Get it on GitHub: https://github.com/ayushsaranGithuB/one-page-portfolios

Building A SimpleNote to Obsidian notes converter

Ayush Saran — Tue, 09 Jul 2024 21:57:06 +0000

Building A SimpleNote to Obsidian notes converter

See it in action:

https://simplenote-to-obsidian.fly.dev/

WHY?

I like quickly jotting down notes in SimpleNote but after seeing Obsidian pop up all over the web I was curious and decided to try it out. And I liked it, the ability to add more structure and link documents seemed like a great way to build up to a longer post by referencing snippets and sections that I had saved up as un-related notes in SimpleNote.

The problem

SimpleNotes saves notes in .txt files and while it's easy to export out of SimpleNotes there's no direct way to transfer the notes over to Obsidian.

I tried looking into the official migration guide at:
https://help.obsidian.md/import

I even tried looking for community plugins that would handle this:
https://obsidian.md/plugins

It seemed bizarre that there was no official way to import from SimpleNotes, which is quite a popular platform. Then I stumbled across this python script from https://github.com/philgyford/simplenote-to-obsidian/blob/main/convert_notes.py and decided to build my own web version in JavaScript.

HOW IT WORKS

Upload the JSON file from SimpleNote that contains an export of all the notes
Parse each note and convert the Tags into Obsidian compatible #Tags
Export this updated note into its own .md file
Change the modified date to reflect the date from the original SimpleNote file
Save these .md notes into a folder
Create a Zip for export that can be downloaded to import into Obsidian

The stack

Usually I'd build this in NEXT.js and host it on Vercel but having recently run into a API route timeout I knew that Vercel allows only ~5-10 seconds for script execution in API routes (on the Hobby Tier) and I was worried that parsing the notes could take some time in larger collections so I built this as a simple Node + Express App and decided to try hosting it on Fly.io

It wasn't AS easy as connecting a github repo to Vercel, but within 20 mins I had my account setup on fly.io and the app deployed

Challenges

File Upload in Express v4

I just couldnt get the uploaded file through to the controller. The POST data from the form looked okay on the front-end and all my code looked ok on the Express end but any calls to req.body came up as an empty object {}!

I cracked my head around this for an hour before stumbling across this StackOverflow post where a kind poster pointed out that file uploads in forms are no longer handled by Express as of v4 and require an external library. I went with 'express-fileupload' https://www.npmjs.com/package/express-fileupload

Creation Date

There is a function to update the 'Modified by' and 'Last Accessed' date in 'fs' for node that works across platforms but changing the 'Created At' date is trickier.

The fs-futimessync function does not handle 'Created At' dates
https://www.geeksforgeeks.org/node-js-fs-futimessync-method/?ref=lbp

I did stumble upon https://github.com/baileyherbert/utimes.
A Cross-platform native addon to change the btime, mtime, and atime of files in Node.js.
It seems to fix the issue for Win + Mac systems so I might give it a try for the next iteration of this tool

Finally...

And that's it. I was finally able to build the tool and test it and it works great!
I was able to import all my SimpleNote data into Obsidian easily and the tags were preserved

See it in action:

https://simplenote-to-obsidian.fly.dev/

Give it a go and let me know if it helps you.

+ BONUS

I've shared the tool as a github-repo for other people to mess around with
https://github.com/ayushsaranGithuB/simplenote-to-obsidian

Comments, criticisms, suggestions are welcome
Please file an issue if you find a bug: https://github.com/ayushsaranGithuB/simplenote-to-obsidian/issues

Using 11ty and DecapCMS for 'non-blog' static websites -pt 2 Pages

Ayush Saran — Sat, 18 May 2024 22:51:53 +0000

While there are plenty of examples for using 11ty and Decap CMS in blog settings, I struggled to find a way to make it work for a non-blog static website.

Leaving these tips out on the internet for anybody else looking to do the same.

Making a static landing page editable in DecapCMS

This tip is for non markdown pages, such as landing pages that have unique structure and content that cant be edited in a WYSIWYG editor because of the complex underlying HTML structure that goes with the landing page.

I still needed to make all the text on this page editable so I decided to make them into strings that can be edited via the DecapCMS admin UI.

There might be a simpler approach but I went with loading all the text as frontmatter in a markdown file. Here is what my home landing page looks like

index.md



---
title: "Uplocal - Homepage"
layout: home.liquid
hero-tagline: THE MARKETING OS for MULTI LOCATION BUSINESSES
hero-h1-firstline: Accelerate Local Demand
hero-h1-secondline: Elevate Customer Experience
hero-h1-thirdline: Collaborate Seamlessly
hero-signup-cta: See how <strong class="handwritten">XXXX</strong> can transform your local marketing
hero-signup-button: Try <strong>XXXX</strong>
leading-brands-headline: Trusted by Leading Brands
local-marketing-callout: Customer journeys are non-linear. Every touch point matters
// ... and so on

Every instance of text that should be editable in the admin is saved as a field in the front-matter. The rest of the file has no body content, just front-matter

The relevant template to use to render this page is also mentioned in the front-matter:
layout: home.liquid

Then I added this page and all these fields to the collections array in the config.yml file for Decap under admin/config.yml



collections:
  - label: "Pages"
    name: "pages"
    files:
      - label: "Home"
        name: "home"
        file: "index.md"
        fields:
          - { label: Title, name: title, widget: string }
          - { label: Layout, name: layout, widget: hidden, default: home.html }
          - { label: hero-tagline, name: hero-tagline, widget: string }
          - {
              label: hero-h1-firstline,
              name: hero-h1-firstline,
              widget: string,
            }
          - {
              label: hero-h1-secondline,
              name: hero-h1-secondline,
              widget: string,
            }
          - {
              label: hero-h1-thirdline,
              name: hero-h1-thirdline,
              widget: string,
            }
          - { label: hero-signup-cta, name: hero-signup-cta, widget: string }
          - {
              label: hero-signup-button,
              name: hero-signup-button,
              widget: string,
            }
//... and so on

That makes these fields editable in the Admin UI

On the template side:

In the template for this page, the fields are referenced using template tags. EJS in my case



---
layout: layout.ejs
---
<section class="hero">
    <div class="content">
        <h2 class="callout">
            {{ hero-tagline }}
        </h2>
        <h1>
            <span>{{ hero-h1-firstline }}</span>
            <span>{{ hero-h1-secondline }}</span>
            <span>{{ hero-h1-thirdline }}</span>
        </h1>
        <div class="signup">
            <p>
                {{ hero-signup-cta }}
            </p>
            <form action="/signup">
                <input type="text" placeholder="Enter Email Address!" />
                <button type="submit">
                    {{ hero-signup-button }}
                </button>
            </form>
        </div>
    </div>
//... and so on

This line at the top in the front-matter of the template
layout: layout.ejs
tells 11ty that this template extends the layout.ejs tenplate to nest the landing page content inside the common header/footer frame that is also used for the rest of the site.

Using 11ty and DecapCMS for 'non-blog' static websites -pt 1 Menus

Ayush Saran — Sat, 18 May 2024 22:32:03 +0000

While there are plenty of examples for using 11ty and Decap CMS in blog settings, I struggled to find a way to make it work for a non-blog static website.

Leaving these tips out on the internet for anybody else looking to do the same.

Making Menus editable in DecapCMS

To make menus that you can edit and update in DecapCMS you need to start with a collection in your config.yml file for DecapCmS [found under admin/config.yml]

mine looked like this:

collections:
  - name: Menus
    label: Menus
    files:
      - file: _data/mainMenu.json
        label: Main Navigation
        name: main-navigation
        extension: "json"
        format: "json"
        fields:
          - label: Primary Menu Items
            name: nav_items
            widget: list
            fields:
              - { label: Label, name: label, widget: string }
              - { label: Path, name: path, widget: string }

The menu items are saved in a JSON file (mainMenu.json) as an object

{
  "nav_items": [
    {
      "label": "Products",
      "path": "/products"
    },
    {
      "label": "Solutions",
      "path": "/solutions"
    },
    {
      "label": "Success Stories",
      "path": "/successStories"
    },
    {
      "label": "Resources",
      "path": "/resources"
    },
    {
      "label": "About",
      "path": "/about"
    }
  ]
}

The file is placed in the '_data' directory in the root of the 11ty folder. This makes it accessible in templates as a Global Data File. See 11ty - Global Data Files

Then inside your template you can access this array and loop over the entries to output your links. I use EJS in my template, it looks like this:

   <ul class="menu">
        <!-- Load Nav items from mainMenu.json file -->
        <% for (let i=0; i < mainMenu.nav_items.length; i++) { %>
            <li><a href="<%= mainMenu.nav_items[i].path %>">
                    <%= mainMenu.nav_items[i].label %>
                </a></li>
            <% } %>
    </ul>

You can also use this for Mega Menus, with nested lists

collections:
 - file: _data/footerMenu.json
        label: Footer Links
        name: footer-navigation
        extension: "json"
        format: "json"
        fields:
          - label: Footer Menu Items
            name: nav_items
            widget: list
            fields:
              - { label: Label, name: label, widget: string }
              - label: Links
                name: links
                widget: list
                fields:
                  - { label: Label, name: label, widget: string }
                  - { label: Path, name: path, widget: string }

Notice the nested 'fields:' arrays in two levels

Then loop over them like this:

 <% for (let i=0; i < footerMenu.nav_items.length; i++) { %>
            <ul class="footer-links">
                <li class="title">
                    <h5>
                        <%= footerMenu.nav_items[i].label %>
                    </h5>
                </li>
                <% for (let j=0; j < footerMenu.nav_items[i].links.length; j++) { %>
                    <li><a href="<%= footerMenu.nav_items[i].links[j].path %>">
                            <%= footerMenu.nav_items[i].links[j].label %>
                        </a></li>
                    <% } %>
            </ul>
            <% } %>