DEV Community: BHARGAB KALITA

How I Automated My GitHub Profile README With GitHub Actions (And How You Can Automate Anything Too)

BHARGAB KALITA — Mon, 01 Dec 2025 22:01:16 +0000

At the end of last year, I started writing a few more articles on Dev.to. I’d been posting for a while, but I’m honestly too lazy to keep updating my GitHub Profile README every time. Opening the repo, editing the README, adding links… it wasn’t difficult, just boring. And boring tasks always make me wonder why I’m still doing them manually.

So I looked around, tried a few things, and with a bit of help from AI, I finally found a way to automate it. After some quick experiments, everything clicked. My profile started updating itself with my latest Dev.to articles. No manual edits. No drama. Just simple automation doing its job.

It was a small win, but it felt great. And that’s what pushed me to share how I did it and what else you can automate using GitHub Actions.

Before You Start: The Special GitHub Profile Repository

This part is important.

GitHub only displays your README as your public profile if you create a repo exactly named after your username.

For example:

Username → yourusername
Profile repo → also yourusername

If you don’t have this repository yet, create a new public one with the same name as your username. Once it exists:

the README becomes your profile README
you can add workflows in .github/workflows
automation will run exactly like in any other repo

This is the repo where I added my workflow last year, and it’s still running perfectly.

How I Automated My Profile README to Show My Latest Dev.to Articles

The goal was simple:
Show the 3 latest posts from Dev.to in my README without updating anything manually.

I changed my workflow so that it fetches only the latest 3 articles and rewrites the section in my README.

Here’s the complete GitHub Actions workflow.

GitHub Actions Workflow (Shows Only 3 Latest Dev.to Articles)

Create:

.github/workflows/update-readme.yml

Add:

name: Update README with latest Dev.to Post

on:
  schedule:
    - cron: '0 0 * * *'
  workflow_dispatch:

jobs:
  update-readme:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3

      - name: Fetch Latest Dev.to Posts (replace the username)
        id: fetch_dev_posts
        run: |
          DEVTO_USERNAME="bhargab"
          API_RESPONSE=$(curl -s "https://dev.to/api/articles?username=${DEVTO_USERNAME}")

          # Get the titles, URLs, and publish dates of the latest 3 posts
          TITLE_1=$(echo $API_RESPONSE | jq -r '.[0].title')
          URL_1=$(echo $API_RESPONSE | jq -r '.[0].url')
          DATE_1=$(echo $API_RESPONSE | jq -r '.[0].published_at' | cut -d'T' -f1)

          TITLE_2=$(echo $API_RESPONSE | jq -r '.[1].title')
          URL_2=$(echo $API_RESPONSE | jq -r '.[1].url')
          DATE_2=$(echo $API_RESPONSE | jq -r '.[1].published_at' | cut -d'T' -f1)

          TITLE_3=$(echo $API_RESPONSE | jq -r '.[2].title')
          URL_3=$(echo $API_RESPONSE | jq -r '.[2].url')
          DATE_3=$(echo $API_RESPONSE | jq -r '.[2].published_at' | cut -d'T' -f1)

          # Pass these values to the GitHub environment for use in the next steps
          echo "title_1=$TITLE_1" >> $GITHUB_ENV
          echo "url_1=$URL_1" >> $GITHUB_ENV
          echo "date_1=$DATE_1" >> $GITHUB_ENV

          echo "title_2=$TITLE_2" >> $GITHUB_ENV
          echo "url_2=$URL_2" >> $GITHUB_ENV
          echo "date_2=$DATE_2" >> $GITHUB_ENV

          echo "title_3=$TITLE_3" >> $GITHUB_ENV
          echo "url_3=$URL_3" >> $GITHUB_ENV
          echo "date_3=$DATE_3" >> $GITHUB_ENV

      - name: Update README
        run: |
          README_FILE="./README.md"
          # Build the new content for the latest 3 posts
          NEW_CONTENT="- [${{ env.title_1 }}](${{ env.url_1 }}) - Published on ${{ env.date_1 }}\n\- [${{ env.title_2 }}](${{ env.url_2 }}) - Published on ${{ env.date_2 }}\n\- [${{ env.title_3 }}](${{ env.url_3 }}) - Published on ${{ env.date_3 }}"

          # Replace the placeholder section using a different delimiter (|)
          sed -i '/<!-- LATEST_DEVTO_POST -->/{n;N;N;s|.*|'"$NEW_CONTENT"'|}' $README_FILE

      - name: Commit Changes
        run: |
          git config --local user.name "github-actions[bot]"
          git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git add README.md --force
          git commit -m "Update README with latest 3 Dev.to posts" || echo "No changes to commit"
          git push

And in your README:

<!-- ARTICLES_START -->
<!-- ARTICLES_END -->

GitHub Actions replaces everything between those two markers every time it runs.

Your profile stays updated. You don’t touch anything.

What Else Can You Automate With GitHub Actions?

Once you get comfortable with this, you’ll realize GitHub Actions can automate pretty much anything you can script. Here are some actually useful examples:

1. Automatically Add Your Latest YouTube Videos

If you upload tutorials or dev content, this keeps your profile fresh.

- name: Fetch YouTube videos
  run: |
    curl -s "https://www.googleapis.com/youtube/v3/search?key=$YT_API_KEY&channelId=$CHANNEL_ID&part=snippet&order=date&maxResults=5" > videos.json

2. Auto-Update GitHub Stats or Badges

You can show live data:

follower count
stars
views
repo metrics

- run: curl -s https://api.github.com/users/YOUR_USER | jq '.followers'

You can then embed that into badges or markdown.

3. Use GitHub Actions as a Free Cron Server

Run scheduled tasks without having any server.

on:
  schedule:
    - cron: "0 */6 * * *"  # Every 6 hours

Good for:

cleaning old branches
updating analytics
regenerating documentation
syncing data

4. Create Your Own Personal Automation Bots

This part is way more fun than it sounds.

Examples:

get a Telegram/Discord ping when someone stars your repo
generate daily GitHub activity summaries
sync posts from X/Twitter or LinkedIn
rebuild your portfolio site every night
generate OG images
track metrics for projects

If you can script it, GitHub Actions can run it for free.

A Simple Automation Template You Can Reuse

name: My Automation

on:
  schedule:
    - cron: "0 0 * * *"
  workflow_dispatch:

jobs:
  run-script:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python3 script.py

Drop your logic into script.py and you’re done.

If you’ve been meaning to automate your own workflows or make your profile more dynamic, this is a great place to start. Try it, break it, tweak it, and make it your own hand✌️.

Optimizing Performance in Next.js and React.js: Best Practices and Strategies

BHARGAB KALITA — Tue, 25 Feb 2025 11:24:23 +0000

Performance optimization forms the backbone of great web experiences, and it’s a critical focus when building with React.js and Next.js. A fast app keeps users satisfied, reduces load times, enhances SEO, and scales effectively. Drawing from practical projects, this guide explores detailed strategies to maximize performance in both frameworks, blending actionable tips with lessons learned from real-world applications.

0. Table of Contents

To navigate directly to a specific optimization strategy, use the links below:

1. Identifying Performance Bottlenecks
2. Image Optimization
3. Code Splitting and Tree Shaking
4. Server-Side and Client-Side Caching
5. Load Balancing for Scalability
6. Lazy Loading and Intersection Observer
7. Preventing Unnecessary Re-Renders and Managing State Efficiently
8. Server-Side Performance Optimization
9. Purging CSS
10. Inlining Critical CSS
11. Optimizing Fonts and Third-Party Scripts
12. Optimizing Third-Party API Calls
13. Offloading Third-Party Scripts with Partytown
14. Using Next Bundle Analyzer to Identify Bundle Size Issues
15. Rendering Huge Lists with Virtual Lists and Infinite Scroll
16. Continuous Monitoring and Performance Improvements

1. Identifying Performance Bottlenecks

Optimization begins with pinpointing issues. Tools like Google PageSpeed Insights and Lighthouse, prove invaluable for assessing metrics such as Largest Contentful Paint (LCP)—the time until main content appears—Cumulative Layout Shift (CLS)—how much the page shifts during loading—and Time to First Byte (TTFB)—the server’s response speed. For React.js, the React Developer Tools Profiler highlights components that re-render excessively or load slowly. Skipping this step once led to hours wasted on misdirected efforts—profiling first has since become a non-negotiable starting point.

2. Image Optimization

Images often hinder performance but offer significant optimization potential. Here’s how they’re managed effectively:

Next.js: The next/image component streamlines the process by resizing images, converting them to efficient formats like WebP, and lazy-loading them by default. For above-the-fold images, such as homepage banners, setting loading="eager" and fetchpriority="high" prioritizes them for LCP. Here’s an example from a recent project:

  import Image from "next/image";
  function HeroSection() {
    return (
      <Image
        src="/banner.jpg"
        width={1200}
        height={600}
        fetchpriority="high"
        alt="Main banner"
      />
    );
  }

React.js: Without Next.js’s built-in tools, libraries like react-lazy-load-image-component or manual loading="lazy" attributes on <img> tags handle the task. It’s less automated but effective.
Extras: Serving images via a CDN (e.g., Cloudflare) speeds up delivery. Modern formats like WebP reduce file sizes dramatically—transforming a 2MB PNG into 200KB proved this point. Skeleton screens or blurred placeholders (auto-generated in Next.js with blurDataURL) prevent CLS, enhancing the loading experience noticeably.

3. Code Splitting and Tree Shaking

Large JavaScript bundles silently degrade performance, but modern bundlers like Webpack mitigate this with code splitting and tree shaking:

React.js: React.lazy and Suspense enable on-demand component loading. In a dashboard app, lazy-loading a chart component not visible on initial load cut the main bundle by 300KB:

  const Chart = React.lazy(() => import("./Chart"));
  function Dashboard() {
    return (
      <Suspense fallback={<div>Loading chart...</div>}>
        <Chart />
      </Suspense>
    );
  }

Next.js: next/dynamic offers similar benefits, often with SSR disabled for client-heavy components. For example:

  import dynamic from "next/dynamic";
  const Map = dynamic(() => import("../components/Map"), { ssr: false });

Next.js also splits code by page automatically, a boon for multi-page apps.

Tree Shaking: Modern bundlers like Webpack enable tree shaking by default in production mode, eliminating unused code. Using named imports (e.g., import { useEffect } from 'react') instead of import * maximizes this feature. A bloated bundle from a library’s import *as once ballooned a build—switching to specific imports let Webpack trim 150KB of dead code.

4. Server-Side and Client-Side Caching

Caching eliminates redundant work, boosting efficiency:

Next.js: getStaticProps pre-renders static pages (e.g., blogs), while getServerSideProps handles dynamic content (e.g., user profiles). Incremental Static Regeneration (ISR) balances static efficiency with freshness, updating pages in the background. An e-commerce site used this to refresh product listings every 10 seconds without full rebuilds:

  export async function getStaticProps() {
    const products = await fetchProducts();
    return { props: { products }, revalidate: 10 };
  }

React.js: Client-side caching with SWR or React Query keeps API calls lightweight and data current:

  import useSWR from "swr";
  const fetcher = (url) => fetch(url).then((res) => res.json());
  function Profile() {
    const { data } = useSWR("/api/user", fetcher);
    return <div>{data?.name}</div>;
  }

Extras: Static assets cached on a CDN with Cache-Control: max-age=31536000 and service workers for offline support dropped one app’s load time from 2 seconds to under 1 second.

5. Load Balancing for Scalability

Load balancing ensures stability during traffic spikes:

Next.js: Deploying with Nginx as a reverse proxy distributes requests across servers. Vercel’s auto-scaling handled a flash sale seamlessly with zero downtime. Stateful apps (e.g., chat features) benefit from sticky sessions to maintain user context across servers.
React.js: As a client-side framework, this applies more to hosting or backend setups, where platforms like Netlify or Cloudflare balance static assets automatically.
Lesson: Neglecting this once crashed a site during a surge—planning for scale is now standard practice.

6. Lazy Loading and Intersection Observer

Lazy loading minimizes initial load overhead:

React.js: The Intersection Observer API loads content as it enters the viewport. An infinite scroll feature built this way runs smoothly:

  const [isVisible, setIsVisible] = useState(false);
  const ref = useRef();
  useEffect(() => {
    const observer = new IntersectionObserver(([entry]) => {
      if (entry.isIntersecting) {
        setIsVisible(true);
        observer.disconnect();
      }
    });
    observer.observe(ref.current);
    return () => observer.disconnect();
  }, []);
  return <div ref={ref}>{isVisible ? <BigVideo /> : <Skeleton />}</div>;

Next.js: Pairing next/dynamic with Intersection Observer defers heavy components, like footers, until needed.

7. Preventing Unnecessary Re-Renders and Managing State Efficiently

Excessive re-renders slow apps, but they’re manageable with the right techniques and state management choices:

React.js:
- React.memo prevents re-renders unless props change, halving render time for a list item component:
```
const ListItem = React.memo(({ text }) => <li>{text}</li>);
```
- useMemo and useCallback optimize costly operations. A filtering app avoided lag by memoizing a sorted array:
```
const sortedData = useMemo(() => data.sort((a, b) => a - b), [data]);
const handleFilter = useCallback(() => applyFilter(), []);
```
- Avoiding inline JSX functions (e.g., <button onClick={() => foo()}>) prevents unnecessary recreations that disrupt child components.
Next.js: These techniques carry over, with SSR and SSG offloading some rendering from the client, reducing client-side re-render pressure.
State Management: Context suits small apps due to its simplicity, but larger Next.js projects benefit from Zustand or Redux for performance and scalability. A key reason to prefer Redux over useContext lies in re-rendering behavior: with useContext, every component consuming the context re-renders whenever any part of the context value changes, even if it doesn’t use the updated state. For example, updating a single user property in a context triggers re-renders across all consumers:

  const MyContext = createContext({ user: { name: "John" }, theme: "light" });
  function NameDisplay() {
    const { user } = useContext(MyContext); // Re-renders on any context change
    return <div>{user.name}</div>;
  }

Redux, by contrast, uses a store with selectors (e.g., via reselect) to ensure components only re-render when their specific slice of state changes:

  const selectUserName = (state) => state.user.name;
  function NameDisplay() {
    const name = useSelector(selectUserName); // Only re-renders if name changes
    return <div>{name}</div>;
  }

This granularity avoids unnecessary re-renders, critical in complex apps with frequent state updates. Normalizing state (e.g., flat objects) further enhances efficiency. Switching from Context to Zustand or Redux once transformed a sluggish UI into a responsive one by eliminating wasteful renders.

8. Server-Side Performance Optimization

Server-side enhancements accelerate Next.js apps:

Load balancers (noted above) pair with optimized database queries—Prisma avoids N+1 issues, cutting TTFB from 500ms to 100ms in one case.
HTTP/2 or HTTP/3, enabled via Nginx, speeds up asset delivery.
Gzip or Brotli compression reduced a docs site’s payload from 1MB to 300KB.
React.js: This applies more to API servers, where lean fetch calls maintain client-side efficiency.

9. Purging CSS

CSS bloat quietly undermines performance, making purging essential:

Next.js: Tailwind CSS’s built-in purging (via purgecss) strips unused classes at build time. Configuring tailwind.config.js to scan relevant files cut a portfolio site’s CSS from 500KB to 20KB:

  module.exports = {
    content: ["./pages/**/*.{js,ts,jsx,tsx}", "./components/**/*.{js,ts,jsx,tsx}"],
    theme: { extend: {} },
    plugins: [],
  };

React.js: Integrating PurgeCSS or cssnano into Webpack works well:

  const PurgeCSSPlugin = require("purgecss-webpack-plugin");
  const glob = require("glob");
  module.exports = {
    plugins: [
      new PurgeCSSPlugin({
        paths: glob.sync(`${__dirname}/src/**/*`, { nodir: true }),
      }),
    ],
  };

Impact: Unpurged CSS from libraries once bloated a React app with 1MB of unused Bootstrap styles—purging became a must after that.

10. Inlining Critical CSS

Inlining critical CSS—styles for above-the-fold content—accelerates first paint:

Next.js: Tools like Critical or Penthouse extract critical CSS for inlining in the <head>. Running Critical in the build process:

  npx critical pages/index.html --base . --inline > pages/index-critical.html

Then injecting it with <Head>:

  import Head from "next/head";
  export default function Home() {
    return (
      <>
        <Head>
          <style dangerouslySetInnerHTML={{ __html: `body { margin: 0; } .hero { font-size: 2rem; }` }} />
        </Head>
        <div className="hero">Welcome!</div>
      </>
    );
  }

Non-critical CSS loads async via <link rel="stylesheet">.

React.js: The critical-css-webpack-plugin automates this in Webpack:

  const CriticalCssPlugin = require("critical-css-webpack-plugin");
  module.exports = {
    plugins: [new CriticalCssPlugin({ src: "index.html", inline: true })],
  };

Approach: Lighthouse identifies critical rendering path blockers, then Critical or Penthouse extracts styles for headers or hero sections. Inlining 2KB of critical CSS on a landing page dropped FCP from 1.5s to 0.8s, deferring the rest.

11. Optimizing Fonts and Third-Party Scripts

Fonts and scripts subtly impact performance:

Next.js: next/font hosts fonts locally, with font-display: swap ensuring text visibility during load.
React.js: Self-hosting critical fonts with <link rel="preload"> and deferring others avoids blocking.
Scripts: Third-party tools (e.g., analytics) use async or defer:

  <script src="tracking.js" defer></script>

Deferring a chat widget once shaved off a 1-second delay.

12. Optimizing Third-Party API Calls

Third-party APIs can bottleneck apps if mishandled:

React.js: Batching requests with axios or Promise.all and debouncing rapid calls (e.g., autocomplete) conserve resources:

  const debounce = (fn, delay) => {
    let timeout;
    return (...args) => {
      clearTimeout(timeout);
      timeout = setTimeout(() => fn(...args), delay);
    };
  };
  const fetchSuggestions = debounce(async (query) => {
    const res = await fetch(`/api/suggestions?q=${query}`);
    setSuggestions(await res.json());
  }, 300);

Next.js: Pre-fetching in getStaticProps or getServerSideProps bakes data into pages, while SWR handles client-side updates:

  import useSWR from "swr";
  function LiveData() {
    const { data } = useSWR("/api/live", fetcher, { refreshInterval: 5000 });
    return <div>{data?.value}</div>;
  }

Takeaway: Setting 5-second timeouts on fetches and falling back to cached data when APIs fail halved one app’s load time after over-fetching issues surfaced.

13. Offloading Third-Party Scripts with Partytown

Third-party scripts like analytics or ads often bog down the main thread, but Partytown offloads them to web workers for better performance:

Next.js with App Router: Introduced in Next.js 12.1 as an experimental feature, the next/script component’s strategy="worker" uses Partytown to run scripts in a web worker. Enable it in next.config.js:

  module.exports = {
    experimental: { nextScriptWorkers: true },
  };

Then, in a layout or page:

  import Script from "next/script";
  export default function RootLayout({ children }) {
    return (
      <html lang="en">
        <body>
          {children}
          <Script src="https://www.googletagmanager.com/gtag/js?id=G-XXXX" strategy="worker" />
        </body>
      </html>
    );
  }

Note: As of February 22, 2025, this remains experimental and unsupported in the Next.js 13+ App Router due to stability issues. Testing revealed Google Analytics offloading worked but took time to register hits, suggesting patience or alternative strategies like lazyOnload for now.

React.js: Partytown integrates manually via @builder.io/partytown/react. Install it (npm install @builder.io/partytown), then add scripts with type="text/partytown" and the Partytown component:

  import { Partytown } from "@builder.io/partytown/react";
  function App() {
    return (
      <>
        <head>
          <script type="text/partytown" src="https://example.com/analytics.js"></script>
          <Partytown forward={["dataLayer.push"]} />
        </head>
        <div>My App</div>
      </>
    );
  }

Copy Partytown’s worker files to public/~partytown for serving.

Benefits and Caveats: Offloading frees the main thread, improving responsiveness—Lighthouse scores jumped from 69 to 85 in one test after moving Google Tag Manager to a worker. However, not all scripts (e.g., those needing synchronous DOM access) work seamlessly. Partytown’s trade-offs documentation advises proxying requests for CORS issues, a common snag with some APIs.

14. Using Next Bundle Analyzer to Identify Bundle Size Issues

Large bundles can silently inflate load times, and Next Bundle Analyzer provides clarity on what’s taking up space in Next.js apps:

Setup: Install @next/bundle-analyzer (npm install @next/bundle-analyzer) and configure it in next.config.js:

  const withBundleAnalyzer = require('@next/bundle-analyzer')({
    enabled: process.env.ANALYZE === 'true',
  });
  module.exports = withBundleAnalyzer({
    // Other Next.js config options
  });

Run it with ANALYZE=true next build to generate a visual report.

Usage: After building, the analyzer opens a browser window with an interactive treemap, showing bundle sizes for pages, components, and dependencies. Each block’s size and color indicate its contribution—larger, darker blocks highlight heavy modules. For example, a bloated third-party library like moment.js might dominate the treemap, signaling a need to switch to a lighter alternative like day.js.
React.js Context: While Next Bundle Analyzer is Next.js-specific, React.js projects can use webpack-bundle-analyzer similarly. Add it to Webpack config:

  const BundleAnalyzerPlugin = require('webpack-bundle-analyzer').BundleAnalyzerPlugin;
  module.exports = {
    plugins: [new BundleAnalyzerPlugin()],
  };

Running the build generates a comparable report.

Practical Impact: Analyzing one Next.js app revealed a 1.2MB bundle from an unused charting library imported across pages—lazy-loading it with next/dynamic slashed the initial load by 800KB. Regularly checking bundle sizes ensures optimization efforts target the biggest culprits, like oversized dependencies or unminified code.

15. Rendering Huge Lists with Virtual Lists and Infinite Scroll

Rendering large datasets, like thousands of items, can cripple performance, but virtual lists and infinite scroll optimize this:

React.js with Virtual Lists: Libraries like react-virtualized or react-window render only visible items in the viewport, not the entire list. For a 10,000-item list, react-window keeps DOM nodes minimal:

  import { FixedSizeList } from 'react-window';
  const Row = ({ index, style }) => <div style={style}>Item {index}</div>;
  function VirtualList() {
    return (
      <FixedSizeList height={400} width={300} itemCount={10000} itemSize={35}>
        {Row}
      </FixedSizeList>
    );
  }

This approach dropped render time from seconds to milliseconds in a data-heavy app.

Infinite Scroll: Building on Intersection Observer (see #6), infinite scroll fetches more items as users scroll. Here’s an example fetching 20 items at a time:

  const [items, setItems] = useState([]);
  const [page, setPage] = useState(1);
  const loader = useRef(null);
  useEffect(() => {
    const observer = new IntersectionObserver(
      (entries) => {
        if (entries[0].isIntersecting) {
          setPage((prev) => prev + 1);
        }
      },
      { threshold: 1.0 }
    );
    if (loader.current) observer.observe(loader.current);
    return () => observer.disconnect();
  }, []);
  useEffect(() => {
    fetch(`/api/items?page=${page}`).then((res) =>
      res.json().then((data) => setItems((prev) => [...prev, ...data]))
    );
  }, [page]);
  return (
    <div>
      {items.map((item, i) => (
        <div key={i}>{item}</div>
      ))}
      <div ref={loader}>Loading...</div>
    </div>
  );

Next.js: Combine next/dynamic with virtual lists for client-side rendering of large datasets, or pre-fetch initial items with getStaticProps. Testing showed a 50,000-item list rendered smoothly with react-virtualized, while infinite scroll kept server requests efficient.
Why It Matters: Rendering all items naively spikes memory usage and slows scrolling—virtualization and infinite scroll ensure smooth performance even with massive datasets.

16. Continuous Monitoring and Performance Improvements

Optimization requires ongoing effort:

Weekly Lighthouse runs and Web Vitals logging catch regressions—CLS spikes once flagged an image issue.
Google Analytics reveals navigation patterns, guiding rendering tweaks like preloading popular pages.
Keeping Next.js and React updated leverages performance fixes—a minor update once shaved 100ms off load time effortlessly.

Conclusion

From tackling slow React re-renders to leveraging Next.js’s SSR and ISR, optimization blends framework features with disciplined coding. Tools like React.memo, next/image, ISR, CSS purging, inlined critical CSS, Partytown offloading, bundle analysis, virtual lists, and API optimizations, paired with caching and monitoring, transform apps into efficient, user-friendly experiences. Applying these strategies in projects consistently yields noticeable improvements.

Note

This article will be continuously updated as new optimization techniques, tools, or insights emerge. Check back periodically for the latest strategies to keep Next.js and React.js applications performing at their peak.

The Intricacies of MongoDB Aggregation Pipeline: Challenges and Insights from Implementing It with Go

BHARGAB KALITA — Sat, 04 Jan 2025 10:21:56 +0000

MongoDB’s aggregation pipeline is a powerful framework for data transformation and computation. It is especially valuable for developers working with NoSQL databases, offering unparalleled flexibility to handle complex data manipulation tasks. However, implementing this feature in a statically typed language like Go presents unique challenges. This article explores the aggregation pipeline's core functionality, underlying mechanics, and the challenges I faced while integrating it with Go. Along the way, I share solutions, recommendations, and practical insights to guide developers in similar scenarios.

Understanding the Aggregation Pipeline

MongoDB’s aggregation pipeline is designed to process data in stages, each performing a specific operation. By chaining these stages, developers can create highly complex queries. Some of the most commonly used stages include:

$match: Filters documents to include only those matching specified conditions.
$group: Aggregates data by a specified field, applying operations like sum, average, and count.
$sort: Orders documents by specified fields.
$project: Modifies the structure of documents, including or excluding fields as needed.
$lookup: Performs a left outer join with another collection.

These stages operate independently, enabling MongoDB to optimize execution through indexing and parallel processing. Understanding these components is crucial for crafting efficient queries.

How the Aggregation Pipeline Works Internally

Internally, MongoDB’s aggregation pipeline relies on a systematic process to maximize efficiency:

Execution Plan Generation: The pipeline is parsed into an optimized execution plan, leveraging indexes and reordering stages for efficiency.
Sequential Data Flow: Data passes through each stage sequentially, with the output of one stage feeding into the next.
Optimization Techniques: MongoDB merges compatible stages and pushes operations like $match and $sort earlier to minimize processed data volume.
Parallel Processing: For large datasets, MongoDB distributes tasks across multiple threads, enhancing scalability.

By understanding these internal mechanisms, developers can design pipelines that efficiently leverage MongoDB’s processing capabilities.

Challenges of Implementing the Aggregation Pipeline with Go

1. Schema-less Nature of MongoDB

MongoDB’s flexible schema can complicate integration with Go, which relies on strict typing. Constructing dynamic aggregation stages in such an environment can be challenging.

Solution: Using the bson.M and bson.D types from the MongoDB Go driver allowed dynamic construction of pipelines. However, careful validation was necessary to ensure consistency, as strict type safety was partially sacrificed.

2. Complex Query Construction

Aggregation pipelines often involve deeply nested structures, making query construction cumbersome and error-prone in Go.

Solution: Helper functions were created to encapsulate repetitive stages like $group. This modular approach improved code readability and reduced the risk of errors.

3. Debugging and Error Handling

Error messages from aggregation pipelines can be vague, making it difficult to identify issues in specific stages.

Solution: Logging the JSON representation of pipelines and testing them in MongoDB Compass simplified debugging. Additionally, the Go driver’s error-wrapping features helped trace issues more effectively.

4. Performance Bottlenecks

Stages like $lookup and $group are resource-intensive and can slow down performance, especially with large datasets.

Solution: Using MongoDB’s explain function helped pinpoint inefficiencies. Optimizing indexes, reordering stages, and introducing batch processing significantly improved performance.

5. Concurrency Management

Running multiple aggregation queries simultaneously can strain resources, leading to latency and connection pool saturation.

Solution: Adjusting connection pool parameters and implementing context-based timeouts ensured better resource management. Monitoring throughput allowed for dynamic scaling, preventing bottlenecks.

Recommendations for Efficient Usage

Run Aggregation Pipelines in Cron Jobs: Aggregation pipelines are resource-intensive and can impact real-time services. Scheduling them as separate cron jobs ensures better system stability.
Define Indexes Clearly: Carefully choose which fields to index to optimize performance. Regularly review query patterns and adjust indexes as needed to reduce execution time.

Lessons Learned

1. Leverage Debugging Tools

Tools like MongoDB Compass and the explain function are invaluable for visualizing query execution plans and identifying bottlenecks.

2. Optimize Pipeline Order

Place filtering and sorting stages like $match and $sort early in the pipeline to minimize data volume processed by subsequent stages.

3. Encapsulate Pipeline Logic

Modularizing commonly used pipeline stages into reusable components simplifies maintenance and reduces duplication.

4. Monitor System Resources

Regularly track connection pool usage, query execution times, and overall system performance. Implement resource thresholds and alerts to avoid service disruptions.

Closing Thoughts 💭

Integrating MongoDB’s aggregation pipeline with Go is both challenging and rewarding. The combination of MongoDB’s dynamic schema and Go’s strict typing requires thoughtful planning and problem-solving. By understanding the pipeline’s mechanics and applying best practices, developers can overcome these challenges to achieve scalable, efficient solutions.

The Art of Prefetching and Preloading: Enhancing Web Performance

BHARGAB KALITA — Mon, 30 Dec 2024 16:08:54 +0000

In today’s fast-paced digital world, speed is everything. A slow-loading website doesn’t just frustrate users—it costs you engagement, conversions, and credibility. Enter prefetching and preloading, two often-underappreciated techniques that can elevate your web performance from "meh" to "wow."

But what are these magic tricks, and how can you wield them effectively? Let’s break them down.

🎯 Understanding Prefetching and Preloading

At a high level, both techniques help browsers fetch resources before they’re explicitly needed, but their purposes differ:

Prefetching: Think of it as a browser’s crystal ball. It predicts which resources you might need next (like a linked page) and loads them in the background.
Preloading: This focuses on resources needed right now (like fonts, images, or critical scripts) and ensures they’re prioritized during page rendering.

Here’s a simple analogy:

Prefetching is like prepping ingredients for tomorrow’s meal.
Preloading is making sure the stove is hot before you start cooking today’s dinner.

🚀 Why You Should Care

Your site’s performance isn’t just about speed—it’s about the perception of speed. Prefetching and preloading make your website feel faster, even when the total load time stays the same.

Benefits include:

Faster navigation and rendering.
Reduced latency for future interactions.
Improved user experience, especially on slower networks.

🔧 How to Implement Prefetching

Prefetching is all about anticipating user behavior. You can use it to pre-load assets like the next page’s HTML, CSS, and JavaScript.

Basic Example

<link rel="prefetch" href="next-page.html">

Here’s how it works:

The browser fetches next-page.html in the background.
When the user clicks the link, the page is loaded almost instantly.

Pro Tips for Prefetching

Focus on High-Probability Actions: Use analytics to predict the user’s next step. For instance, e-commerce sites can prefetch product details after users browse categories.
Use rel="prerender" for Entire Pages: When confident, use prerendering to fully load the next page in a hidden tab.

   <link rel="prerender" href="checkout.html">

Avoid Overloading the Network: Fetching too many resources can hurt overall performance.

🔥 Mastering Preloading

Preloading helps you prioritize essential resources for the current page.

Basic Example

<link rel="preload" href="styles.css" as="style">

The as attribute specifies the type of resource (e.g., style, script, font), so the browser can handle it efficiently.

When to Use Preloading

Critical Fonts: Avoid the dreaded “flash of invisible text” (FOIT) by preloading your web fonts.

   <link rel="preload" href="my-font.woff2" as="font" type="font/woff2" crossorigin="anonymous">

Above-the-Fold Images: Ensure the hero image is visible as soon as the page loads.
Key Scripts: For JavaScript critical to page interactivity.

🛠 Best Practices for Both

Monitor Resource Usage: Tools like Chrome DevTools and Lighthouse can help identify what to preload or prefetch.
Combine with Frameworks: If you’re using React or Next.js, these frameworks often have built-in support for prefetching.

   import Link from 'next/link';

   <Link href="/about" prefetch={true}>About</Link>

Test and Iterate: Not every resource needs prefetching or preloading. A/B test to find the sweet spot.

🧪 Case Study: Boosting Navigation Speed

Let’s say you’re running an e-commerce site:

Use prefetching to load product pages as users hover over links in the product grid.
Use preloading to prioritize the checkout button’s JavaScript and styles.

The Result:

Users experience near-instant page transitions.
Cart abandonment rates drop because the checkout process feels seamless.

⚙️ Tools to Measure Impact

Google Lighthouse: Evaluate preloading efficiency and identify unused resources.
WebPageTest: See how prefetching impacts real-world performance.
Network Tab in DevTools: Check which resources are being preloaded or prefetched.

🎨 The Art of Balance

Prefetching and preloading aren’t silver bullets. They require careful tuning to avoid bloated resource usage or negative impacts on slower connections.

Use prefetching to anticipate user behavior strategically.
Apply preloading sparingly for critical assets.

When done right, these techniques can create a fast, frictionless experience that users love.

Final Thoughts

The art of prefetching and preloading lies in understanding your users and optimizing their journey. It’s not just about technical finesse—it’s about delivering a digital experience that feels intuitive, smooth, and almost magical.

Now that you know the ropes, it’s time to put these techniques into action. Happy optimizing!

Let me know your thoughts or any questions in the comments. 👇

Optimizing Docker Image Size for Go Applications

BHARGAB KALITA — Sat, 26 Oct 2024 09:29:16 +0000

In the world of software development, I’m constantly trying to improve my applications. Recently, I faced a challenge: my Docker images for a Go application were bloated and unwieldy. This not only slowed down deployment times but also put unnecessary strain on my resources. Specifically, the size of the images was causing issues, as they filled the available disk space of my EC2 instance, which had a limit of 1GB. After optimization, the image size dramatically decreased from approximately 800MB (compressed) to just 7.95MB (compressed). Join me as I share my journey to trim down those images, the culprits I identified, and the valuable lessons I learned along the way.

The Initial Dockerfile: The Culprit Unveiled

At first glance, my Dockerfile seemed straightforward. Here’s how it looked:

# Use the latest golang base image
FROM golang:1.21.6

# Set the working directory
WORKDIR /app

# Copy go.mod and go.sum
COPY go.mod go.sum ./
RUN go mod download

# Copy the entire application code
COPY . .

# Build the Go application
RUN go build -o main .

# Expose port 4040
EXPOSE 4040

# Command to run the executable
CMD ["./main"]

While it worked, it became painfully clear that this approach led to massive image sizes. The inclusion of unnecessary files and the use of a bulky base image were the main offenders.

The Unveiling: What Went Wrong?

During my investigation, I identified several culprits contributing to my oversized Docker images:

All Files, No Filters: My initial setup copied everything from the project directory, including test files, documentation, and any stray files that had no business being in a production environment.
Heavyweight Base Image: I was using the full Go image for both building and running my application. This included a ton of tools and libraries that I simply didn’t need in production.
Missing Multi-Stage Builds: My Dockerfile didn’t leverage multi-stage builds, which allow you to build an application in one image and then use a minimal image for running it. This meant my final image included build dependencies and tools that were unnecessary for execution.

The Transformation: Streamlining the Dockerfile

To address these issues, I rolled up my sleeves and made some significant changes. Here’s the revamped Dockerfile that emerged from my efforts:

# Stage 1: Build the Go application
FROM golang:1.21.6 AS builder

# Set the working directory
WORKDIR /app

# Copy go.mod and go.sum files
COPY go.mod go.sum ./
RUN go mod download

# Copy the source code
COPY . .

# Build the Go application with optimizations
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o main .

# Stage 2: Create a minimal runtime image with CA certificates
FROM alpine:latest

# Install CA certificates
RUN apk add --no-cache ca-certificates

WORKDIR /app

# Copy the binary from the builder stage
COPY --from=builder /app/main .

EXPOSE 4040

# Command to run the executable
CMD ["./main"]

The Key Changes

Multi-Stage Builds: I split my Dockerfile into two stages. The first stage is dedicated to building the application, while the second stage uses a minimal base image (in this case, Alpine) to run it. This separation allowed me to leave behind all the unnecessary build tools and dependencies.
Selective Copying: I started being selective about what files I copied into the image. This is where using a .dockerignore file became invaluable. I ensured only the necessary files made it into my Docker image.

Example of a .dockerignore file:

# Ignore test files and directories
*.test
testdata/

# Ignore version control directories
.git

Optimizing the Go Binary: By setting CGO_ENABLED=0 and using build flags -ldflags="-w -s", I produced a smaller, statically linked binary.

Why I Added CA Certificates

In my final image, I installed CA certificates using the command RUN apk add --no-cache ca-certificates. This step was crucial for several reasons:

Secure Connections: My application communicates with external services, including databases and APIs, that require secure connections over HTTPS. Installing CA certificates enables the Go application to verify the TLS/SSL certificates presented by these services, ensuring secure communication.
Minimized Vulnerabilities: Using CA certificates helps prevent man-in-the-middle attacks. By validating certificates, I can trust that the data exchanged with external services is secure and not being intercepted.

Understanding `CGO_ENABLED=0` and Build Flags

I included CGO_ENABLED=0 and the flags -ldflags="-w -s" for several reasons:

Static Binary Creation: Setting CGO_ENABLED=0 tells the Go compiler to build a statically linked binary. This means the resulting executable does not rely on any shared libraries from the host system. This is particularly beneficial in a Docker environment where I may not have the same libraries available as I do on my local machine.
Reduced Image Size: A statically linked binary is generally smaller because it includes only the code and libraries necessary to run the application. This reduction in size is crucial when deploying to environments like Docker, where every byte counts.
Optimized Performance: The -ldflags="-w -s" flags strip debug information and symbol tables from the binary. This not only reduces the binary size further but also enhances startup performance, which is beneficial in containerized environments where quick startup times are essential.

Lessons Learned: More Than Just Size

The Right Base Image Matters: The choice of base image can make or break your Docker image size. Going with minimal images not only helps in reducing size but also enhances security by limiting the attack surface.
Power of Multi-Stage Builds: These builds are a game-changer. They help streamline the build process and keep the final image clean and lean.
Be Mindful of Your Copying: Always scrutinize what you include in your Docker images. Using .dockerignore helps keep your builds clean.
Continuous Optimization is Key: Image optimization isn’t a one-time task. Regular reviews and improvements will help maintain a lean codebase and efficient images.

Conclusion

Through this journey, I successfully reduced my Docker image sizes, leading to faster deployments and improved resource efficiency. This experience taught me not just about the technical aspects of Docker but also reinforced the importance of mindful practices in software development.

In the world of DevOps, every byte counts, and I'm excited to continue this journey of optimization and discovery!

Understanding Parallelism vs Concurrency in Go

BHARGAB KALITA — Wed, 09 Oct 2024 07:18:00 +0000

When we talk about parallelism and concurrency, it's easy to confuse the two because they both involve executing multiple tasks at the same time. However, they are fundamentally different concepts with distinct applications, especially in Go, a language designed with these features at its core. Let's break down what each means, their differences, and how we can achieve them in Go.

Concurrency: Dealing with Lots of Tasks

Concurrency is the ability to handle multiple tasks at once, even if they aren't being executed simultaneously. Think of it as juggling several tasks, but not necessarily completing them at the same time.

Imagine you’re working on a report while also replying to emails. You aren't writing the report and typing emails simultaneously, but you’re switching back and forth between the two. This switching allows both tasks to make progress without waiting for one to be completed before starting the other. Similarly, concurrency in computing lets a system handle multiple operations or tasks by switching between them, giving the illusion of simultaneous execution.

In Go, concurrency is achieved using goroutines. A goroutine is a lightweight thread of execution, allowing Go to run functions concurrently. For example, you can have multiple goroutines that make network calls, process data, or run other functions at the same time.

Here's a simple example of concurrency in Go:

package main

import (
    "fmt"
    "time"
)

func printNumbers() {
    for i := 1; i <= 5; i++ {
        fmt.Println(i)
        time.Sleep(100 * time.Millisecond)
    }
}

func printLetters() {
    for i := 'A'; i <= 'E'; i++ {
        fmt.Printf("%c\n", i)
        time.Sleep(100 * time.Millisecond)
    }
}

func main() {
    go printNumbers() // Run printNumbers concurrently
    go printLetters() // Run printLetters concurrently

    time.Sleep(1 * time.Second) // Wait to ensure both goroutines finish
}

In this example, printNumbers and printLetters are run as goroutines, meaning they execute concurrently. The system alternates between running them, resulting in interleaved output, even though the two functions aren’t running at the exact same time.

Parallelism: Doing Multiple Things at Once

Parallelism, on the other hand, is about actually performing multiple tasks simultaneously. This only occurs when you have multiple processors or cores. Imagine you and a friend are both typing two different documents at the same time. That's parallelism—two tasks happening at the same time on different CPUs or cores.

Parallelism is a subset of concurrency. All parallelism is concurrency, but not all concurrency is parallelism. Parallelism requires hardware that supports simultaneous task execution, whereas concurrency is more about structuring tasks to improve responsiveness and throughput, regardless of whether they run in parallel.

In Go, parallelism is achieved when goroutines are distributed across multiple CPU cores. Go allows you to control the number of cores that can execute goroutines using the GOMAXPROCS setting. By default, Go will run goroutines across as many CPU cores as are available, making it easy to take advantage of parallelism when the hardware allows it.

Here’s an example of how you can use GOMAXPROCS to enable parallelism in Go:

package main

import (
    "fmt"
    "runtime"
    "sync"
)

func task(id int, wg *sync.WaitGroup) {
    defer wg.Done()
    fmt.Printf("Task %d is running\n", id)
}

func main() {
    runtime.GOMAXPROCS(4) // Set the number of OS threads (cores) to use

    var wg sync.WaitGroup
    for i := 1; i <= 10; i++ {
        wg.Add(1)
        go task(i, &wg) // Run tasks concurrently and in parallel (if available)
    }
    wg.Wait() // Wait for all tasks to complete
}

In this case, the GOMAXPROCS function sets the number of CPU cores available to run goroutines in parallel. With 4 cores, Go can execute up to 4 tasks simultaneously, fully utilizing the available hardware.

Key Differences Between Concurrency and Parallelism

Concurrency is about dealing with multiple tasks at the same time, but not necessarily running them at the same instant. It’s more about structuring the program to make progress on many tasks by rapidly switching between them.
Parallelism is about executing multiple tasks simultaneously on different processors or cores.

Think of concurrency as juggling many balls (tasks) by switching focus between them, while parallelism is juggling many balls by using multiple hands (cores).

Achieving Concurrency and Parallelism in Go

Go makes both concurrency and parallelism simple to implement, thanks to its goroutine model and built-in support for multi-core systems.

Concurrency is achieved by using goroutines to run tasks independently without waiting for each to complete.
Parallelism happens when these goroutines are distributed across multiple CPU cores, which Go handles automatically but can be fine-tuned using GOMAXPROCS.

Go's design philosophy emphasizes making concurrency easy to work with through its channels and goroutines, and it automatically optimizes for parallelism when the underlying hardware supports it.

Real-Life Examples of Concurrency and Parallelism

Web Server (Concurrency)

Imagine running an e-commerce website like Amazon. Multiple users are browsing, placing orders, and checking out simultaneously. The web server handles each request concurrently, ensuring that no single user is blocked by another. The server can switch between handling requests for displaying product pages, adding items to the cart, or processing payment, without waiting for each to complete sequentially.
Video Streaming Platform (Parallelism)

In a video-sharing platform like YouTube, users upload videos that need to be transcoded into multiple formats (1080p, 720p, etc.). Instead of processing each video sequentially, parallelism allows the platform to transcode several videos at the same time using multiple CPU cores. This speeds up processing, allowing quicker availability of videos for streaming.
Chat Application (Concurrency)

A real-time chat application like WhatsApp needs to handle messages from many users concurrently. Each user sends and receives messages independently, so the system processes each user’s messages concurrently, ensuring a smooth real-time experience for everyone without messages being delayed or blocked by others.
Machine Learning (Parallelism)

In machine learning, training models on large datasets can take hours. By splitting the dataset into chunks, a system can use parallelism to train on multiple data chunks simultaneously across different CPU or GPU cores. This speeds up the training process, enabling faster model development for tasks like image recognition or recommendation engines.
Traffic Management System (Concurrency)

In a city traffic management system where cameras at multiple intersections send updates to a central server, concurrency is used to process real-time data from multiple locations at once. The system can concurrently handle traffic signal data, vehicle counts, and accident reports, ensuring smooth city-wide traffic flow without delay in processing data from different intersections.

Conclusion

In Go, concurrency is about managing multiple tasks and making progress on them, while parallelism takes that one step further by executing them simultaneously. Both are crucial for building efficient, scalable programs, and Go makes it incredibly easy to implement them with minimal overhead. Understanding the distinction allows developers to choose the right approach based on the problem they are solving—whether it’s to improve the responsiveness of an application with concurrency or to fully leverage CPU power with parallelism.

Playground: https://codesandbox.io/p/devbox/2qg64j

Rollup.js Made Easy: A Step-by-Step Guide to Building and Publishing NPM Packages

BHARGAB KALITA — Wed, 04 Sep 2024 15:17:48 +0000

Rollup.js is a fantastic tool for bundling JavaScript and TypeScript libraries, especially when you’re aiming for top-notch performance and smooth integration. Unlike other bundlers, Rollup focuses on creating smaller, optimized code by utilizing ES modules. If you’re looking to build efficient packages, Rollup is your go-to choice.

In this post, we’ll explore Rollup’s key features, guide you through setting it up for package building, show you how to test your package locally, then publishing it to NPM. We’ll also highlight some essential plugins to enhance your build process, handle CSS files and images, and configure TypeScript for your project.

What is Rollup.js?

Rollup.js is a module bundler for JavaScript and TypeScript that compiles small pieces of code into something larger, like a library or application. It takes advantage of ES modules to streamline your code, removing unused elements and resulting in bundles that are smaller and faster.

Key Features of Rollup.js

Tree-Shaking: Automatically removes unused code, slimming down the final bundle and boosting efficiency.
ES Module Support: Rollup uses ES module syntax (import and export), which is ideal for modern JavaScript development.
Plugins: Easily extensible with plugins that add functionality, such as handling non-JS files or minifying output.
Code Splitting: Allows splitting of bundles, which is particularly useful for larger applications or libraries.
Output Formats: Supports a variety of output formats like CommonJS, ES Modules, UMD, and IIFE, making it versatile for different project needs.
Performance Optimizations: Rollup produces highly optimized code with smaller bundle sizes, making it perfect for performance-critical applications.

Setting Up Rollup.js for Building Packages

Let’s dive into setting up Rollup.js to build a package with React.js and TypeScript.

1. Install Rollup and Basic Dependencies

First, let’s get Rollup and some essential plugins installed:

npm install -D rollup @rollup/plugin-node-resolve @rollup/plugin-commonjs @rollup/plugin-typescript rollup-plugin-peer-deps-external rollup-plugin-postcss tslib @rollup/plugin-image rollup-plugin-terser

Here’s what each plugin does:

@rollup/plugin-node-resolve: Helps Rollup find third-party modules in node_modules.
@rollup/plugin-commonjs: Converts CommonJS modules to ES modules so Rollup can handle them.
@rollup/plugin-typescript: Compiles TypeScript files seamlessly.
rollup-plugin-peer-deps-external: Automatically marks peer dependencies as external, keeping your bundle light.
rollup-plugin-postcss: Processes and bundles CSS, which is crucial for handling styles in React components.
@rollup/plugin-image: Allows images to be imported directly into your components.
rollup-plugin-terser: Minifies the output, reducing file size and improving load times.
tslib: This is a runtime library for TypeScript that contains all of the TypeScript helper functions.

2. Setting Up TypeScript Configuration

Create a tsconfig.json in your project root to configure TypeScript:

{
    "compilerOptions": {
      "target": "ES6",
      "module": "ESNext",
      "jsx": "react",
      "declaration": true,
      "declarationDir": "./",
      "rootDir": "./src",
      "outDir": "./dist",
      "strict": true,
      "esModuleInterop": true,
      "skipLibCheck": true,
      "forceConsistentCasingInFileNames": true,
      "sourceMap": true,
    },
    "include": ["src/**/*"],
    "exclude": ["node_modules", "dist"]
}

This configuration ensures your TypeScript code is compiled correctly and efficiently.

3. Create a Basic Rollup Configuration

Next, create a rollup.config.js file in your project root:

import resolve from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';
import typescript from '@rollup/plugin-typescript';
import peerDepsExternal from 'rollup-plugin-peer-deps-external';
import postcss from 'rollup-plugin-postcss';
import image from '@rollup/plugin-image';
import { terser } from 'rollup-plugin-terser';

export default {
  input: 'src/index.ts',
  output: [
    {
      file: 'dist/index.cjs.js',
      format: 'cjs',
      sourcemap: true,
    },
    {
      file: 'dist/index.esm.js',
      format: 'esm',
      sourcemap: true,
    },
  ],
  plugins: [
    peerDepsExternal(),
    resolve(),
    commonjs(),
    typescript({
      tsconfig: './tsconfig.json',
      declaration: true,
      declarationDir: './',
      rootDir: './src',
    }),
    postcss(),
    image(),
    terser(),
  ],
  external: ['react', 'react-dom'],
};

Building and Watching with Rollup

To make your development process smoother, add these scripts to your package.json:

"scripts": {
  "build": "rollup -c",
  "watch": "rollup -c --watch"
}

Now, you can build your package or watch for changes and rebuild automatically during development.

Testing Your Rollup Package Locally

Before publishing, it’s essential to test your package locally. Two effective ways are npm pack and npm link.

1. Testing with `npm pack`

npm pack simulates the npm install process, letting you test how your package would behave once published.

Run:

   npm pack

This creates a .tgz file in your project directory.
To test, install it in another project:

   npm install ../path-to-your-package/your-package-name-version.tgz

2. Testing with `npm link`

npm link lets you link your package globally and use it across projects without publishing.

Inside your package project, run:

   npm link

Then, in the project where you want to use the package, run:

   npm link your-package-name

Handling CSS Files and Images in Rollup

Here’s how to handle CSS and images in your Rollup setup:

1. Handling CSS

Using rollup-plugin-postcss, you can process and bundle CSS files, even extracting them into separate files.

import postcss from 'rollup-plugin-postcss';

export default {
  plugins: [
    postcss({
      extract: true,
      minimize: true,
      modules: true,
    }),
  ],
};

2. Handling Images

@rollup/plugin-image makes it easy to import images directly, and you can optimize them with additional plugins like imagemin.

import image from '@rollup/plugin-image';

export default {
  plugins: [
    image(),
    // Add further image optimization here if needed
  ],
};

Publishing Your Package to npm

With Rollup set up, the final step is to publish your package on npm:

1. Prepare Your `package.json`

Ensure that your package.json includes essential details such as name, version, main, and files. Additionally, always use semantic versioning for versioning.

{
  "name": "my-awesome-package", // The name of your package
  "version": "1.0.0",           // Version number
  "description": "A description of what your package does.",
  "main": "dist/index.cjs.js",  // Entry point for CommonJS
  "module": "dist/index.esm.js", // Entry point for ES Modules
  "types": "dist/types/index.d.ts", // Entry point for TypeScript declarations
  "scripts": {
    "build": "rollup -c",
    "watch": "rollup -c --watch",
  },
  "files": [                    // Specify which files to include in the npm package
    "dist"
  ],
  "repository": {
    "type": "git",
    "url": "git+https://github.com/your-username/your-repo.git"
  },
  "keywords": ["microfrontend", "rollup", "typescript", "react"],
  "author": "Your Name",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/your-username/your-repo/issues"
  },
  "homepage": "https://github.com/your-username/your-repo#readme",
  "dependencies": {
    "react": "^18.0.0",
    "react-dom": "^18.0.0"
  },
  "devDependencies": {
    "@rollup/plugin-commonjs": "^26.0.1",
    "@rollup/plugin-image": "^3.0.3",
    "@rollup/plugin-node-resolve": "^15.2.3",
    "@rollup/plugin-typescript": "^11.1.6",
    "rollup": "^2.79.1",
    "rollup-plugin-peer-deps-external": "^2.2.4",
    "rollup-plugin-postcss": "^4.0.2",
    "rollup-plugin-terser": "^7.0.2",
    "tslib": "^2.7.0",
    "typescript": "^5.5.4"
  }
}

2. Log In and Publish to npm

If you’re not logged in, use:

npm login

Then, build your package:

npm run build

And publish it:

npm publish --access public

Now, your package is live on npm, ready for anyone to install with:

npm install my-awesome-package

GitHub Repository:

To access the codebase, please visit the dedicated GitHub repository:

GitHub Repository: create-your-own-package

Feel free to clone the repository, explore the code, and use it as a reference for your own projects. If you encounter any issues or have questions, don't hesitate to open an issue on the repository. Your feedback and contributions are highly appreciated.

Conclusion

Rollup.js is a powerhouse for building efficient and optimized packages. Its modularity, plugin support, and ES module compatibility make it invaluable for developers. Whether you're crafting a simple package or tackling complex micro-frontends, Rollup.js provides the tools you need for streamlined, high-quality builds.

You can apply the same configuration approach to build a micro-frontend, leveraging Rollup's flexibility to create modular and reusable components. Just be sure to research the plugins you include to avoid unnecessary dependencies and keep your build process lean and efficient.

And there you have it! With Rollup.js in your toolkit, you’re all set to bundle your projects like a pro. As you navigate the sea of code, remember to enjoy the journey—after all, it's not just about the destination, but how you bundle along the way! 🚀

Happy bundling, and may your code be ever concise and your builds ever smooth! 😁

How to implement Oauth in Go?

BHARGAB KALITA — Mon, 11 Mar 2024 10:44:18 +0000

Hi guys, this article will guide you through the process of implementing OAuth in Go, leveraging the power of Gin for routing, Redis for session management, and Goth for handling OAuth providers. Before you begin, make sure to install and set up a Redis server on your local machine by following this link.

We are going to use the below packages

Gin - github.com/gin-gonic/gin
Redis - github.com/redis/go-redis/v9
Goth - github.com/markbates/goth
Gorilla Session - github.com/gorilla/sessions

Setting Up Redis for Session Storage:

To store user sessions, we'll initialize a Redis client and define functions for setting and retrieving key-value pairs. Ensure to replace the placeholder values with your own Redis configuration.

import (
    "context"

    "github.com/redis/go-redis/v9"
)

var ctx = context.Background()

// Replace these values with your own configuration
const (
    redisAddr     = "localhost:6379"
    redisPassword = ""
)

var rdb *redis.Client

func InitRedis() {
    // Set up Redis session store
    rdb = redis.NewClient(&redis.Options{
        Addr:     redisAddr,
        Password: redisPassword, // no password set
        DB:       0,             // use default DB
    })
}

func GetRedisClient() *redis.Client {
    return rdb
}

func SetValue(key string, value string) {
    err := rdb.Set(ctx, key, value, 0).Err()
    if err != nil {
        panic(err)
    }
}

func GetValue(key string) (string, error) {
    val, err := rdb.Get(ctx, key).Result()
    if err != nil {
        return "", err
    }
    return val, nil
}

Session Management Helper Functions:

Next, implement helper functions for managing user sessions. These functions create, retrieve, and delete sessions, ensuring a seamless authentication experience.

package helpers

import (
    "context"
    "encoding/json"
    "time"

    "auth-go/model"

    "github.com/google/uuid"
    "github.com/markbates/goth"
    "github.com/redis/go-redis/v9"
)

var ctx = context.Background()

func CreateSession(client *redis.Client, user *goth.User) (*model.Session, error) {
    sessionID := generateSessionID() // Generate a unique ID
    session := &model.Session{
        ID:           sessionID,
        UserID:       user.UserID,
        RefreshToken: user.RefreshToken,
        ExpiresAt:    time.Now().Add(time.Hour * 24), // Set expiration time
    }

    sessionData, err := json.Marshal(session)
    if err != nil {
        return nil, err
    }

    err = client.Set(ctx, sessionID, sessionData, session.ExpiresAt.Sub(time.Now())).Err()
    if err != nil {
        return nil, err
    }

    return session, nil
}

func GetSession(client *redis.Client, sessionID string) (*model.Session, error) {
    sessionData, err := client.Get(ctx, sessionID).Result()
    if err == redis.Nil {
        return nil, nil // Session not found
    } else if err != nil {
        return nil, err
    }

    var session model.Session
    err = json.Unmarshal([]byte(sessionData), &session)
    if err != nil {
        return nil, err
    }

    // Check if session is expired
    if session.ExpiresAt.Before(time.Now()) {
        return nil, nil // Session expired
    }

    return &session, nil
}

func DeleteSession(client *redis.Client, sessionID string) error {
    return client.Del(ctx, sessionID).Err()
}

// Helper functions
func generateSessionID() string {
    return uuid.New().String()
}

Initializing Authentication Configuration:

Now, let's set up the authentication configuration using the Goth library and configure Google as the OAuth provider. The Goth package supports a variety of providers, allowing you to explore and integrate additional providers according to your needs. This flexibility is crucial for handling user authentication seamlessly.

Before proceeding, ensure you have created a project in the Google Console and obtained the GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET. For testing purposes, make sure to add test users to the "Test Users" section in the Google Console, as this step is essential for the proper functioning of the authentication.

package auth

import (
    "os"

    "github.com/markbates/goth"
    "github.com/markbates/goth/gothic"
    gothGoogle "github.com/markbates/goth/providers/google"
    "github.com/gorilla/sessions"
)

func InitAuth() {
    // Replace with your SESSION_SECRET or a secure secret key
    sessionSecretKey := "Secret-session-key"

    // Session configuration
    maxAge := 86400 * 30 // 30 days
    isProd := false      // Set to true when serving over HTTPS

    // Create a new CookieStore for session management
    store := sessions.NewCookieStore([]byte(sessionSecretKey))
    store.Options.Path = "/"
    store.Options.HttpOnly = true // HttpOnly should always be enabled
    store.Options.Secure = isProd
    store.MaxAge(maxAge)

    // Set the store for the Goth library
    gothic.Store = store

    // Configure Google OAuth provider
    googleClientID := os.Getenv("GOOGLE_CLIENT_ID")
    googleClientSecret := os.Getenv("GOOGLE_CLIENT_SECRET")

    goth.UseProviders(
        gothGoogle.New(googleClientID, googleClientSecret, "http://localhost:3000/auth/google/callback", "email", "profile"),
    )

}

Middleware Setup:

Two essential middlewares are introduced – CORS middleware for handling cross-origin requests and an authentication middleware to verify user sessions.

CORS Middleware

package middleware

import (
    "github.com/gin-gonic/gin"
)

func CORSMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        c.Writer.Header().Set("Access-Control-Allow-Origin", "http://localhost:5173")
        c.Writer.Header().Set("Access-Control-Max-Age", "86400")
        c.Writer.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE, UPDATE")
        c.Writer.Header().Set("Access-Control-Allow-Headers", "X-Requested-With, Content-Type, Origin, Authorization, Accept, Client-Security-Token, Accept-Encoding, x-auth")
        c.Writer.Header().Set("Access-Control-Expose-Headers", "Content-Length")
        c.Writer.Header().Set("Access-Control-Allow-Credentials", "true")

        if c.Request.Method == "OPTIONS" {
            c.AbortWithStatus(200)
        } else {
            c.Next()
        }
    }
}

Auth Middleware

In addition to the authentication middleware, consider enhancing your security by adding access-token (JWT) verification in the headers. It adds an extra layer of validation, ensuring the integrity and authenticity of the tokens.

It's essential to note a key concept of JWT – the ability to validate the token without contacting the issuer every time. By checking the ID and verifying the signature of the token with the known public key of the certificate Google used to sign the token, you can achieve efficient and secure validation. This approach eliminates the need to call Google APIs for validation, enhancing the performance of your authentication system.

For a detailed guide on consuming a Google ID token from a server, you can refer to this insightful article. The article provides valuable insights into the JWT validation process and can serve as a helpful resource in implementing secure token verification.

Ensure that you adapt the JWT validation process based on your specific requirements and security considerations.

package middleware

import (
    "net/http"
    "auth-go/model"
    "auth-go/redis"
    "auth-go/utils/ids"
    "auth-go/utils/messages"

    "github.com/gin-gonic/gin"
)

func AuthMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        sessionId, err := c.Request.Cookie("session_id")
        if err != nil {
            c.AbortWithStatusJSON(http.StatusBadRequest, model.BaseResponse{Message: messages.INVALID_SESSION, Code: ids.INVALID_SESSION})
        }
        // Verify session data in Redis
        _, err = redis.GetValue(sessionId.Value)
        if err != nil {
            c.AbortWithStatusJSON(http.StatusBadRequest, model.BaseResponse{Message: messages.INVALID_SESSION, Code: ids.INVALID_SESSION})
        }
        // Session is valid, continue processing
        c.Next()
    }
}

Routing Setup:

Now, let's define routes and initialize the necessary components in the main function.

func init() {
    // Load environment variables from .env file
    err := godotenv.Load()
    if err != nil {
        fmt.Println("Error loading .env file:", err)
        return
    }
    // initialize redis client
    redis.InitRedis()
}

func main() {

    router := gin.Default()

    auth.InitAuth()

    router.Use(middleware.CORSMiddleware())
    router.GET("/auth/:provider/callback", controllers.HandleAuthCallback)
    router.GET("/auth/:provider", controllers.HandleAuthProvider)
    router.GET("/logout/:provider", controllers.LogoutHandler)
    router.Use(middleware.AuthMiddleware())
    router.GET("/check", controllers.CheckHandler)

    fmt.Println("Listening on localhost:3000")
    if err := router.Run(":3000"); err != nil {
        fmt.Println("Error starting server:", err)
    }
}

Authentication Controllers:

Implement controllers for handling authentication callbacks, provider initiation, and user logout.

package controllers

import (
    "context"

    "net/http"
    "auth-go/helpers"
    "auth-go/model"
    "auth-go/redis"
    "auth-go/utils/ids"
    "auth-go/utils/messages"

    "github.com/gin-gonic/gin"
    "github.com/markbates/goth/gothic"
)

func HandleAuthCallback(c *gin.Context) {
    res := c.Writer
    req := c.Request
    // Get the value of the "provider" path parameter
    provider := c.Param("provider")
    // Set the provider value in the request context
    ctx := context.WithValue(req.Context(), "provider", provider)
    req = req.WithContext(ctx)
    user, err := gothic.CompleteUserAuth(res, req)
    if err != nil {
        // c.String(http.StatusInternalServerError, fmt.Sprintf("Authentication error: %s", err))
        c.AbortWithStatusJSON(http.StatusInternalServerError, model.BaseResponse{Message: messages.AUTH_ERROR, Code: ids.AUTH_ERROR})
        return
    }
    session, err := helpers.CreateSession(redis.GetRedisClient(), &user)
    if err != nil {
        // Handle error
        c.AbortWithStatusJSON(http.StatusInternalServerError, model.BaseResponse{Message: messages.WENT_WRONG, Code: ids.WENT_WRONG})
    }
    // Set the session ID in the cookie
    c.SetCookie("session_id", session.ID, 60*60, "/", "localhost", true, false)
    // disable-caching
    c.Header("Cache-Control", "no-cache, no-store, must-revalidate")
    // user.AccessToken
    c.Redirect(http.StatusFound, "http://localhost:5173")
}

func HandleAuthProvider(c *gin.Context) {
    res := c.Writer
    req := c.Request
    provider := c.Param("provider")
    // Set the provider value in the request context
    ctx := context.WithValue(req.Context(), "provider", provider)
    req = req.WithContext(ctx)
    gothic.BeginAuthHandler(res, req)
}

func LogoutHandler(c *gin.Context) {
    sessionId, err := c.Request.Cookie("session_id")
    if err != nil {
        c.AbortWithStatusJSON(http.StatusBadRequest, model.BaseResponse{Message: messages.INVALID_SESSION, Code: ids.INVALID_SESSION})
    }
    helpers.DeleteSession(redis.GetRedisClient(), sessionId.Value)
    c.JSON(http.StatusOK, model.BaseResponse{Message: messages.SUCCESS, Code: ids.SUCCESS})
}

Check Endpoint Controller:

Finally, a controller is provided to check if the authentication is working after login.

package controllers

import (
    "net/http"

    "auth-go/model"
    "auth-go/utils/ids"
    "auth-go/utils/messages"

    "github.com/gin-gonic/gin"
)

func CheckHandler(c *gin.Context) {
    c.JSON(http.StatusOK, model.BaseResponse{Message: messages.SUCCESS, Code: ids.SUCCESS})
}

GitHub Repository:

To access the complete codebase for this OAuth implementation in Go, including the backend and frontend components, please visit the dedicated GitHub repository:

GitHub Repository - OAuth in Go

Conclusion:

Congratulations on successfully implementing OAuth in Go using Gin, Redis, and Goth! We've covered the essential steps for authentication, set up middleware, and even explored the concept of JWT token verification for enhanced security.

If you have any questions, need clarification, or want to discuss any aspect of the implementation, feel free to reach out. Your feedback is valuable, and I'm here to assist you in any way possible.

Stay tuned for the next installment, and happy coding!

How to upload images to Firebase like a Pro?

BHARGAB KALITA — Thu, 11 Feb 2021 13:59:32 +0000

So the first question in which database should you use to upload the images? 🤔, cause we have both Firestore and Realtime Database in the Firebase, but we are not going to upload the images directly to the Firestore or Realtime Database due to limited size for each "Document" which is "1MB" 😬 set by Firebase, hence we are going to upload the image files to the Firebase Storage and then will take the Download URL and add that URL to the Firestore (or Realtime DB) 😋

TLDR;

import Firebase from 'Firebase'
import { v4 as uuid } from "uuid";


  export default function Somefunction(){

    const storage = firebase.storage()
    const db = firebase.firestore()

    const upload = () => {
      const storageRef = storage.ref();
      const Imageref = storageRef.child(
          `images/${uuid()}-${Image.name}` 
        );
      await Imageref.put(Image);
      const ImageFileUrl = await Imageref.getDownloadURL();

      await db.collection("ImageUrls").add({
        image : ImageFileUrl
      })
    }

  return()
  }

PS: we are going to use React for this example

First Step: Setting our main file

Second Step: Writing our upload function

Third Step: Now we have a problem of repetition of the Image References if we upload images of same names or uploading the same image multiple times while will lead to failure in fetching Images with same names, so to fix that we will be using uuid (NPM Package)

Fourth Step: Since all the things are done, now the only thing left is to add the ImageFileUrl to Firestore

Yayyyyy, you have successfully uploaded the image and added it to the Firestore 🎉

Thanks for reading this far, I hope y'all will love this 😊

DEV Community: BHARGAB KALITA

How I Automated My GitHub Profile README With GitHub Actions (And How You Can Automate Anything Too)

Before You Start: The Special GitHub Profile Repository

How I Automated My Profile README to Show My Latest Dev.to Articles

GitHub Actions Workflow (Shows Only 3 Latest Dev.to Articles)

What Else Can You Automate With GitHub Actions?

1. Automatically Add Your Latest YouTube Videos

2. Auto-Update GitHub Stats or Badges

3. Use GitHub Actions as a Free Cron Server

4. Create Your Own Personal Automation Bots

A Simple Automation Template You Can Reuse

Optimizing Performance in Next.js and React.js: Best Practices and Strategies

0. Table of Contents

1. Identifying Performance Bottlenecks

2. Image Optimization

3. Code Splitting and Tree Shaking

4. Server-Side and Client-Side Caching

5. Load Balancing for Scalability

6. Lazy Loading and Intersection Observer

7. Preventing Unnecessary Re-Renders and Managing State Efficiently

8. Server-Side Performance Optimization

9. Purging CSS

10. Inlining Critical CSS

11. Optimizing Fonts and Third-Party Scripts

12. Optimizing Third-Party API Calls

13. Offloading Third-Party Scripts with Partytown

14. Using Next Bundle Analyzer to Identify Bundle Size Issues

15. Rendering Huge Lists with Virtual Lists and Infinite Scroll

16. Continuous Monitoring and Performance Improvements

Conclusion

Note

The Intricacies of MongoDB Aggregation Pipeline: Challenges and Insights from Implementing It with Go

Understanding the Aggregation Pipeline

How the Aggregation Pipeline Works Internally

Challenges of Implementing the Aggregation Pipeline with Go

1. Schema-less Nature of MongoDB

2. Complex Query Construction

3. Debugging and Error Handling

4. Performance Bottlenecks

5. Concurrency Management

Recommendations for Efficient Usage

Lessons Learned

1. Leverage Debugging Tools

2. Optimize Pipeline Order

3. Encapsulate Pipeline Logic

4. Monitor System Resources

Closing Thoughts 💭

The Art of Prefetching and Preloading: Enhancing Web Performance

🎯 Understanding Prefetching and Preloading

🚀 Why You Should Care

🔧 How to Implement Prefetching

🔥 Mastering Preloading

🛠 Best Practices for Both

🧪 Case Study: Boosting Navigation Speed

⚙️ Tools to Measure Impact

🎨 The Art of Balance

Final Thoughts

Optimizing Docker Image Size for Go Applications

The Initial Dockerfile: The Culprit Unveiled

The Unveiling: What Went Wrong?

The Transformation: Streamlining the Dockerfile

The Key Changes

Why I Added CA Certificates

Understanding CGO_ENABLED=0 and Build Flags

Lessons Learned: More Than Just Size

Conclusion

Understanding Parallelism vs Concurrency in Go

Concurrency: Dealing with Lots of Tasks

Parallelism: Doing Multiple Things at Once

Key Differences Between Concurrency and Parallelism

Achieving Concurrency and Parallelism in Go

Real-Life Examples of Concurrency and Parallelism

Conclusion

Rollup.js Made Easy: A Step-by-Step Guide to Building and Publishing NPM Packages

What is Rollup.js?

Key Features of Rollup.js

Setting Up Rollup.js for Building Packages

1. Install Rollup and Basic Dependencies

2. Setting Up TypeScript Configuration

3. Create a Basic Rollup Configuration

Understanding `CGO_ENABLED=0` and Build Flags

1. Testing with `npm pack`

2. Testing with `npm link`

1. Prepare Your `package.json`