DEV Community: chanon-mike

Add local git repository submodule

chanon-mike — Thu, 11 Jan 2024 07:41:32 +0000

When there are a submodule inside the folder and it is not a remote repository.

.
├── submodule
│   └── submodule.md
└── test.md

Inside this root directory, there are a submodule repo.
You can add it by

git submodule add git:submodule submodule

Then .gitmodules will be created at root directory, and here is the content

[submodule "submodule"]
    path = submodule
    url = git:submodule

Vercel Module not found: Can't resolve 'xxx'

chanon-mike — Tue, 09 Jan 2024 02:26:11 +0000

Work in local but not working in vercel

Reason

My exact error Module not found: Can't resolve '@/client/client'
This happened after I change file name from Client to client.

It seems like git is case-insensitive and the changes are not pushed.

Fix

git rm -r --cached .
git add --all .
git commit -a -m "versioning untracked files"
git push

References

Jul 16 '13 Comments: 4 Answers: 22

2143

I have changed a few files name by de-capitalize the first letter, as in Name.jpg to name.jpg. Git does not recognize this changes and I had to delete the files and upload them again. Is there a way that Git can be case-sensitive when checking for changes in file…

Open Full Question

Using Python and ORM (SQLAlchemy) with Google BigQuery

chanon-mike — Thu, 30 Nov 2023 03:35:24 +0000

This is a tutorial for using SQLAlchemy with BigQuery. Normally, you can use Python client for BigQuery for connecting it. However, you can use SQLAlchemy for connecting it to BigQuery too.

Check the official document here!

googleapis / python-bigquery-sqlalchemy

SQLAlchemy dialect for BigQuery

SQLAlchemy Dialect for BigQuery

SQLALchemy Dialects

Quick Start

In order to use this library, you first need to go through the following steps:

Installation

Install this library in a virtualenv using pip. virtualenv is a tool to create isolated Python environments. The basic problem it addresses is one of dependencies and versions, and indirectly permissions.

With virtualenv, it's possible to install this library without needing system install permissions, and without clashing with the installed system dependencies.

Supported Python Versions

Python >= 3.8

Unsupported Python Versions

Python <= 3.7.

Mac/Linux

pip install virtualenv
virtualenv <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install sqlalchemy-bigquery

Windows

pip install virtualenv
virtualenv <your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exe install sqlalchemy-bigquery

Installations when processing large datasets

When handling large datasets, you may see speed increases by also…

View on GitHub

If you want to have overview of BigQuery, try reading this blog!

How to query your data in BigQuery | Google Cloud Blog

Learn how to query datasets in BigQuery using SQL, save and share queries, and create views and materialized views.

cloud.google.com

Installation



pip install sqlalchemy-bigquery

Make a connection



from sqlalchemy import MetaData
from sqlalchemy import create_engine

engine = create_engine(
    'bigquery://',
    credentials_info=credentials,
)
metadata = MetaData()

You can pass credentials info by passing the path or passing the json directly



# Passing with json
credentials = {
    'type': 'xxxx'
    'project_id': 'xxxx'
    # and other more
}
engine = create_engine('bigquery://', credentials_info=credentials)

# Passing with path
engine = create_engine('bigquery://', credentials_path='/path/to/keyfile.json')

You can set default project name and dataset when instantiate the engine



engine = create_engine('bigquery://project_name/dataset_name', credentials_info=credentials)

I recommend you to register the dialect first before having a connection, as I got an error

NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:bigquery



from sqlalchemy.dialects import registry

registry.register('bigquery', 'sqlalchemy_bigquery', 'BigQueryDialect')

Query

Here, I will be using SQLAlchemy syntax for querying without writing SQL.

First, you need to get the table.



from sqlalchemy import Table

def get_table(project_name: str, dataset_name: str, table_name: str) -> Table:
        table = Table(f'{project_name}.{dataset_name}.{table_name}', metadata, autoload_with=engine)
        return table

Then, use that table to query. I create the select method to be able to use that in many query.



from sqlalchemy.engine.row import Row
from sqlalchemy.orm import Session
from sqlalchemy.sql import Selectable

def select(stmt: Selectable) -> list[dict]:
        with Session(engine) as session:
            results: list[Row] = session.execute(stmt).fetchall()
            return [r._asdict() for r in results]

Then, put that all together



from sqlalchemy.sql import select

def select_something() -> list[dict]:
        table = get_table('project_name', 'dataset', 'table')
        stmt = (
            select(
                table.c.id.label('ID'),
                table.c.name.label('Name'),
                table.c.categoryID.label('CategoryID'),
                table.c.sortID.label('SortID'),
            )
            .filter(
                table.c.categoryID == 1,
            )
            .order_by(table.c.id)
        )
        return select(stmt)

Done!

Summary

orm.py



from sqlalchemy import MetaData
from sqlalchemy import Table
from sqlalchemy import create_engine
from sqlalchemy.dialects import registry
from sqlalchemy.engine.row import Row
from sqlalchemy.orm import Session
from sqlalchemy.sql import Selectable


class SQLAlchemyBigQueryDataSource:
    def __init__(self, credentials: dict) -> None:
        registry.register('bigquery', 'sqlalchemy_bigquery', 'BigQueryDialect')
        self.engine = create_engine(
            'bigquery://',
            credentials_info=credentials,
        )
        self.metadata = MetaData()

    def get_table(self, project_name: str, dataset_name: str, table_name: str) -> Table:
        table = Table(f'{project_name}.{dataset_name}.{table_name}', self.metadata, autoload_with=self.engine)
        return table

    def select(self, stmt: Selectable) -> list[dict]:
        with Session(self.engine) as session:
            results: list[Row] = session.execute(stmt).fetchall()
            return [r._asdict() for r in results]

table_name.py



from sqlalchemy import Table
from sqlalchemy.sql import select

from orm import SQLAlchemyBigQueryDataSource


class SomethingBigQueryDataSource:
    def __init__(self, data_source: SQLAlchemyBigQueryDataSource) -> None:
        self.data_source = data_source

    def select_something(self) -> list[dict]:
        table = get_table('project_name', 'dataset', 'table')
        stmt = (
            select(
                table.c.id.label('ID'),
                table.c.name.label('Name'),
                table.c.categoryID.label('CategoryID'),
                table.c.sortID.label('SortID'),
            )
            .filter(
                table.c.categoryID == 1,
            )
            .order_by(table.c.id)
        )
        return select(stmt)

Extract texts and image from Google Slide embedded iframe with Plasmo extensions

chanon-mike — Tue, 28 Nov 2023 02:29:16 +0000

In this blog, I am trying to fetch the text and image from emded google slide iframe using TypeScript.
You can either try to run it as script but here I'm using Plasmo, a React-based extension framework.

I want to make this because of how tiring to take notes for every class in the school. I make it an extension because you can easily add a download button directly through the website without switching tab and running scripts everytime.

Environment setup

Following Plasmo official documentation, by installing the dependencies. If you don't have node installed yet, install it first.

They strongly recommend pnpm for package management, so here we are using pnpm (If you don't have it, follow pnpm doc

I would like to use src directory, so I will include --with-src flag

# Fill out the project description and install dependencies
pnpm create plasmo --with-src
pnpm install

Running development server

pnpm dev

Then, load it in to Google Chrome extension.
Go to Extension management page, then click on load unpacked on the top left corner

Then, select and load build/chrome-mv3-dev folder.

Done!
Now you can focus on developing the extension itself!

Development

Content Scripts

You can create contents directory directly inside src directory to make it run when the website url matches.
Here I will make the file for the downloading scripts named google-slide.ts.

import type { PlasmoCSConfig } from 'plasmo';

export const config: PlasmoCSConfig = {
    matches: ['https://docs.google.com/*']
};

window.addEventListener('load', () => {
    console.log('content script loaded');
});

This script will run when the url start with https://docs.google.com/.

You only need to make the function to extract the text from slide itself! In my usecase, I want to be able to download it as html file. So, I created the function inside the same file.

Caution: Your slide structure may not be the same as mine, so please adjust the code

const downloadSlide = async () => {
    console.log('downloadSlide run');

    let totalPage = '';
    const name = document.title;

    while (true) {
        const pageTitle =
            document
                .querySelector('.punch-viewer-svgpage-a11yelement')
                ?.getAttribute('aria-label')
                ?.replace(/\x0B/g, '<br>') || '';
        const svgContainer = document.querySelector('.punch-viewer-svgpage-svgcontainer:last-child svg') as SVGSVGElement;
        let text = '';

        // Get text in svg
        svgContainer?.querySelectorAll('g').forEach((g) => {
            const label = g.getAttribute('aria-label');
            if (label) {
                text += label + '<br />';
            }
        });

        // Add text to each page
        const page = `<div class="row"><h4>${pageTitle}</h4><div></div>${new XMLSerializer().serializeToString(
            svgContainer
        )}<div>${text}</div></div>`;
        totalPage += page;

        // End the loop if the last slide is reached
        if (
            document.querySelector('.docs-material-menu-button-flat-default-caption').getAttribute('aria-setsize') ==
            document.querySelector('.docs-material-menu-button-flat-default-caption').getAttribute('aria-posinset')
        ) {
            break;
        }

        // Go to the next slide and wait for slide transition
        document.dispatchEvent(new KeyboardEvent('keydown', { keyCode: 39 }));
        await new Promise((resolve) => setTimeout(resolve, 1));
    }

    const html = `<!DOCTYPE html><html><head><title>${name}</title></head><body><main class="container">${totalPage}</main></body></html>`;

    const link = document.createElement('a');
    link.href = URL.createObjectURL(new Blob([html], { type: 'text/html' }));
    link.download = name + '.html';
    link.click();
    window.close();
};

To explain the code, I extract the title of the document for the file name.
Then running while loop to get text from each page of the slide. Here is the part where it extract text

// title
const pageTitle =
    document
        .querySelector('.punch-viewer-svgpage-a11yelement')
        ?.getAttribute('aria-label')
        ?.replace(/\x0B/g, '<br>') || '';

// text container in svg
const svgContainer = document.querySelector('.punch-viewer-svgpage-svgcontainer:last-child svg') as SVGSVGElement;
let text = '';

// Get text in svg
svgContainer?.querySelectorAll('g').forEach((g) => {
    const label = g.getAttribute('aria-label');
    if (label) {
        text += label + '<br />';
    }
});

Then, I would like to have slide image too, so I use new XMLSerializer().serializeToString(svgContainer) for convert svg to image.

I loop this for every page and end it if last slide is reached. Finally, create a hidden button with a link to download the html created.

Now, the only things left is to create condition to be able to download this slide! In my case, I add a parameter download=true for the url. For example https://docs.google.com/presentation/d/e/xxxx/embed?download=true

window.addEventListener('load', () => {
    console.log('content script loaded');

    if (window.location.search.match(/download=true/)) {
        downloadSlide();
    }
});

You can add the parameter directly and download the slide now!

Download Button

I don't want to add parameter everytime, so I gonna add the button for download the slide at the bottom of embeded iframe.

Here, created new file inside src/contents directory as a .tsx file.

src/content/download-slide.tsx

import type { PlasmoCSConfig, PlasmoGetInlineAnchor } from 'plasmo';

export const config: PlasmoCSConfig = {
    matches: ['Your url'] # Change url here
};

export const getInlineAnchor: PlasmoGetInlineAnchor = async () => document.querySelector('.embed-responsive');

const DownloadSlideButton = () => {
    const url = document.querySelector(`.embed-responsive iframe`).getAttribute('src');

    return (
        <>
            <a href={`${url}&download=true`} target="_blank" rel="noopener noreferrer">
                <button>Download as HTML</button>
            </a>
        </>
    );
};

export default DownloadSlideButton;

After reloading your site, you will be able to see the button at the bottom of your embeded slide iframe!

Summary

You are able to create extension to download the slide in HTML! If you want to deploy it to chrome extension store, try following plasmo official documentation.

Below is the code summary

src/contents/google-slide.ts

import type { PlasmoCSConfig } from 'plasmo';

export const config: PlasmoCSConfig = {
    matches: ['https://docs.google.com/*']
};

window.addEventListener('load', () => {
    if (window.location.search.match(/download=true/)) {
        downloadSlide();
    }
});

const downloadSlide = async () => {
    console.log('downloadSlide run');

    let totalPage = '';
    const name = document.title;

    while (true) {
        const pageTitle =
            document
                .querySelector('.punch-viewer-svgpage-a11yelement')
                ?.getAttribute('aria-label')
                ?.replace(/\x0B/g, '<br>') || '';
        const svgContainer = document.querySelector('.punch-viewer-svgpage-svgcontainer:last-child svg') as SVGSVGElement;
        let text = '';

        // Get text in svg
        svgContainer?.querySelectorAll('g').forEach((g) => {
            const label = g.getAttribute('aria-label');
            if (label) {
                text += label + '<br />';
            }
        });

        // Add text to each page
        const page = `<div class="row"><h4>${pageTitle}</h4><div></div>${new XMLSerializer().serializeToString(
            svgContainer
        )}<div>${text}</div></div>`;
        totalPage += page;

        // End the loop if the last slide is reached
        if (
            document.querySelector('.docs-material-menu-button-flat-default-caption').getAttribute('aria-setsize') ==
            document.querySelector('.docs-material-menu-button-flat-default-caption').getAttribute('aria-posinset')
        ) {
            break;
        }

        // Go to the next slide and wait for slide transition
        document.dispatchEvent(new KeyboardEvent('keydown', { keyCode: 39 }));
        await new Promise((resolve) => setTimeout(resolve, 1));
    }

    const html = `<!DOCTYPE html><html><head><title>${name}</title><style>/* Your styles here */</style></head><body><main class="container">${totalPage}</main></body></html>`;

    const link = document.createElement('a');
    link.href = URL.createObjectURL(new Blob([html], { type: 'text/html' }));
    link.download = name + '.html';
    link.click();
    window.close();
};

src/content/download-slide.tsx

import type { PlasmoCSConfig, PlasmoGetInlineAnchor } from 'plasmo';

export const config: PlasmoCSConfig = {
    matches: ['Your url'] # Change url here
};

export const getInlineAnchor: PlasmoGetInlineAnchor = async () => document.querySelector('.embed-responsive');

const DownloadSlideButton = () => {
    const url = document.querySelector(`.embed-responsive iframe`).getAttribute('src');

    return (
        <>
            <a href={`${url}&download=true`} target="_blank" rel="noopener noreferrer">
                <button>Download as HTML</button>
            </a>
        </>
    );
};

export default DownloadSlideButton;