DEV Community: Ivan Yu

Using Django ORM only without web server

Ivan Yu — Sat, 20 Apr 2024 11:04:10 +0000

Django is a great framework, but sometimes you just want to use its ORM without web server function. This blog will introduce how to do it.

1. Step

1.1. Setup environment

Create a virtual environment named ormOnlyEnv and then activate it

python -m venv ormOnlyEnv
ormOnlyEnv\Scripts\activate

Copy the requirement.txt to the same directory as the folder ormOnlyEnv and install it using below command.

pip install -r requirements.txt

Copy manage.py and settings.py to the directory and create directory db. Your project structure should look like as below:

1.2. Create Model class

Create models.py under directory db. Place your own model (data stored to db) in models.py. We use the class Person as an example.

models.py

from django.db import models
from django.utils import timezone

## Create your models here.
class Person(models.Model):

    name = models.CharField(
        max_length=50,
        blank=False
    )
    age = models.IntegerField()
    birth_date = models.DateTimeField(
        blank=False
    )

1.3. Generate database and table

For simplicity, in this example, we use sqlite.

Check if there is any migration script generated and output

python manage.py showmigrations db

db
 (no migrations) --> No script is generated

Generate script and output

python manage.py makemigrations db

Migrations for 'db':
  db\migrations\0001_initial.py
    - Create model Person

Generate table in database and output

python manage.py migrate

Migrations for 'db':
  db\migrations\0001_initial.py
    - Create model Person

1.3.1. Remark

If you have update models.py, remember to run following commands again to synchronize database.

python manage.py makemigrations db
python manage.py migrate

2. Create your own script

Here is the most IMPORTANT point, place the following code at the TOP in the script file to initialize Django ORM in the beginning.

import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')
import django
django.setup()

Save the data to db by calling the model class

import datetime
from db.models import Person

print("Project started!!")
person = Person()
person.name = 'John Doe'
person.age = 30
person.birth_date = datetime.datetime(1963, 2, 5)
person.save()
print("Saved!!")

Whole file main.py

##############################
## Django specific settings (Please this BEFORE import model class)
##############################
import django
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')
django.setup()
##############################

import datetime
from db.models import Person

print("Project started!!")
person = Person()
person.name = 'John Doe'
person.age = 30
person.birth_date = datetime.datetime(1963, 2, 5)
person.save()
print("Saved!!")

Run the files and you can see the record is saved to db

python main.py

Project started!!
Saved!!

3. Git repo

https://github.com/ivanyu199012/16-DjangoUsedOrmOnly/

4. Reason I wrote this blog

Recently, I needed to build a python script to save some records to database. I considered manually creating tables and coding sql insert command, but this would cost a lot of times.

I remembered that in the past, I used Django ORM only without the web server to save records to database, so I decided to use it.

I forgot how to do it, so I spent some time to test and refresh my memory and finally I succeed it. This blog is wrote so that in the future, if I forgot how to do it, I can check this blog and hope it can also help the others.

Thanks for reading.😀

Remember to use break in switch

Ivan Yu — Sun, 14 Apr 2024 04:59:54 +0000

Recently, I introduced a bug because I forget to add break after case in switch. So, I wrote this blog to remind myself and hope that it can help the others not to make same mistake again.

Misunderstanding about `switch`

First, we look at following:

public static void main(String[] args) throws Exception {
  String input = "B";
  switch(input)
  {
    case "A":
      System.out.println("This is A!");
    case "B":
      System.out.println("This is B!");
    case "C":
      System.out.println("This is C!");
    case "D":
      System.out.println("This is D!");
  }
}

At first, I thought the execution order as below:

Since input is "B", it falls into condition case "B" and then print "This is B!".
Since "B" does not match "C" and "D", nothing should be printed out afterward.

Test result not matched

However, the test result does not match my expectation:

This is B!
This is C!
This is D!

According to geeks for geeks explanation for fall through condition in switch,

Fall through condition: This condition occurs in the switch control statement when there is no break keyword mention for the particular case in the switch statement and cause execution of the cases till no break statement occurs or exit from the switch statement.

i.e. When a case is matched, it continues to execute below case statements until:

There is a break OR
Exit from the switch statement

Actual execution:

Correction

Add break for each case, and the problem is resolved.

Output:

This is B!

Script for Easy Git Branch Management

Ivan Yu — Fri, 01 Mar 2024 05:13:37 +0000

This bash script aims to simplify your workflow within Git by offering a convenient way to manage branches.

Setup

Download the script in https://github.com/ivanyu199012/14-gsw/blob/master/gsw.sh
Run source gsw.sh in the terminal of your linux machine,

Feature

1. `gsw` --> Change git branch using no.

Type gsw and it lists all available local branches and ask you to input the branch no., which you would like to switch to. Note that branch marked with * is your current branch.

$ gsw
1.   master
2. * test-branch-1
3.   test-branch-2
4.   test-branch-3
5.   test-branch-4
6.   test-branch-5
Enter branch no.:

Enter the branch no., e.g. 4 for test-branch-3, and press enter. Then it switches to the branch test-branch-3;

$ gsw
1.   master
2. * test-branch-1
3.   test-branch-2
4.   test-branch-3
5.   test-branch-4
6.   test-branch-5
Enter branch no.: 4
Switched to branch 'test-branch-3'

$ git branch
  master
  test-branch-1
  test-branch-2
* test-branch-3
  test-branch-4
  test-branch-5

2. `gsw -m` --> Fast switch back to `master` branch

use gsw -m to switch back to master branch directly

$ gsw -m
Switched to branch 'master'

$ git branch
* master
  test-branch-1
  test-branch-2
  test-branch-3
  test-branch-4
  test-branch-5

3. `gsw -c <branch name>` --> create new branch

use gsw -c <branch name> to create new branch

$ gsw -c test-new-branch
Switched to a new branch 'test-new-branch'

$ git branch
  master
  test-branch-1
  test-branch-2
  test-branch-3
  test-branch-4
  test-branch-5
* test-new-branch

Summary

Command	Description
`gsw` + no.	Change git branch using no.
`gsw -m`	Switch to `master` branch
`gsw -c <branch name>`	Create a branch with name `<branch name>`

Reason I wrote this blog

How do you switch git branch? Most of you may follow below steps:

Use git branch to list all the branches
Find your desired branch and copy the name
Type git checkout or git checkout and then paste the name

Before developed this script, I did above steps a lot of times every day and found it very troublesome. I asked myself:

Why do I type 2 commands and use my mouse to copy the branch name for just switching branch?
Can I switch branch using number instead of name?

As a result, the script was made and hope that it can help you to streamline git branch management.🙂

C++, Implement filter function for refactoring to keep code dry, clean and flexible

Ivan Yu — Sun, 25 Feb 2024 08:59:20 +0000

You may be asked to add a function to get a list of nested items fulfilling certain condition and there is already another function doing similar thing, but with different condition. You can certainly create a new function to achieve this and ignore the existing one, but this may produce a lot of repeated code. So here, we suggest to use filter function and explain the usage with the following example.

Scenario

Example project: https://github.com/ivanyu199012/13-Refactor-FilterFunctions

Current code is as below (filter1.cpp):

PersonVector getPeopleOldThan18(const CityVector &cityVector);

int main()
{
  const CityVector cities = initData();
  std::cout << "----------------------------" << std::endl;
  std::cout << "People larger than 18:" << std::endl;
  PersonVector personVector = getPeopleOldThan18(cities);
  for (auto person : personVector)
  {
    std::cout << person->getName() << ", Age: " << person->getAge() << std::endl;
  }
}

PersonVector getPeopleOldThan18(const CityVector &cityVector)
{
  PersonVector outputPersonVector;
  for (auto city : cityVector)
  {
    for (auto pPerson : city->getCitizens())
    {
      if (pPerson->getAge() > 18)
      {
        outputPersonVector.push_back(pPerson);
      }
    }
  }
  return outputPersonVector;
}

There is an existing function getPeopleOldThan18() to get a vector of person older than 18 years old and each class City instance contains a vector of people.

City data structure(Model.cpp):

new Person("John", 34, 10000), // Name, Age, Income

const CityVector initData()
{
  std::vector<Person *> londonCitizens{new Person("John", 34, 10000),
                                       new Person("Peter", 54, 20000),
                                       new Person("Mary", 17, 3000)};

  std::vector<Person *> beijingCitizens{new Person("Wong Yin", 32, 10000),
                                        new Person("Xiu Ming", 52, 20000),
                                        new Person("Tai Man", 15, 3000)};

  std::vector<Person *> seoulCitizens{new Person("Mr. Pak", 32, 10000),
                                      new Person("Gilbin", 52, 20000),
                                      new Person("YoungSoo", 15, 3000)};

  return CityVector {new City("London", londonCitizens),
                    new City("Beijing", beijingCitizens),
                    new City("Seoul", seoulCitizens)};
}

Requirement: Add a function to retrieve people with income larger than 15000

You can add a new function getPeopleEarnMoreThan15000() and ignore the existing function as below(filter2.cpp):

// New function
PersonVector getPeopleEarnMoreThan15000(const CityVector &cityVector)
{
  PersonVector outputPersonVector;
  for (auto city : cityVector)
  {
    for (auto pPerson : city->getCitizens())
    {
      if (pPerson->getIncome() > 15000)
      {
        outputPersonVector.push_back(pPerson);
      }
    }
  }
  return outputPersonVector;
}

PersonVector getPeopleOldThan18(const CityVector &cityVector)
{
  PersonVector outputPersonVector;
  for (auto city : cityVector)
  {
    for (auto pPerson : city->getCitizens())
    {
      if (pPerson->getAge() > 18)
      {
        outputPersonVector.push_back(pPerson);
      }
    }
  }
  return outputPersonVector;
}

But except the if statements if (pPerson->getAge() > 18) and if (pPerson->getIncome() > 15000), remaining code of 2 functions are the same and a lot of repeated code is produced.

Better solution: Filter function

The concept is to add a new function getPeopleWithFilter() with an extra function parameter personFilter, which function's input is Person and output is boolean, and call the function in the if statement if(personFilter(pPerson)).(filter3.cpp)

PersonVector getPeopleWithFilter(const CityVector &cityVector,
 // Function Parameter
 const std::function<bool(Person *)> &personFilter )
{
  PersonVector outputPersonVector;
  for (auto city : cityVector)
  {
    for (auto pPerson : city->getCitizens())
    {
      // Call the filter function
      if (personFilter(pPerson))
      {
        outputPersonVector.push_back(pPerson);
      }
    }
  }
  return outputPersonVector;
}

Call above function with the existing function getPeopleOldThan18() and new function getPeopleIncomeMoreThan15000().

PersonVector getPeopleIncomeMoreThan15000(const CityVector &cityVector)
{
  std::function<bool(Person *)>
      personFilter = [](const Person *pPerson) -> bool
  {
    return pPerson->getIncome() > 15000;
  };
  // Call the above function
  return getPeopleWithFilter(cityVector, personFilter);
}

PersonVector getPeopleOldThan18(const CityVector &cityVector)
{
  std::function<bool(Person *)>
      personFilter = [](const Person *pPerson) -> bool
  {
    return pPerson->getAge() > 18;
  };
  // Call the above function
  return getPeopleWithFilter(cityVector, personFilter);
}

This code has 2 advantages:

Keep the code dry (reduce repeated code).
Only need to define the filter function and call getPeopleWithFilter() if you need to add another function to retrieve a list of nested item in the future.

Reason I wrote this blog

I tried to share the coding technique I have learnt and used in my work and hope that others can gain advantage from it. Also, I can refer to this blog if I forget the skill in the future.

Maybe some of you notice, I have not wrote any blog for about a year since I have moved to another company from last year and I spend a lot of time to adapt the new environment.

In this company, rather than developing new software, mostly I fix bug, improve and enhance current software. So refactoring code becomes my main task.

I have concluded some important points for refactoring the code.

Understand what the code is about.
Find out why the code is wrote like this.
For fixing bug, discovering root cause is crucial
- If time is limited, you may fix problem with a fast temporary solution first and later and DON'T FORGET, pinpoint root cause
- If root cause identified, this CAN probably lead to more serious problem later
MINIMIZE CODE CHANGE since more change may produce more bugs or errors. In this blog, we only modify the content of the function getPeopleOldThan18() without changing name, input and output.

First glimpse into gRPC through Python (Part 2)

Ivan Yu — Thu, 19 Jan 2023 18:52:00 +0000

1. Previous Post

First glimpse into gRPC through Python (Part 1)

2. Implementation and Examples

After introduced gRPC concept and development flow in previous post, we will show how to implement gRPC in python.

Example Code link (To run the below examples, please go through README.md in the link)

2.1. 4 kinds of service method

There are 4 kinds of service methods in gRPC and the difference between them is to apply steaming or not in request or response.

2.1.1. Simple RPC

This type is just like the http request and response. Client send a single request to server and server reply a single reponse.

Proto file

rpc doSimple ( Request ) returns ( Response ) {}

Server

def doSimple(self, request, context):
    output = f"Hello { request.input }!"
    return Response( output=f"Hello { request.input }!" )

Client

def call_doSimple( self ):
    with grpc.insecure_channel('localhost:50051') as channel:
        stub = SampleServiceStub( channel )
        request = Request( input=self.faker.name() )
        logger.info( f"doSimple client sent: { request.input }" )
        response = stub.doSimple( request )
        logger.info( f"doSimple client received: { response.output }" )

Command line output

[ 2022-07-15 09:43:27,530 ][ INFO ][ call_doSimple ] doSimple client sent: Lindsay Ross
[ 2022-07-15 09:43:27,531 ][ INFO ][ call_doSimple ] doSimple client received: Hello Lindsay Ross!

2.1.2. Response-streaming RPC

For this type, streaming happens in the response. Client sends a single request to server and server returns a streaming of multiple responses.

Proto file

rpc doResponseStreaming( Request ) returns ( stream Response ) {}

Server

def doResponseStreaming(self, request, context):
    faker = Faker()
    name_list = [ *[ faker.name() for i in range( 3 ) ], request.input ]

    for name in name_list:
        time.sleep( 0.5 )
        yield Response( output=name )

Client

def call_doResponseStreaming( self ):
    with grpc.insecure_channel('localhost:50051') as channel:
        stub = SampleServiceStub( channel )
        request = Request( input=self.faker.name() )
        logger.info( f"doResponseStreaming client sent: { request.input }" )
        response_generator = stub.doResponseStreaming( request )
        for response in response_generator:
            logger.info( f"doResponseStreaming client received: { response.output }" )

Command line output

[ 2022-07-15 10:14:27,347 ][ INFO ][ call_doResponseStreaming ] doResponseStreaming client sent: Veronica Good
[ 2022-07-15 10:14:27,971 ][ INFO ][ call_doResponseStreaming ] doResponseStreaming client received: Richard Torres
[ 2022-07-15 10:14:28,472 ][ INFO ][ call_doResponseStreaming ] doResponseStreaming client received: Monica Russo
[ 2022-07-15 10:14:28,985 ][ INFO ][ call_doResponseStreaming ] doResponseStreaming client received: Sean Lane
[ 2022-07-15 10:14:29,498 ][ INFO ][ call_doResponseStreaming ] doResponseStreaming client received: Veronica Good

2.1.3. Request-streaming RPC

Similar to the above type, but the streaming happens in the request. Client sends a streaming of multiple requests to server and server returns a single response.

Proto file

rpc doRequestStreaming ( stream Request ) returns ( Response ) {}

Server

def doRequestStreaming(self, request_iterator, context):
    result_list = []
    for request in request_iterator:
        result_list.append( request.input.upper() )

    return Response( output=", ".join( result_list ) )

Client

def call_doRequestStreaming( self ):

    def get_fake_name_generator():
        faker = Faker()
        for _ in range( 10 ):
            time.sleep( 0.5 )
            name = faker.name()
            logger.info( f"doRequestStreaming client sent: { name }." )
            yield Request( input=name )

    with grpc.insecure_channel('localhost:50051') as channel:
        stub = SampleServiceStub( channel )
        request = get_fake_name_generator()
        response = stub.doRequestStreaming( request )
        logger.info( f"doRequestStreaming client received: { response.output }" )

Command line output

[ 2022-07-15 10:21:08,058 ][ INFO ][ get_fake_name_generator ] doRequestStreaming client sent: Courtney Hammond.
[ 2022-07-15 10:21:08,562 ][ INFO ][ get_fake_name_generator ] doRequestStreaming client sent: James Petersen.
[ 2022-07-15 10:21:09,070 ][ INFO ][ get_fake_name_generator ] doRequestStreaming client sent: Tom Anderson.
[ 2022-07-15 10:21:09,073 ][ INFO ][ call_doRequestStreaming ] doRequestStreaming client received: COURTNEY HAMMOND, JAMES PETERSEN, TOM ANDERSON

2.1.4. Bidirectionally-streaming RPC

As you may already guess the meaning from the name, the streaming happens in both request and response.

Proto file

rpc doBidirectional ( stream Request ) returns ( stream Response ) {}

Server

def doBidirectional(self, request_iterator, context):
    for request in request_iterator:
        yield Response( output=request.input + " is excellent!" )

Client

def call_doBidirectional( self ):

    def get_fake_name_generator():
        faker = Faker()
        for _ in range( 3 ):
            time.sleep( 0.5 )
            name = faker.name()
            logger.info( f"doRequestStreaming client sent: { name }." )
            yield Request( input=name )

    with grpc.insecure_channel('localhost:50051') as channel:
        stub = SampleServiceStub( channel )
        request = get_fake_name_generator()
        response_generator = stub.doBidirectional( request )
        for response in response_generator:
            logger.info( f"doBidirectional client received: { response.output }" )

Command line output

[ 2022-07-15 10:41:11,994 ][ INFO ][ get_fake_name_generator ] doRequestStreaming client sent: Sherry Hanson.
[ 2022-07-15 10:41:11,996 ][ INFO ][ call_doBidirectional ] doBidirectional client received: Sherry Hanson is excellent!
[ 2022-07-15 10:41:12,497 ][ INFO ][ get_fake_name_generator ] doRequestStreaming client sent: Danielle Jones.
[ 2022-07-15 10:41:12,499 ][ INFO ][ call_doBidirectional ] doBidirectional client received: Danielle Jones is excellent!
[ 2022-07-15 10:41:12,999 ][ INFO ][ get_fake_name_generator ] doRequestStreaming client sent: Alexis Goodwin.
[ 2022-07-15 10:41:13,001 ][ INFO ][ call_doBidirectional ] doBidirectional client received: Alexis Goodwin is excellent!

2.2. Special Data Types and default value

gRPC can handles some speical data types, like date time, list or map, but need to pay some attention to avoid bug.

2.2.1. Date Time

gRPC use timestamp to handle date time and it uses timezone.

In proto file

google.protobuf.Timestamp date = 1;

You need to include timezone in the input date time in order to transfer the data correctly to the server side.

Below is the output send the date WITHOUT timezone from client to server side:

[ 2022-07-15 11:04:52,633 ][ INFO ][ call_doSpecialDataType ] doSpecialDataType client sent: request.date=2022-07-15 11:04:52
[ 2022-07-15 11:04:52,654 ][ INFO ][ doSpecialDataType ] doSpecialDataType Server received: request.date=2022-07-15 20:04:52

We can see there are 9 hours difference between client and server since gRPC regards the date time without timezone as UTC time. When the date input received by the server, it automatically add 9 hours as we are in UTC+09:00 region.

2.2.2. List

gRPC can handle list datatype by simply adding repeated in front of the field in proto file.

repeated string names = 1;

2.2.3. Map

gRPC can handle map datatype by defining map as the type and specify key and value type in proto file.

map<string, string> name2phoneNumMap = 3;

2.2.4. Default value

According to this link, it mentioned that

if a scalar message field is set to its default, the value will not be serialized on the wire.

We will explain the meaning in the following example:

proto file

repeated CardInfo cardInfos = 4;

message CardInfo {
    string name = 1;
    int32 numberOfCreditCard = 2;
}

We have a list of cardInfo and each card info contains an integer field called numberOfCreditCard. In the response, we set numberOfCreditCard of last CardInfo of the cardInfo list to be 0.

Server

cardInfos = request.cardInfos
cardInfos.append( CardInfo( name="Boris Lee", numberOfCreditCard=0 ) )

Command line output

[ 2022-07-15 11:21:39,200 ][ INFO ][ call_doSpecialDataType ] doSpecialDataType client received:  response.cardInfos= [name: "Katherine Soto"
numberOfCreditCard: 1
, name: "Kerry Powell"
numberOfCreditCard: 1
, name: "Mrs. Christina Hicks DDS"
numberOfCreditCard: 1
, name: "Boris Lee" <- No "numberOfCreditCard"
]

We can see that Boris Lee does NOT have numberOfCreditCard. Since 0 is regarded as default value, it will not be serialized on the wire and transfer back to client.

To solve this problem, we need to add optional in front of the field in proto file.

optional int32 numberOfCreditCard = 2;

Generate the code and run the program again, you can see the zero appeared.

, name: "Boris Lee"
numberOfCreditCard: 0
]

3. Unit test

We can use python native unit test framework to write unit test for gRPC. This is crucial as you do not need to switch on/off the gRPC server again and again whenever there is change in your code in order to test your code manually.

3.1. Create test server

First, you need to create a test server for receiving gRPC call in the setUp method so that whenever the test method is called, it will first set up the test server.

class SampleServiceTest(unittest.TestCase):

    def setUp(self):
        logger.info( f"=== Method: { self._testMethodName } =======" )
        servicers = {
            sample_service_pb2.DESCRIPTOR.services_by_name['SampleService']: SampleService()
        }

        self.test_server = grpc_testing.server_from_dictionary(
            servicers, grpc_testing.strict_real_time())

3.2. Create test method

Next, you need to create at least one test method for each gRPC method you want to test.

def test_doSimple(self):
    faker = Faker()
    request = sample_service_pb2.Request( input=faker.name() )

    doSimple_method = self.test_server.invoke_unary_unary(
        method_descriptor=(sample_service_pb2.DESCRIPTOR
            .services_by_name['SampleService']
            .methods_by_name['doSimple']),
        invocation_metadata={},
        request=request, timeout=1)

    response, _, code, _ = doSimple_method.termination()
    self.assertEqual( code, grpc.StatusCode.OK )
    self.assertEqual( response.output, f"Hello { request.input }!" )

3.3. Run test

Finally, run your test in the main thread

if __name__ == '__main__':
    unittest.main()

For detail, take a look in sample_service_test.py

4. Reason I wrote this series of blogs

I wrote this blog as I found that when I first learnt about gRPC, it is a little bit difficult to grasp the development method in a fast manner. There are a lot of built-in examples in the gRPC site, but it is a little bit too complicate for a beginner.

At first, what I wanted to know the most are:

1. How can I make a new method?
2. How can I call the method in a microservice through gRPC?
3. How can I handle the input and output?
4. How can I do the unit test?

There are resources in Internet, but they are either scattered around or they used a lot of words to explain without a diagram. As a result, I decided to write this blog to present gRPC (using Python) in a simple and practical way so newcomer can pick up the concept easily and then start development immediately.

First glimpse into gRPC through Python (Part 1)

Ivan Yu — Thu, 12 Jan 2023 16:45:00 +0000

1. What is gRPC and the purpose of it?

RPC (Remote Procedure Call) is a protocol that allows two programs to communicate with each other, those programs can locate in the same computer or in different computers.

gRPC(Google Remote Procedure Call) is a modern open source high performance Remote Procedure Call (RPC) framework, which is typically used in development of microserivces.

gRPC can be implmented by various programming languages, and in this blog, we will give the examples in Python.

2. Why do we learn it?

Large backend system development is shifting from monolith to microservice architecture. gRPC is one of the common ways to implement the microservice architecture.

Based on the graph below, there is more than 1 millions daily downloads for the package grpcio, which is used by Python for gRPC development. (Python alone!!😱)

Ref: Pypl Stats: grpcio download rate

3. Difference between gRPC and REST

You may say that we have REST to handle the communication between services, why do we need to learn one more new framework to waste our time?🤔

Compare gRPC and REST:

Ref: gRPC vs REST: comparing APIs architectural styles

I would like to highlight few points:

If data transmission speed plays an important role in the consideration of choosing framework, gRPC should be selected instead of REST.

gRPC used Protocol Buffer (binary data) for data transmisstion, which is more light-weight and faster compared to JSON or XML used by REST
If streaming communication matters to your project, you should think about gRPC
Note that gRPC has a limited support for browser and it may not be suitable to be used as a API open to the world. gRPC is mainly used for internal/private system.

4. Development Process Explanaiton

Development process of gRPC is as follows:

Create a proto file and define function, input and output in it
Generate pb2 .py and pb2_grpc .py files from the proto file by command
Create a service .py to implement the Servicer interface in pb2_grpc .py
Start service in the main function in service .py
If there is any update in proto file, you MUST genearte pb2 .py and pb2_grpc .py files again
Most of the use case is to use proto file to create a client object to send request to server. If the client is the same language as the server, then client can use Stub class in pb2_grpc .py file to send request.

For detail implementation steps, please refer to the Example Code, Quick Start and Python Basic Tutorial

5. Related Post & Github Link

Electron + Django ( Part 2 ), package it to production

Ivan Yu — Thu, 05 Jan 2023 15:50:00 +0000

1. Introduction & POC

How do we package the electron app with django? you may be more eager to know the answer if you have completed reading "Electron + Django, desktop app integrate JavaScript & Python".

In this blog, I would like to explain the package process in detail and discuss what you may need to pay attention during the process. 😎

Let's show you the final result first:

In the video, we can see:

Open the app by clicking exe file
Testing the app to see if it works
In the task manager, you can see there are 5 processes running

The one on the top is the python app and the remainings are the electron app

2. Prerequisites

We will base on this example to show the steps to package the app. You may follow the README to build up the project first.

3. Package Django app

We will use the pyinstaller to package Django app.

Activate the environment and install pyinstaller

pip install pyinstaller

Add settings folder under edtwExample

cd python\edtwExample\edtwExample
mkdir settings

Copy the settings.py file into the folder and rename to dev.py and prod.py, remove the original settings.py

copy settings.py settings\dev.py
copy settings.py settings\prod.py
del settings.py

Change the following configuration in prod.py
- Debug, True > False
- ALLOWED_HOSTS, [] > [ '127.0.0.1', 'localhost' ]
- Add 'edtwExampleAPI' in INSTALLED_APPS
- DATABASES, BASE_DIR / 'db.sqlite3' > BASE_DIR.parent / 'db.sqlite3'

## Django production configuration
DEBUG = False

## Only allow localhost to connect to Django apps
ALLOWED_HOSTS = [ '127.0.0.1', 'localhost' ]

## Adding edtwExampleAPI in INSTALLED_APPS to acknowledge pyinstaller to include it during the build
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    'edtwExampleAPI',
]

## Since we move the setting file into the folder,
## sqlite db file is one level higher than BASE_DIR
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': BASE_DIR.parent / 'db.sqlite3',
    }
}

( We will use the production configuration to package our app )

Based on the docs, create __init__.py in settings folder
Paste the following code in __init__.py

from .prod import *

The folder structure will look like this:

Go back to the edtwExample and build Django app with the following command:

cd ../..
pyinstaller --name=edtwExample edtwExample\manage.py --noconfirm

If the build complete successfully, the message will be shown:
Go to dist\edtwExample and run the following command

cd dist\edtwExample
edtwExample.exe runserver --settings=edtwExample.settings.prod --noreload

(It seems that edtwExample.exe wraps up the whole python virtual environment and the manage.py)

If everything is fine, it will show the following:

Please make sure that the app is using edtwExample.settings.prod

Then we have finished our first step. 😊

3.1. Remark

There are few things I would like to point out in this step.

3.1.1. Server Error (500)

After you finish the first step and try to browse the API (http://127.0.0.1:8000/edtwExampleAPI/get_val_from/?input=AAAAcccc), you may see the following:

This DOES NOT mean that there is any error. After building Django app by the pyinstaller, the exe app will block browser from accessing the API.

If you would like to test the API, please use Postman or other API tools

3.1.2. TemplateDoesNotExit: DEBUG = True

If we set DEBUG = True and build the app, when we browse the url http://127.0.0.1:8000/edtwExampleAPI/get_val_from/?input=AAAAcccc, the following error may be shown:

The pyinstaller assumes DEBUG = False and does not include any html, css or js file into the exe app

3.1.3. ModuleNotFoundError: No module named XXX

After building the app and run the exe file, the error ModuleNotFoundError may occur

ModuleNotFoundError: No module named 'edtwExampleAPI'

One of the reason is that you may not include the required module in INSTALLED_APPS under configuration prod.py.

3.1.4. Pyinstaller rather than copy virutalenv

You may ask why don't we just copy virtual environment to the new PC instead of using pyinstaller?

I tried that but when I activated the environment, it showed that there was no package installed. 😢 Here is a rather clear explanation.

Link: Create a copy of virtualenv locally without pip install

So I decide to use pyinstaller instead.

4. Package Electron app

After generated the Django exe app, we will package the Electron app

Go back to the base folder and run the following.

cd ..\..\..
npm run package

The success output will be shown as below.

Go to out\edtwexample-win32-x64 and run edtwexample.exe
When you test the app, you may see the following error:

This is because we did not copy the Django exe app to the package folder and We will fix this error in the next step.

5. Include Django exe in Electron package

5.1. Copy Django exe app to Electron package folder

Based on this link, in package.json, add the following afterExtract configuration

"afterExtract": [
    "./src/build/afterExtract.js"
]

In package.json,

  "license": "MIT",
  "config": {
    "forge": {
      "packagerConfig": {
            "afterExtract": [
                "./src/build/afterExtract.js"
            ]
      },

Create a file ./src/build/afterExtract.js and paste the following code into the file

// eslint-disable-next-line @typescript-eslint/no-var-requires
const fs = require('fs-extra');
// eslint-disable-next-line @typescript-eslint/no-var-requires
const path = require('path');

module.exports = function( extractPath, electronVersion, platform, arch, done )
{
    console.log({ extractPath });
    fs.copy('./python/dist/edtwExample', path.join( extractPath, 'python' ), () => {

        console.log('Finished Copy Python Folder');
        done();
    } );
 }

The function is to copy our Django exe app to the Electron package folder.

5.2. Start the Django exe app during the electron startup process.

In index.ts, first define a variable DJANGO_CHILD_PROCESS

let DJANGO_CHILD_PROCESS: ChildProcessWithoutNullStreams = null;

Create 2 functions spawnDjango and isDevelopmentEnv

const spawnDjango = () =>
{
    if ( isDevelopmentEnv() )
    {
        return spawn(`python\\edtwExampleEnv\\Scripts\\python.exe`,
        ['python\\edtwExample\\manage.py', 'runserver', '--noreload'], {
            shell: true,
        });
    }
    return spawn(`cd python && edtwExample.exe runserver --settings=edtwExample.settings.prod --noreload`,  {
        shell: true,
    });
}

const isDevelopmentEnv = () => {
    console.log( `NODE_ENV=${ process.env.NODE_ENV }` )
    return process.env.NODE_ENV == 'development'
}

Call spawnDjango in function startDjangoServer and change it as follow:

const startDjangoServer = () =>
{
    DJANGO_CHILD_PROCESS = spawnDjango();
    DJANGO_CHILD_PROCESS.stdout.on('data', data =>
    {
        console.log(`stdout:\n${data}`);
    });
    DJANGO_CHILD_PROCESS.stderr.on('data', data =>
    {
        console.log(`stderr: ${data}`);
    });
    DJANGO_CHILD_PROCESS.on('error', (error) =>
    {
        console.log(`error: ${error.message}`);
    });
    DJANGO_CHILD_PROCESS.on('close', (code) =>
    {
        console.log(`child process exited with code ${code}`);
    });
    DJANGO_CHILD_PROCESS.on('message', (message) =>
    {
        console.log(`stdout:\n${message}`);
    });
    return DJANGO_CHILD_PROCESS;
}

We only need to start the Django exe app in production but not in development.

5.3. Skip open dev tool in production

Create the following new function.

const openDevTools = ( mainWindow : BrowserWindow ) => {

    if ( isDevelopmentEnv() )
    {
        mainWindow.webContents.openDevTools();
    }
}

Call it during the createWindow method.

const createWindow = (): void => {

    ...
    // Open the DevTools.
    openDevTools( mainWindow );
};

5.4. A complete overlook

Here is the complete index.ts

import { app, BrowserWindow } from 'electron';
import { spawn, ChildProcessWithoutNullStreams } from 'child_process';

declare const MAIN_WINDOW_WEBPACK_ENTRY: string;
let DJANGO_CHILD_PROCESS: ChildProcessWithoutNullStreams = null;

if (require('electron-squirrel-startup')) {
    // eslint-disable-line global-require
    app.quit();
}

const createWindow = (): void =>
{
    startDjangoServer();

    // Create the browser window.
    const mainWindow = new BrowserWindow({
        height: 600,
        width: 800,
    });

    mainWindow.webContents.session.webRequest.onBeforeSendHeaders(
        (details, callback) =>
        {
            const { requestHeaders } = details;
            UpsertKeyValue(requestHeaders, 'Access-Control-Allow-Origin', ['*']);
            callback({ requestHeaders });
        },
    );

    mainWindow.webContents.session.webRequest.onHeadersReceived((details, callback) =>
    {
        const { responseHeaders } = details;
        UpsertKeyValue(responseHeaders, 'Access-Control-Allow-Origin', ['*']);
        UpsertKeyValue(responseHeaders, 'Access-Control-Allow-Headers', ['*']);
        callback({
            responseHeaders,
        });
    });

    // and load the index.html of the app.
    mainWindow.loadURL(MAIN_WINDOW_WEBPACK_ENTRY);

    // Open the DevTools.
    openDevTools(mainWindow);
};

const UpsertKeyValue = (obj: any, keyToChange: string, value: string[]) =>
{
    const keyToChangeLower = keyToChange.toLowerCase();
    for (const key of Object.keys(obj)) {
        if (key.toLowerCase() === keyToChangeLower) {
            obj[key] = value;
            return;
        }
    }
    obj[keyToChange] = value;
}

const startDjangoServer = () =>
{
    DJANGO_CHILD_PROCESS = spawnDjango();
    DJANGO_CHILD_PROCESS.stdout.on('data', data =>
    {
        console.log(`stdout:\n${data}`);
    });
    DJANGO_CHILD_PROCESS.stderr.on('data', data =>
    {
        console.log(`stderr: ${data}`);
    });
    DJANGO_CHILD_PROCESS.on('error', (error) =>
    {
        console.log(`error: ${error.message}`);
    });
    DJANGO_CHILD_PROCESS.on('close', (code) =>
    {
        console.log(`child process exited with code ${code}`);
    });
    DJANGO_CHILD_PROCESS.on('message', (message) =>
    {
        console.log(`stdout:\n${message}`);
    });
    return DJANGO_CHILD_PROCESS;
}

const spawnDjango = () =>
{
    if (isDevelopmentEnv()) {
        return spawn(`python\\edtwExampleEnv\\Scripts\\python.exe`,
            ['python\\edtwExample\\manage.py', 'runserver', '--noreload'], {
            shell: true,
        });
    }
    return spawn(`cd python && edtwExample.exe runserver --settings=edtwExample.settings.prod --noreload`, {
        shell: true,
    });
}

const openDevTools = (mainWindow: BrowserWindow) =>
{

    if (isDevelopmentEnv()) {
        mainWindow.webContents.openDevTools();
    }
}

const isDevelopmentEnv = () =>
{
    console.log(`NODE_ENV=${process.env.NODE_ENV}`)
    return process.env.NODE_ENV == 'development'
}

app.on('ready', createWindow);

app.on('window-all-closed', () =>
{
    if (process.platform !== 'darwin') {
        app.quit();
    }
});

app.on('activate', () =>
{
    if (BrowserWindow.getAllWindows().length === 0) {
        createWindow();
    }
});

5.5. Package and run

Package Electron app and run again. you should see the app running smoothly. 🎉

npm run package
cd out\edtwexample-win32-x64\
edtwexample.exe

6. Close the Django exe app when the window is closed

You may notice that Django exe process is still running even you close the app windows

We need to tell the app to kill the process once the window is close.

First, we will install tree-kill package

npm install tree-kill

Then add the following code in index.ts

app.on('before-quit', async function ()
{
    // Kill python process when the window is closed
    kill( DJANGO_CHILD_PROCESS.pid );
});

Add the line kill( DJANGO_CHILD_PROCESS.pid ) also in window-all-closed

app.on('window-all-closed', () => {
    if (process.platform !== 'darwin') {
        app.quit();
    }
    kill( DJANGO_CHILD_PROCESS.pid );
});

Package the electron app again and the problem should be fixed. 👏👏

6.1. Reason behind it

6.1.1. Django exe was spawned by shell

The Django exe process is spawned with shell: true option, which means that the process is started by the cmd rather than the exe file directly.

In index.ts

spawn(`cd python && edtwExample.exe runserver --settings=edtwExample.settings.prod --noreload`, {
        shell: true,
    });

When we close the window, we only close the shell but the process is still running.

As a result, we need to kill the process in the close windows event listener.

Explanation and solution: link.

6.1.2. Kill the process 2 event listeners

We need to kill the process in below BOTH event listeners.

window-all-closed
before-quit

I tried to include this line kill( DJANGO_CHILD_PROCESS.pid ) in either one event only and Django process is not killed even the app window is closed.

7. Source Code

6.-Package_Electron_n_django_app

8. Reason of writing this blog

After writing the blog "Electron + Django, desktop app integrate JavaScript & Python", I think that packaging an electron app with django app is just a simple task by running one or two command, but I was wrong 😢.

When I was packaging the app, I did a lot of google searches to fix the problems emerged during the process and this was a hard time 😑 for me.

Also, during the search, I noticed that there was lack of an organized approach to explain the whole packaging process and that's why I wrote this blog.

Electron + Django ( Part 1 ), desktop app integrate JavaScript & Python

Ivan Yu — Thu, 29 Dec 2022 15:04:00 +0000

Introduction & Concept

Building a desktop app from Electron and Django is actually building a web application, which uses Electron as the front-end by a local browser and Django as the backend.

Then, why do we need to build the desktop application instead of web application?🤔

Because:

desktop app can run offline
desktop app can access client PC low-level API (e.g. file system 📁)

POC (Proof-Of-Concept)

We will build an simple deskop application using Electron, TypeScript, Webpack and Django.

As shown above,

User first inputs a text.
The input is passed to Django web server and return output, which combined the input, date and time and some other text.
Finally, the output will be shown in the window

0. Prerequisites

Assume that you have installed:

Node js, v14.19.1
python, 3.9.10
virtualenv, 20.13.1

1. Setup Electron

We will use Electron Forge TypeScript + Webpack template to create our desktop application.

Run the following command in command window:

npx create-electron-app edtwExample --template=typescript-webpack

We named tha example edtw, whick stands for Electron, Django, TypScript, Webpack

After running command, you should see the output of the command window:

File Structure:

Run npm run start insides edtwExample folder and the following window should be popped up

2. Setup Django

Create a folder called python in edtwExample folder

mkdir python
cd python

Create a virtual environment and activate it

virtualenv edtwExampleEnv
edtwExampleEnv\Scripts\activate

Install Django and Django REST framework (with the version)

pip install django==4.0.3 djangorestframework==3.13.1

Initiate Django project

django-admin startproject edtwExample

Here is the result file structure:

Run Django app by the following command

python manage.py runserver

Open 127.0.0.1:8000 in browser and you should see the following:

3. Start Django app when the electron start (using spawn)

In order to do so, we create a startDjangoServer method in index.ts that use spawn to run django runserver command

import { spawn } from 'child_process';

const startDjangoServer = () =>
{
    const djangoBackend = spawn(`python\\edtwExampleEnv\\Scripts\\python.exe`,
        ['python\\edtwExample\\manage.py', 'runserver', '--noreload']);
    djangoBackend.stdout.on('data', data =>
    {
        console.log(`stdout:\n${data}`);
    });
    djangoBackend.stderr.on('data', data =>
    {
        console.log(`stderr: ${data}`);
    });
    djangoBackend.on('error', (error) =>
    {
        console.log(`error: ${error.message}`);
    });
    djangoBackend.on('close', (code) =>
    {
        console.log(`child process exited with code ${code}`);
    });
    djangoBackend.on('message', (message) =>
    {
        console.log(`message:\n${message}`);
    });
    return djangoBackend;
}

The following script calls cmd to run a new process with the command python\edtwExampleEnv\Scripts\python.exe with the arguments ['python\\edtwExample\\manage.py', 'runserver', '--noreload'].

const djangoBackend = spawn(`python\\edtwExampleEnv\\Scripts\\python.exe`,
        ['python\\edtwExample\\manage.py', 'runserver', '--noreload']);

The following script log the output of the django process

djangoBackend.stdout.on('data', data =>
{
    log.info(`stdout:\n${data}`);
});
djangoBackend.stderr.on('data', data =>
{
    log.error(`stderr: ${data}`);
});
djangoBackend.on('error', (error) =>
{
    log.error(`error: ${error.message}`);
});
djangoBackend.on('close', (code) =>
{
    log.info(`child process exited with code ${code}`);
});
djangoBackend.on('message', (message) =>
{
    log.info(`stdout:\n${message}`);
});

We call the startDjangoServer method in the createWindow method.

const createWindow = (): void => {

    startDjangoServer();

  // Create the browser window.
  const mainWindow = new BrowserWindow({
    height: 600,
    width: 800,
  });

  // and load the index.html of the app.
  mainWindow.loadURL(MAIN_WINDOW_WEBPACK_ENTRY);

  // Open the DevTools.
  mainWindow.webContents.openDevTools();
};

Run npm run start and open the task manager, you should be able to see python process
If you close the app window, the python process will stop

Note

The argument --noreload in ['python\\edtwExample\\manage.py', 'runserver', '--noreload'] MUST BE INCLUDED to prevent django application started twice.

If --noreload is omitted, you will have 4 python instance running in the background.

Even you close application window, there are 2 python instance left and you can still access django site.

4. Construct API method in Django

Add rest_framework in INSTALLED_APPS in settings.py

Running the below command in command window to create an app named edtwExampleAPI

python manage.py startapp edtwExampleAPI

You should see the below file structure:

Add path('', include('edtwExampleAPI.urls')), in edtwExample\urls.py

Create urls.py under the folder edtwExampleAPI and paste the following content there

from django.urls import include, path
from rest_framework import routers
from . import views

router = routers.DefaultRouter()
router.register( 'edtwExampleAPI', views.EdtwViewSet, basename='edtwExampleAPI' )

## Wire up our API using automatic URL routing.
## Additionally, we include login URLs for the browsable API.
urlpatterns = [
    path('', include(router.urls)),
]

In views.py, copy and paste the following content

from datetime import datetime
from rest_framework import viewsets
from rest_framework import status
from rest_framework.decorators import action
from rest_framework.response import Response

class EdtwViewSet(viewsets.ViewSet):
    ## Create your views here.
    @action(methods=['GET'],  detail=False, name='Get Value from input' )
    def get_val_from( self, request ):

        input = request.GET[ 'input' ]

        return Response( status=status.HTTP_200_OK,
                data=f"[{ datetime.now() }] input= { input }, value from Django" )

Restart Django web server and go to http://127.0.0.1:8000/edtwExampleAPI/get_val_from/?input=This+is+an+input+text.

The following will be shown:

5. Call Django API from Electron

Copy and paste the following code to index.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8" />
    <title>Hello World!</title>
  </head>
  <body>
    <h1>💖 Hello World!</h1>
    <p>Welcome to your Electron application.</p>
    <input id="input_text" type="text"></body>
    <button id="btn_get_val_from_django" >Get Value From Django</button>
    <h2>Output</h2>
    <p id="p_output"></p>
  </body>
</html>

Copy and paste the following code to renderer.ts

import axios from 'axios';
import './index.css';

const btnGetValFromDjango = document.getElementById('btn_get_val_from_django');
btnGetValFromDjango.onclick = async () => {

    const res = await axios.get("http://127.0.0.1:8000/edtwExampleAPI/get_val_from/", { params: {
        input: ( document.getElementById('input_text') as HTMLInputElement ).value
    } });

    const result = res.data;
    document.getElementById('p_output').innerHTML = result;
};

console.log('👋 This message is being logged by "renderer.js", included via webpack');

Our logic is finished. In here, there will be 2 errors coming up. Do not worry, I will let you know how to solve it. 😊

If you test the app, it will show the following error.

The above error was due to the content security policy. We will fix it by adding devContentSecurityPolicy in package.json and restart the application. (See this for more info.)

"@electron-forge/plugin-webpack",
{
    "mainConfig": "./webpack.main.config.js",
    "devContentSecurityPolicy": "connect-src 'self' http://127.0.0.1:8000 'unsafe-eval'",
    "renderer": {
        "config": "./webpack.renderer.config.js",
        "entryPoints": [
            {
                "html": "./src/index.html",
                "js": "./src/renderer.ts",
                "name": "main_window"
            }
        ]
    }
}

After that, if you try the application again, there will be another error.

This is due to the common CORS policy. We choose the fix introduced in here.

The concept is to replace the header before browser check the origin.

Add the following method in index.ts

const UpsertKeyValue = (obj : any, keyToChange : string, value : string[]) => {
    const keyToChangeLower = keyToChange.toLowerCase();
    for (const key of Object.keys(obj)) {
        if (key.toLowerCase() === keyToChangeLower) {
        // Reassign old key
        obj[key] = value;
        // Done
        return;
        }
    }
    // Insert at end instead
    obj[keyToChange] = value;
}

Change createWindow method as follow in index.ts

const createWindow = (): void => {

    startDjangoServer();

    // Create the browser window.
    const mainWindow = new BrowserWindow({
        height: 600,
        width: 800,
    });

    mainWindow.webContents.session.webRequest.onBeforeSendHeaders(
        (details, callback) => {
            const { requestHeaders } = details;
            UpsertKeyValue(requestHeaders, 'Access-Control-Allow-Origin', ['*']);
            callback({ requestHeaders });
        },
    );

    mainWindow.webContents.session.webRequest.onHeadersReceived((details, callback) => {
        const { responseHeaders } = details;
        UpsertKeyValue(responseHeaders, 'Access-Control-Allow-Origin', ['*']);
        UpsertKeyValue(responseHeaders, 'Access-Control-Allow-Headers', ['*']);
        callback({
            responseHeaders,
        });
    });

    // and load the index.html of the app.
    mainWindow.loadURL(MAIN_WINDOW_WEBPACK_ENTRY);

    // Open the DevTools.
    mainWindow.webContents.openDevTools();
};

Restart the application and Done! 🎉👏

Source code

You may check for the source code for more information.

Reason for building a Desktop app using Electron and Django

Initially, I would like to build an desktop app and focus on using Python to build backend or business logic so I first search for Python's Desktop frameworks. However, they are either

Not very user friendly
Lack of nice-looking UI framework
Not free of charge

Due to the above 3 reasons, I expected I may need to spend a lot of time to develop if I choose them. (Here is a good ref.)

As I am a web developer, I asked myself, can I use as much as what I already know (e.g. JavaScript & Python) to build a Desktop app?

That is why Electron get into my sight.

Electron + Django is a good approach if
- You already have a web app using Django as the backend and JavaScript for the frontend and you would like to convert it to Desktop app
- You want to be more excellent on Django and develop the frontend using your favorite frontend library (e.g. React, Angular or vue e.t.c)
It may not be an nice approach if
- You build a desktop app from nothing. (Electron + Django takes relative more time to set up and you need to maintain 2 programming languages).
  
  For this case, I suggest to use Electron itself combined with TypeScript as there is already existing template to handle this. Also, Electron itself can access PC low-level API to fulfill your need

Sharepoint Signature: PowerApp PenInput

Ivan Yu — Thu, 22 Dec 2022 16:00:00 +0000

Introduction & Concept

Sharepoint is an awesome tool to share knowledge within a team. Often, we would like to enhance its functionality to fulfill our needs.

And this time, we would like to add a signature field ✍ in SharePoint.

You may say it sounds great, but how do we do it? 🤔

Application Demo

Well, first look at the result application:

When a user first creates a record, user can see a signature box and sign in it. The user can also click "Reset Signature" to clear the original one and sign again

When user view the record again, user can see the signature

When user edit the record, user can click "Modify Signature" to sign again

Steps

1. Add a multiple lines of text column

So, create a multiple lines of text column and named it "imageData"
in the SharePoint list. This is for saving signature image byte.

(There is also image type column in SharePoint, but this type of column CANNOT BE UPDATED IN POWER APPS.)

2. Add PenInput & Reset Button

Then, direct to Power Apps by using Customize forms function in SharePoint

Add a PenInput into the app and place it a suitable position.
You can hide the control by set false to ShowControls.

Add a button and named it as "Reset Signature".
Add the command Reset( PenInput1 ); on the property OnSelect

3. Save the signature when click Save

Add the following script to SharePoint Integrations's OnSave so when user click "Save", it will insert the image byte to the column.

SubmitForm(SharePointForm1);
Set( currentSharePointItem,LookUp( '<Your List Name>', ID=SharePointForm1.LastSubmit.ID ) );
Patch( '<Your List Name>', currentSharePointItem, { imageData :  Substitute( JSON( PenInput1.Image, IncludeBinaryData ), """", "" )  } );

For Substitute( JSON( PenInput1.Image, IncludeBinaryData ), """", "" ), the output of JSON( PenInput1.Image, IncludeBinaryData ) is like "data... EWQS==", which the image byte is surrounded by double quotes ". In order to remove it, we used Substitute.

You can retrieve the SharePoint record id by SharePointForm1.LastSubmit.ID.

Ref:

Check Point

You can save and publish the app. When you create a record, and if you follow the steps correctly, you should be able to see saved image byte in the imageData field.

4. Create image to show Signature

Go back to Power Apps. Insert an image and place it on top of the PenInput.

(PenInput.Image cannot be modified. So we need to add an image for showing signature.)

Input LookUp( '<Your List Name>', ID=SharePointIntegration.SelectedListItemID ).imageData in the property Image

Check Point

You can replace SharePointIntegration.SelectedListItemID into the record's id in LookUp( '<Your List Name>', ID=SharePointIntegration.SelectedListItemID ).imageData. If everything is fine, you can see the signature image you just signed

You can also view the record in SharePoint to check if the signature is correctly shown.

5. Add logic to hide and show the signature Image

You may already found that as the image has been on top of the Signature, you cannot modify the signature.

In order to resolve this problem, we will show and hide the signature image accordingly:

New Record, hide the signature image
View Record/Edit Record, show the signature image
Edit the signature, hide the signature image

In order to accomplish the above, create a global variable isSignatureImageVisible and set it to true/false accordingly:

For "Edit the signature, hide the signature image" part, when user edit the record, we will do as follow:

Change the button text from "Reset Signature" to "Modify Signature"
After user clicked "Modify Signature", it will clear the current signature and change the text back to "Reset Signature"

Set the "Reset Signature" button's property as shown below:

If( isSignatureImageVisible ,"Modify Signature", "Reset Signature")

Set( isSignatureImageVisible, false ); Reset( PenInput1 );

Finally, we would like to save the signature only if user has changed the signature.

SubmitForm(SharePointForm1);
If( ! isSignatureImageVisible,
    Set( currentSharePointItem,LookUp( '<Your List Name>', ID=SharePointForm1.LastSubmit.ID ) );
    Patch(  '<Your List Name>', currentSharePointItem, { imageData :  Substitute( JSON( PenInput1.Image, IncludeBinaryData ), """", "" )  } ) );

Done!🎉👏

Conclusion

When I first receive this request, I have done some google search on the Internet and found that there are mainly 2 ways to accomplish this. Both of the methods need to use Power Apps since there is no signature field in SharePoint. The differences are:

Save the signature image byte in the multiple lines of text created in Sharepoint. (The one we introduced)
Save the signature image file in the one drive and then get the link and pass to hyperlink field or attach it to the record in SharePoint.

I have chosen the 1st method because:

First method is more straight-forward. It only needs to consider Power Apps and SharePoint, but second method involves NOT JUST the above two apps, but also OneDrive and Power Automate.
Signature image has a more direct relation with the SharePoint record in the first method, but for the second method, the signature image is related to the SharePoint record through a hyperlink.
Since first method is simpler, you can pay less effort when you maintain the application

Office Scripts (Typescript) in Excel

Ivan Yu — Thu, 15 Dec 2022 16:00:00 +0000

Office Scripts Purpose

Office Scripts primary concern is to address the automation tasks in the Excel ON THE WEB (Not for the local excel file).

Use Office Scripts in Excel on the web to automate your common tasks.

Office Scripts | Microsoft Docs

On the other hand, since you can program in Office Scripts, it can also handle data in excel in a fast and accurate manner and other stuffs you can imagine.

Also, the same office scripts can be applied to MULTIPLE excel files. So one script for all 😎

Office Scripts Principle & Usage

Principle

As a developer, when I first look at Office Scripts my first impression is "What?😡 Macro again?"

However, thanks God, this is not Macro, but Typescript 👏👏👏, which you can develop base on TypeScript (or JavaScript) experience.

Usage

After talking so much, you may ask how to create a new script, you may follow the instruction below:

Create a new script by Automate --> New Script

On the right hand side, you should see a code editor and you can write your code insides the main function

The main function has only one parameter workbook, which you can manipulate the current excel sheet through it.

As you may already notice, the code editor looks very similar to Visual Studio Code. Yes, you are right. 🎉

The Code Editor is based on Visual Studio Code, so if you've used that environment before, you'll feel right at home.

Office Scripts Code Editor environment

No need to adapt a new code editor and a set of new shortcut. 😀

The Office Scripts Code Editor uses the same IntelliSense engine as Visual Studio Code. Intellisense

Most of the keyboard shortcuts for Visual Studio Code also work in the Office Scripts Code Editor. Shortcut

Office Scripts Example & Tips

Example (Read & Write to Excel)

function main(workbook: ExcelScript.Workbook) {
  // Get the "TutorialWorksheet" worksheet from the workbook.
  let worksheet = workbook.getWorksheet("TutorialWorksheet");

  // Get the cells at A1 and B1.
  let dateRange = worksheet.getRange("A1");
  let timeRange = worksheet.getRange("B1");

  // Get the current date and time using the JavaScript Date object.
  let date = new Date(Date.now());

  // Add the date string to A1.
  dateRange.setValue(date.toLocaleDateString());

  // Add the time string to B1.
  timeRange.setValue(date.toLocaleTimeString());
}

It prints the current date in A1 cell and time in B1 cell. Output of the example is as below:

My impression is Office Scripts API itself is very similar to other Excel libraries so if you have used one excel library before, you should find that it is quite familiar.

Tip 1 (Batch Read)

In above example, I would like to point out 2 things:

When reading large amount of date, getting data one by one by using getRange or getCell will result a performance issue

// Get the cells at A1 and B1.
let dateRange = worksheet.getRange("A1");
let timeRange = worksheet.getRange("B1");
.
.
.

This is because calling getRange or getCell triggers a network call, so calling getRange or getCell multiple times will result multiple network calls, which slows down performance.

The solution is to use getUsedRange, which read all the data in the excel once and put the values in a variable and read data from that variable.

let usedRange = workbook.getActiveWorksheet().getUsedRange();
let usedRangeValues = usedRange.getValues();
// Get values by i (row number) and j (column number)
usedRangeValues[ i ][ j ]

getValues(): (string | number | boolean)[][];

I have done an experiment to check this issue. In the experiment, it loops through 11 x 365 grid to get the values. The code difference and time cost ( 341s vs 0.38s) are as follow:

// Time cost: 341s
let cellValue = metaData.dataWorksheet.getCell(rowNum, 4 + i ).getValue() as string;

// Time cost: 0.38s
let cellValue = usedRangeValues[rowNum][4 + i] as string;

Reference: Read workbook data outside of a loop

Tip 2 (Batch Write)

Similar to tip 1, but this time is about writing data. Instead of writing large amount of data one by one using setValue, you can write all data once by Range.setValues()

setValues(values: (string | number | boolean)[][]): void;

Traps (Power Automate) & Limitation

Traps (Power Automate)

Power Automate can run Office Scripts by using the step Run Script. However, there are times that it runs fine in Excel but prompts ERROR in Power Automate. 😢

Trap 1 ( console.log( Object with circular structure ) )

This following script print out an object

function main(wb: ExcelScript.Workbook) {

    const obj = {
        val: "This is an value",
        wb
    }
    console.log({ obj });
}

Output:

But if we run it in Power Automate, it will show the following error:

We were unable to run the script. Please try again.
Runtime error: Line 7: Converting circular structure to JSON
--> starting at object with constructor 'n'
| property 'm_trackedObjects' -> object with constructor 'e'
--- property 'm_context' closes the circle
clientRequestId: 19eccc72-205f-4fd6-94e7-90c13610e91f

wb is the circular structure which causes the problem.

Trap 2 ( Power Automate CANNOT call external APIs )

You can make call with external API in Excel through Office Scripts but NOT in Power Automate. 😢

Calls to external APIs can be only be made through the Excel application, not through Power Automate under normal circumstances.

There is also other traps with Power Automate.

Limitation

External JavaScript is NOT supported

This one makes your life harder when you need to handle date in Typescript WITHOUT moment.js or any other date library. 😢

No 'any' type in Office Scripts

For more limitation, you may refer this

Conclusion

Office Scripts provide an great alternative to customize (although it only limit to Excel) function in Office 365.

Nginx: custom log file, reverse proxy, and load balancer

Ivan Yu — Thu, 08 Dec 2022 07:53:57 +0000

Summary

This blog is an explanation of Nginx configuration of:

Custom log file, specifies log files path
Reverse proxy server, hide the web application server from host
Load balancer, distribute network requests to different servers

Custom Log file, specifies log files path

Specifies the log file so you can get the log when any issue occurs.

You can tell Nginx to log the information by adding 2 parameters access_log and error_log

server {
    listen 30066;

    access_log /var/log/nginx/reverse-access.log;
    error_log /var/log/nginx/reverse-error.log;
}

access_log = log files consist of all requests access the server (filename should be different from access.log)
error_log = log files, which record error (filename should be different from error.log)

If you do not include the settings access_log or error_log, although you can still find the log files under /var/log/nginx/, those files are empty

Reverse proxy server, hide the web application server from host

By using the reverse proxy server, you can hide the server hosting web application from user.

Consider the code below:

Only open 30066 port for external access in the Linux hosting Ngnix
User access Ngnix by 30066
Ngnix passes the requests from 30066 to 30077 by

location / {
    proxy_pass http://localhost:30077;
}

For 30077, it is the server that hosts the HTML and js files

server {
    listen 30077;

    root /usr/share/nginx/html;
    index index.html index.htm;
    location / {
        try_files $uri $uri/ /index.html =404;
    }
}

You can copy the code below:

server {
    listen 30066;

    location / {
        proxy_pass http://localhost:30077;
    }
}

## real front-end server
server {
    listen 30077;

    root /usr/share/nginx/html;
    index index.html index.htm;
    location / {
        try_files $uri $uri/ /index.html =404;
    }
}

You can also set up different rules to pass to different locations

server {
    listen 30066;

    location /sysaid {
        proxy_pass http://172.18.0.5:31339;
    }

    location / {
        proxy_pass http://localhost:30077;
    }
}

The above code means that:

First, it checks whether the request url has /sysaid, if yes, it will pass to http://172.18.0.5:31339
Otherwise, it will pass to http://localhost:30077

Load balancer, distribute network request

When there is a lot of users or the web is heavily used, we need to distribute the workload to different web servers in order to maintain the stability and performance.

Consider the code below:

First, user access Nginx by 30066 port
Then, Nginx pass the request to the proxy
Ngnix will pass the request to the upstream server web
Finally, it will pass to request to either one of the server specifies under upstream web

Requests will be distributed equally to each server.

For example, if you have 4 requests coming in order (A, B, C, D), A and C will be passed to web_primary_1 and B and D will be passed to web_secondary_1

A simple approach for background task in Django

Ivan Yu — Fri, 18 Nov 2022 06:26:39 +0000

1. Introduction & Proof-Of-Concept

When there is a long running task, there are usually below 2 requirements:

As a user, I want to know the progress of the task
As a user, I want to get the output of the task if it is finished

We will use the out of the box features Threading and Cache in Python and Django respectively to achieve this. Let's see the demo below:

The logic is as follow:

Call API to start a task and get task id
Use task id to continue to get the task's status
Get the output if the task is finished

2. Steps

2.1. Basic Setup

Create a Django site called bgTaskEx
Create an app called bgTaskExApi
Install Django REST framework (pip install djangorestframework)
Setup API
Setup Django Cache

If you are not sure how to do it, you may check the below link:

If you setup correctly, you should see the below file structure:

2.2. Create a Task Handler class

Create a file named taskHandler.py and copy the following code to the file

from enum import Enum
from django.core.cache import cache
import threading
from typing import Union
from uuid import uuid1

class TaskHandler:

    def start_task(self, method, args):

        task_progress = TaskProgress()
        t = threading.Thread( target=method, args=[ *args, task_progress ] )
        t.setDaemon(True)
        t.start()

        return task_progress.get_task_id()

    @staticmethod
    def get_task_progress( task_id : str ):
        return cache.get( task_id )

class TaskProgress:

    task_id = str

    ## default constructor
    def __init__(self):
        self.task_id = str( uuid1() )
        cache.set( self.task_id, self, 3600 )

    def set( self,
        status : Enum,
        progress_message : Union[ str, None ] = None,
        output : Union[ str, None ] = None,) -> object:

        self.status = status.value
        self.progress_message = progress_message
        self.output = output

        cache.set( self.task_id, self, 3600 )

    def get_task_id( self ):
        return self.task_id

class Status(Enum):
    STARTED = 'STARTED'
    RUNNING = 'RUNNING'
    SUCCESS = 'SUCCESS'

TaskHandler has 3 classes:

TaskHandler = start task and get task progress
TaskProgress = store the task info in the cache
Status = an enum to define the status

The start_task process basically open a new thread to run the task and pass the task_progess into the thread to log the process.

2.3. Create a time-consuming method

Create a file named longRunningMethod.py and copy the following code to the file

from datetime import datetime
import time
from .taskHandler import Status


def long_running_method( input : str, task_progress ):

    task_progress.set( Status.STARTED, progress_message="The process has been started" )

    for i in range( 20 ):
        time.sleep( 0.5 )
        task_progress.set( Status.RUNNING, progress_message=f"{ 5 * i + 1 }% has been processed" )

    output = f"[{ datetime.now() }] input= { input }, value from Django"
    task_progress.set( Status.SUCCESS, output=output )

We give you an example of a time-consuming method, and you can change it to the method you like, but just keep in mind the following points:

In the parameter, you MUST include task_progress
During task execution, please make sure that you have logged the task progress by

task_progress.set( Status.RUNNING, progress_message=f"{ 5 * i + 1 }% has been processed" )

When the execution finished, please log the output by

task_progress.set( Status.SUCCESS, output=output )

2.4. Create API methods

Create the following 2 methods in views.py

@action(methods=['GET'],  detail=False, name='Start the task to background' )
def start_long_running_task( self, request ):

    input = request.GET[ 'input' ]

    task_id = TaskHandler().start_task( long_running_method, [ input ] )

    return JsonResponse({'task_id':task_id})

@action(methods=['GET'],  detail=False, name='Get Task Progress' )
def get_task_progress( self, request ):

    task_id = request.GET[ 'task_id' ]

    task_progress : TaskProgress = TaskHandler.get_task_progress( task_id )

    return JsonResponse( vars(task_progress) )

This 2 methods are the API endpoint of the method start_task and get_task_progress in TaskHandler respectively.

That's it.🎉👏

Additionally, you may add a test for those API methods in tests.py.

import re
import time
from django.test import SimpleTestCase

class MyTests(SimpleTestCase):

    def test_long_running_task(self):

        input = 'aaaaaa'
        print( f'{ input= }' )

        task_id = self.__start_task( input )
        print( f'{ task_id= }' )

        while True:

            time.sleep( 1 )
            result_dict = self.__get_task_progress_response( task_id )

            if result_dict[ 'status' ] == "SUCCESS":
                self.print_output(result_dict)
                break

            self.print_progress_message(result_dict)

    def print_output(self, result_dict):
        status = result_dict[ 'status' ]
        output = result_dict[ "output" ]
        print( f'{status=}, { output= }' )

    def print_progress_message(self, result_dict):
        status = result_dict[ 'status' ]
        progress_message = result_dict[ 'progress_message' ]
        print( f'{ status= },{ progress_message= }' )

    def __start_task( self, input ):
        res = self.client.get( f'/bgTaskExAPI/start_long_running_task/?input={ input }' )
        self.assertEqual( res.status_code, 200 )

        task_id = res.json()[ 'task_id' ]
        UUID_V1_PATTERN = re.compile( '[a-f0-9]{8}-[a-f0-9]{4}-1[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$', re.IGNORECASE)
        self.assertEqual( UUID_V1_PATTERN.match( task_id ) is not None, True )

        return task_id

    def __get_task_progress_response( self, task_id : str ):
        res = self.client.get( f'/bgTaskExAPI/get_task_progress/?task_id={ task_id }' )
        self.assertEqual( res.status_code, 200 )

        return res.json()

3. Source code

ivanyu199012/5.-Django_background_task

4. Reason of this approach

During development, you may face the following problem:

You would like to run a task in Django but it will take a long time. When the task is running, user need to check the process status continuously.

The simple request-response approach cannot solve this problem as when the request is sent to Django, if the task is still running, there will be no feedback from the server until it is finished or unfortunately, prompts out error.

After doing some google search, the logic of the existing solutions are about the same (mentioned in the logic section), but there are tones of different libraries for this.

Celery is one of those libraries. It is a popular framework with rich features, which can help you to handle a lot of different long-running tasks in a large scale project.

However, if you have only one or two long-running tasks and the project scale is small, this may not be a good approach as:

Celery + Django + Redis / RabbitMQ is not easy to setup
Celery mostly need to use Redis or RabbitMQ, but if you used Windows, you can only use RabbitMQ (need installation😢) (If you insist to use Redis in Windows, yes you can but you need do it through WSL or Docker. Again more complicated😢)

In this case, using Celery to like hitting a fly with a cannon, a little bit too much for it. And for the other libraries, you also need to install it if you want to use it.

I just wonder whether we can use what is existing in Django with a small amount of code to solve this problem.

In python - Simple approach to launching background task in Django - Stack Overflow, nbwoodward proposed an answer for the problem.

Github link

In his example, he saved the task progress in the database but not the cache.

If you need the task progress to be stored persistently, saving it to database will be a good choice.
Otherwise, using Django cache is easier as you do not need to set up a model and migration.

DEV Community: Ivan Yu

Using Django ORM only without web server

1. Step

1.1. Setup environment

1.2. Create Model class

1.3. Generate database and table

1.3.1. Remark

2. Create your own script

3. Git repo

4. Reason I wrote this blog

Remember to use break in switch

Misunderstanding about switch

Test result not matched

Correction

Script for Easy Git Branch Management

Setup

Feature

1. gsw --> Change git branch using no.

2. gsw -m --> Fast switch back to master branch

3. gsw -c <branch name> --> create new branch

Summary

Reason I wrote this blog

C++, Implement filter function for refactoring to keep code dry, clean and flexible

Scenario

Requirement: Add a function to retrieve people with income larger than 15000

Better solution: Filter function

Reason I wrote this blog

First glimpse into gRPC through Python (Part 2)

1. Previous Post

2. Implementation and Examples

2.1. 4 kinds of service method

2.1.1. Simple RPC

2.1.2. Response-streaming RPC

2.1.3. Request-streaming RPC

2.1.4. Bidirectionally-streaming RPC

2.2. Special Data Types and default value

2.2.1. Date Time

2.2.2. List

2.2.3. Map

2.2.4. Default value

3. Unit test

3.1. Create test server

3.2. Create test method

3.3. Run test

4. Reason I wrote this series of blogs

First glimpse into gRPC through Python (Part 1)

1. What is gRPC and the purpose of it?

2. Why do we learn it?

3. Difference between gRPC and REST

4. Development Process Explanaiton

5. Related Post & Github Link

Electron + Django ( Part 2 ), package it to production

1. Introduction & POC

2. Prerequisites

3. Package Django app

3.1. Remark

3.1.1. Server Error (500)

3.1.2. TemplateDoesNotExit: DEBUG = True

3.1.3. ModuleNotFoundError: No module named XXX

3.1.4. Pyinstaller rather than copy virutalenv

4. Package Electron app

5. Include Django exe in Electron package

5.1. Copy Django exe app to Electron package folder

5.2. Start the Django exe app during the electron startup process.

5.3. Skip open dev tool in production

5.4. A complete overlook

5.5. Package and run

6. Close the Django exe app when the window is closed

6.1. Reason behind it

6.1.1. Django exe was spawned by shell

6.1.2. Kill the process 2 event listeners

7. Source Code

8. Reason of writing this blog

Electron + Django ( Part 1 ), desktop app integrate JavaScript & Python

Introduction & Concept

POC (Proof-Of-Concept)

0. Prerequisites

1. Setup Electron

2. Setup Django

3. Start Django app when the electron start (using spawn)

Misunderstanding about `switch`

1. `gsw` --> Change git branch using no.

2. `gsw -m` --> Fast switch back to `master` branch

3. `gsw -c <branch name>` --> create new branch