DEV Community: lightkun

Remote Debug Django3(Docker) with VSCode

lightkun — Wed, 30 Dec 2020 01:31:50 +0000

sample github

https://github.com/katsuya-n/pub_docker_django

environment

mysql 5.7
python 3.9
django 3

install Python extension for Visual Studio Code

Install VSCode python library. Reopen VSCode after install.

docker-compose

Django project needs to allow port 3000 to use remote debug, so write - "3000:3000".

 web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - "8000:8000"
      - "3000:3000"
    depends_on:
      - db

my sample docker-compose.yml

launch.json

Create file .vscode/launch.json. Port sets 3000. [your docker project remote root path] changes yourself.

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Django",
            "type": "python",
            "request": "attach",
            "port": 3000,
            "host": "localhost",
            "pathMappings": [
                {
                    "localRoot": "${workspaceRoot}",
                    "remoteRoot": "[your docker project remote root path]"
                }
            ],
            "django": true
        }
    ]
}

※When request change from attach to launch, you may not success remote debug.

...
"request": "launch",
"program": manage.py",
...

my sample launch.json

Dockerfile(Django)

Install python library called ptvsd. It has been developed by Microsoft company, but github's star is few...

RUN pip install ptvsd

my sample Dockerfile

start project

$ docker-compose up -d

enable ptvsd

Edit manage.py

# add
import ptvsd
...
# add, port=3000
ptvsd.enable_attach(address=('0.0.0.0', 3000))

my sample manage.py(remove comment out L23)

Attention! Docker container can't seem to start if you enable ptvsd before start docker container.

start debug

Enter fn + F5(Mac OS). Under bar color changes from blue to orange.

check break point

Attention! Break point can't seem to use some of other files(etc. polls/url.py, manage.py ...). Sorry, I don't know the reason.

access URL

Access your project url(etc. http://127.0.0.1:8000/polls/), and program will stop break point.

stop break point

Try using swagger-ui

lightkun — Sat, 23 May 2020 14:36:30 +0000

Hi all,
I try using swagger-ui on docker. It is too easy to build.

github

https://github.com/katsuya-n/docker-swagger-ui

environment

docker image : swaggerapi/swagger-ui

directory structure

.
├── docker-compose.yml
└── swagger
                  └── api.yaml

1. make docker-compose.yml

version: '3.3'
services:
  swagger-ui:
    image: swaggerapi/swagger-ui
    container_name: swagger-ui
    ports:
      - "8001:8080"
    volumes:
      - ./swagger:/usr/share/nginx/html/swagger
    environment:
      API_URL: swagger/api.yaml

If you use port number 8001, you should change other.
Acocording to my research, docker swagger-ui image seems to use nginx. So, file path is /usr/share/nginx/html/swagger.

2. make swagger directory and api.yaml

If you just want to build swagger-ui, I recommend copy swagger editor sample and paste in api.yaml.

3. build

Run docker command.
docker-compose up -d

You check http://localhost:8001/ when docker build is completed.

how to edit api.yaml?

By the way, do you think it is not easy to edit yaml file?
If you use Visual Studio Code, you can use Swagger Viewer. It can see preview of editing file in real time and is comfortable.

Solve Elasticsearch complex sort using script

lightkun — Sat, 11 Apr 2020 11:54:54 +0000

Overview

One day, I had to write program of complex sort in elasticsearch.
At first, I tries to operate score field but I noticed that it didn't work and it cannot explain what I want to do.
Therefore, I used script to solve this question. This article is a report of this solution.

Environment

※use docker container

Elasticsearch version 7.1.0
Kibana version 7.1.0

Background

I thought simple example for you.
What do you choose cloud service? AWS? Azure?

To take an example, 4 companies using cloud service.

A company
- main cloud servie : AWS
- sub cloud service: GCP, Azure
B company
- main : GCP
- sub : AWS
C company
- main : GCP
- sub : AWS, Azure
D company
- main : Azure
- sub : GCP, AWS

company name	main	sub
A	AWS	GCP, Azure
B	GCP	AWS
C	GCP	AWS, Azure
D	Azure	GCP, AWS

If I input "Azure", I expect hitting company names using Azure． To sum up, A, C and D.
Additionally, I want to add 1 search option of sort. If input value is equal to main cloud service, it's on top in search result. On the contrary, if input value is equal to sub cloud service, it's on under.
Finally, search result I expect is D, A and C.

input value	expect result
Azure	D, A, C
AWS	A,B,C,D
GCP	B,C,A,D

※If hitting column is same, random order.

Azure : (sub)A and C
AWS : (sub)B, C and D
GCP : (main)B and C. (sub)A and D

In the case of database(MySQL)

It was difficult for me to think elasitcsearch code, so that I decided to think about using database.

※use docker container

mysql version 5.7

Preparation

①Create table.

CREATE TABLE cloud_service (
  id INT PRIMARY KEY,
  company_name VARCHAR(50),
  main_name VARCHAR(30)
);

CREATE TABLE cloud_service_sub (
  id INT PRIMARY KEY,
  cloud_service_main_id INT,
  sub_name VARCHAR(30)
);

②Insert table data.

INSERT INTO cloud_service (id, company_name, main_name) VALUES
  (1, 'A', 'AWS'),
  (2, 'B', 'GCP'),
  (3, 'C', 'GCP'),
  (4, 'D', 'Azure');

INSERT INTO cloud_service_sub VALUES
  (1, 1, 'GCP'）,
  (2, 1, 'Azure'）,
  (3, 2, 'AWS'）,
  (4, 3, 'AWS'）,
  (5, 3, 'Azure'）,
  (6, 4, 'GCP'）,
  (7, 4, 'AWS');

③Check table data.

SELECT * FROM cloud_service;

+----+--------------+-----------+
| id | company_name | main_name |
+----+--------------+-----------+
|  1 | A            | AWS       |
|  2 | B            | GCP       |
|  3 | C            | GCP       |
|  4 | D            | Azure     |
+----+--------------+-----------+
4 rows in set (0.00 sec)

SELECT * FROM cloud_service_sub;

+----+-----------------------+----------+
| id | cloud_service_main_id | sub_name |
+----+-----------------------+----------+
|  1 |                     1 | GCP      |
|  2 |                     1 | Azure    |
|  3 |                     2 | AWS      |
|  4 |                     3 | AWS      |
|  5 |                     3 | Azure    |
|  6 |                     4 | GCP      |
|  7 |                     4 | AWS      |
+----+-----------------------+----------+
7 rows in set (0.00 sec)

sort SQL sentence

When I ran the following SQL, I got expected result.
"ORDER BY CASE ~" sentence is an extremely important point.

When input value is equal to main.main_name, this record sort no is 0. When input value isn't equal to main.main_name, this record sort no is 1.
Then, By using "ASC", main → sub.

SELECT DISTINCT company_name, main.main_name
FROM cloud_service main 
LEFT JOIN cloud_service_sub sub ON main.id = sub.cloud_service_main_id
WHERE main.main_name = "Azure" OR sub.sub_name = "Azure"
ORDER BY 
CASE main.main_name 
WHEN "Azure" THEN 0 
ELSE 1 END 
ASC;

+--------------+-----------+
| company_name | main_name |
+--------------+-----------+
| D            | Azure     |
| A            | AWS       |
| C            | GCP       |
+--------------+-----------+
3 rows in set (0.00 sec)

SELECT DISTINCT company_name, main.main_name
FROM cloud_service main 
LEFT JOIN cloud_service_sub sub ON main.id = sub.cloud_service_main_id
WHERE main.main_name = "AWS" OR sub.sub_name = "AWS"
ORDER BY 
CASE main.main_name 
WHEN "AWS" THEN 0 
ELSE 1 END 
ASC;

+--------------+-----------+
| company_name | main_name |
+--------------+-----------+
| A            | AWS       |
| C            | GCP       |
| D            | Azure     |
| B            | GCP       |
+--------------+-----------+
4 rows in set (0.00 sec)

SELECT DISTINCT company_name, main.main_name
FROM cloud_service main 
LEFT JOIN cloud_service_sub sub ON main.id = sub.cloud_service_main_id
WHERE main.main_name = "GCP" OR sub.sub_name = "GCP"
ORDER BY CASE main.main_name 
WHEN "GCP" THEN 0 
ELSE 1 END 
ASC;

+--------------+-----------+
| company_name | main_name |
+--------------+-----------+
| C            | GCP       |
| B            | GCP       |
| A            | AWS       |
| D            | Azure     |
+--------------+-----------+
4 rows in set (0.00 sec)

In the case of Elasticsearch(operating kibana console)

Preparation

①create template

PUT /_template/my_template?pretty
{
  "index_patterns": "mytest",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "company_name": {
        "type": "keyword"
      },
      "main_cloud_name": {
        "type": "keyword"
      },
      "sub_cloud_name": {
        "type": "keyword"
      }
    }
  }
}

②post data

POST /mytest/_doc
{
  "company_name" : "A",
  "main_cloud_name": "AWS",
  "sub_cloud_name": ["GCP","Azure"]
}

POST /mytest/_doc
{
  "company_name" : "B",
  "main_cloud_name": "GCP",
  "sub_cloud_name": ["AWS"]
}

POST /mytest/_doc
{
  "company_name" : "C",
  "main_cloud_name": "GCP",
  "sub_cloud_name": ["AWS", "Azure"]
}

POST /mytest/_doc
{
  "company_name" : "D",
  "main_cloud_name": "Azure",
  "sub_cloud_name": ["GCP", "AWS"]
}

③check document

GET /mytest/_search?pretty

sort Elasticsearch search API

When I ran the following search API, I got expected result.
"_script" sentence is an extremely important point.

"type" : This is the type using sort. I can choose "string" or "number".

"lang" : "painless" is original langrage of Elasticsearch and can use part of java API.(Sorry, I don't know this in detail. Maybe, "painless" is java base.)

"source" : I can write the script of "lang". The mechanism of script contents is the same as sort SQL sentence

GET /mytest/_search?pretty
{
  "query": { 
    "bool" : {
      "should" : [
        {
          "term" : {
            "main_cloud_name" : "Azure"
          }
        },
        {
          "term" : {
            "sub_cloud_name" : "Azure"
          }
        }
      ]
    }
  },
  "sort" : {
    "_script" : {
      "type" : "number",
      "script" : {
        "lang": "painless",
        "source": """
            if (params.main == (doc['main_cloud_name'].value)) {
              return 0;
            } else {
              return 1;  
            }
          """,
        "params" : {
          "main" : "Azure"
        }
      },
      "order" : "ASC"
    }
  }
}

GET /mytest/_search?pretty
{
  "query": { 
    "bool" : {
      "should" : [
        {
          "term" : {
            "main_cloud_name" : "AWS"
          }
        },
        {
          "term" : {
            "sub_cloud_name" : "AWS"
          }
        }
      ]
    }
  },
  "sort" : {
    "_script" : {
      "type" : "number",
      "script" : {
        "lang": "painless",
        "source": """
            if (params.main == (doc['main_cloud_name'].value)) {
              return 0;
            } else {
              return 1;  
            }
          """,
        "params" : {
          "main" : "AWS"
        }
      },
      "order" : "ASC"
    }
  }
}

GET /mytest/_search?pretty
{
  "query": { 
    "bool" : {
      "should" : [
        {
          "term" : {
            "main_cloud_name" : "GCP"
          }
        },
        {
          "term" : {
            "sub_cloud_name" : "GCP"
          }
        }
      ]
    }
  },
  "sort" : {
    "_script" : {
      "type" : "number",
      "script" : {
        "lang": "painless",
        "source": """
            if (params.main == (doc['main_cloud_name'].value)) {
              return 0;
            } else {
              return 1;  
            }
          """,
        "params" : {
          "main" : "GCP"
        }
      },
      "order" : "ASC"
    }
  }
}

DEV Community: lightkun

Remote Debug Django3(Docker) with VSCode

sample github

environment

install Python extension for Visual Studio Code

docker-compose

launch.json

Dockerfile(Django)

start project

enable ptvsd

start debug

check break point

access URL

stop break point

Try using swagger-ui

github

environment

1. make docker-compose.yml

2. make swagger directory and api.yaml

3. build

how to edit api.yaml?

Solve Elasticsearch complex sort using script

Overview

Environment

Background

In the case of database(MySQL)

Preparation

sort SQL sentence

In the case of Elasticsearch(operating kibana console)

Preparation

sort Elasticsearch search API

Reference