DEV Community: Tolgahan ÜZÜN

Create Your Own S3 Service For Your Django Project

Tolgahan ÜZÜN — Tue, 10 Mar 2020 11:38:37 +0000

There are many projects with which you can install replica of Amazon S3 service in local environment. I am using the project named S3 Ninja from these. Let me tell you how to make an S3 replica with simple steps.

S3 Ninja

Installation and Run Serve.

docker pull scireum/s3-ninja:4
docker run -p 9444:80 scireum/s3-ninja:4

I assume you have no problems and the installation is complete!

Create a Bucket

This was so easy!

Codes to add settings for Django

You must install this package and add it to the app list. django-storages

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_ACCESS_KEY_ID = ‘access_key_id’
AWS_SECRET_ACCESS_KEY = ‘secret_access_key’
AWS_STORAGE_BUCKET_NAME = 'Bucket'
AWS_S3_ENDPOINT_URL = 'http://localhost:9444/s3/'

And action

Upload an image in the project. You need to do this using from django.core.files.storage import default_storage. Now your installation process will serve your own S3 service.

Django Test Plus Extensions

Tolgahan ÜZÜN — Wed, 18 Dec 2019 11:51:58 +0000

https://github.com/tolgahanuzun/test_plus

Django Test Plus Extension is an advanced tool for Django tests. Motivation; This is a detailed analysis of the atomic parts of Django tests. In short, each running test understands how long it runs.

The total time information of Django tests is returned. There are more than a thousand test cases in the company I work for. I wanted to analyze long tests and correct mistakes.I saw such a shortcoming, I aimed to write a package.

Support and Update

Inspired by the Django Extensions package. It is free and open to improvement. I can improve if you tell me your needs. You can also support the development.

Installing

pip install test_plus

and

Add it to INSTALLED_APPS in your projects settings.py file:

INSTALLED_APPS = (
    ...
    'test_plus',
    ...
)

Using and Result

It takes the same parameter in Django tests.

python manage.py test_plus

python manage.py test_plus app_name.test.test_file -k -v 2

Simple tricks for map solution

Tolgahan ÜZÜN — Thu, 15 Aug 2019 08:36:42 +0000

In the past, we used the map google map service for the map solution. As the project grew, google map began to write high amounts of cost. We looked for a few solutions here and, divided our needs into 2 parts.

Dynamic display of latitude and longitude data. (embed)
Find latitude and longitude data using street, shop and home address. (geocoding)

Embed solution

Here are a few alternatives. (openstreetmap, google map, mapbox vs...) I will evaluate the difference between google map and mapbox.

Legacy solution: Google Map: SKU: Dynamic Maps

For more than 100,000 requests:
```
- 5.60 USD per 1000
```
Example: 500.000 (100k free) requests: 2.240 USD

New solution: Map Loads for Web

For more than 50,000 requests:

- (50,001 to 100,000) 5.00 USD per 1000
- (100,001 to 200,000) 4.00 USD per 1000
- (20,001 to 1000,000) 3.00 USD per 1000

Example: 500.000 (50k free) requests: 1.560 USD

Profit : 680 USD

Development process

I was using the react-leaflet library in the front. (openstreetmap) The Leaflet library is very convenient and convenient. You can define api as Custom.

Highly recommend the react-leaflet. :)

import { Map as LeafletMap, TileLayer } from 'react-leaflet';

<LeafletMap>
    <TileLayer url={`https://api.mapbox.com/styles/v1/{#username}/{#project}/tiles/{z}/{x}/{y}?access_token={#token}`}/>
</LeafletMap>

Geocoding solution

There is no alternative to this substance at this time. The alternatives are quite inefficient. It negatively impacts user experience. I recommend using Google Map.

Standard and Streaming Response Comparison with Aiohttp Library

Tolgahan ÜZÜN — Thu, 04 Jul 2019 08:23:38 +0000

Github Repository: Streaming_API

In the project I developed, we return a great response message to the API calls of the customers. It takes a long time to process large data. At this stage, the memory swells and the server CPU increases. To prevent this, we started using streaming method.

I've done an analysis to explain what this process has brought us. I have compared a normal response to a call request and streamed a call request.

Working area

Python3.6
Aiohttp
PostgreSql (SQLAlchemy)
500000 rows of data set

(Referenced from the github readme file.)

How to install

git clone git@github.com:tolgahanuzun/Streaming_API.git
virtualenv -p python3 venv
source venv/bin/activate
cd Streaming_API
pip install -r requirement.txt

vim settings.py and you need to edit the settings file yourself.

How to run

Serve

python run.py

Client

JWT needs it. You can remove it if you want.

curl -H 'Accept: text/plain' -H "Authorization: JWT eyJ0eX....."  -v http://0.0.0.0:8080/standard

curl -H 'Accept: text/plain' -H "Authorization: JWT eyJ0eX....."  -v http://0.0.0.0:8080/chunked

Process analysis

Chunked data comes to you piece by piece.
In the standard, the data comes at once.
Incoming data sample.

Standard Response

All data is captured, processed and sent at one time. Blocks each other. In this example, it corresponds to 500000 row.

Up to 0.16 seconds no data flow occurs. The data comes after that second.

Streaming Response

The dataset size is 500000 rows. Data from the database is taken as 1000 row. Each group of data is processed and streamed without waiting.

Up to 0.04 seconds no data flow occurs. The data comes after that second.
Look out! The Streaming API finished before the Standard API started sending data. It didn't block itself because it sent and received data in pieces.