DEV Community: george_pollock

Use Python to translate text from images to any language

george_pollock — Thu, 27 May 2021 00:00:00 +0000

Who is it for?

Have you ever found yourself looking for one of those bilingual books but none of them caught your interest? Maybe you thought that they re too expensive, after all - most of them are already on a public domain.

Or maybe you were abroad, went to a book store and found a really interesting book available only in the local language?

Scientific papers? That fits too!

This post explains how to translate photos, pictures that have text within them so that consists photos, screenshots, images, etc.

In order to initialize the project you will need a Google API key from Google Console which can take a bit to generate especially if you have never done it before. If you want to translate small batches or just a couple of images you may try to use other tools.

I love languages

In the following example you will see that we are going translate Siddhartha by Herman Hesse which is on public domain. The book we translate is in German and you can pick up just about any language google offers (one of 64). That is something that could really ease a language learning!

Copyrights

How does copyright work arrive in public domain and can be used freely?

Most of works people create is protected with Copyrights. In most of the countries they last a lifetime of an author plus 25-100 years. Depending on the country you may find books of foreign authors sooner - verify detailed world map to see exact timings.

You may wonder how to ask permission to translate a book, on public domain it is not required anymore (although if you are still wondering then just writing to Authors with a polite request will do). So can you translate public domain books? Yes, you can!

Once the Copyrights expire, works are becoming publicly available, they become part of a public domain. This is where we are going to get the book to translate - Siddhartha by Herman Hesse. On Archive.org we can see usage category which states: Usage Public Domain Mark 1.0

In order to show the full process I printed a few pages:

Let’s start

We need to first scan those pages. We will use mobile app called Microsoft Lens:

Once you download it, try scanning the pages. It has nice option of making the scanned documents flat and readable.

Once you finish scanning all the pages, select a black and white sharpening filter and sve those files.

Example scanned document should look like this:

Now put all those scanned photos into one folder on your computer and let´s go to the code.

The intro

The script I wrote utilizes python packages for OCR scanning, translating and saving into docx file. It´s sort of a free document translation software and although it´s just a few python lines it can perform a full book translation and later you can generate pdf out of this docx.

Even if you have just pdf it can be tweaked and used as ebook translator. You can just use free pdf to png converter to obtain photos and run the script. This translation will only run if you´re online. The Python translate logic does not work offline because we are querying Google Translate API.

Maybe you´ve heard about other services like reverso, bing translator or deepl translator. Using this script you can have unlimited, ad free experience that you can tweak to your needs and make your own book translation app.

(If you do not want to use Google Services you may check apertium that also let´s you translate documents).

The code

Code is available here under GNU licence. You may want to check the usage and instalation steps.

The result is two files generated with original and translated text:

Let me know what do you think and feel free to reach out if you need help.

_Tell me about your insights and leave a comment - you are most welcome to see more posts of this type just go to home page. Cover:_Photo by Beatriz Pérez Moya on Unsplash

Software Development Engineer in Test || How Google tests software?

george_pollock — Sun, 14 Mar 2021 00:00:00 +0000

I wanted to know if at Google they do testing manually, if they have manual testers. If you want TLDR answer, here is the answer ( 2 quotes taken from the books ):

In 2005 most of the testing was done manually if it was done at all.

Currently:

Attempting to assess product quality by asking humans to manually interact with every feature just doesn’t scale. When it comes to testing, there is one clean answer: automation.

If you want to read more on how they test their software , keep reading!

I will be quoting the books from the picture a lot here.

Why is Google’s software so good?

Google engineers their products.

Software Engineering differs from programming in dimensionality programming is about producing code. Software engineering extends that to include the maintenance of that code for its useful life span.

In order to engineer software properly a pipeline of continuos delivery of new features, changes and improvements has to be set up. This pipeline has to stream tested and approved changes to a user in order to avoid expensive fixes ( shifting left concept ) and lose of trust.

In order to have well-working pipeline a few steps have to be taken.

Clear documentation and specification that allows for .
Process of testing that is mostly automated which allows scaling.
Reusability of code: components, dependencies, etc.

Let’s break down those 2 points and analyse them

Documentation - Holy Grail that takes time to shine

The development should be about development. Engineers should focus on investigating the code, implementing, developing features, testing. The should not waste time on figuring out what they have to develop in the first place. They should have clean documentation both for new projects:

Project teams are more focused when their design goals and team objectives are clearly stated.

and for legacy code:

Quality documentation has tremendous benefits for an engineering organization. Code and APIs become more comprehensive and reduce mistakes.

Writing documentation, maintaining it and spending time on it can have a very big feedback loop when it comes to profits.

Benefits aren’t immediate.

It may be difficult to see immediate gains but in a long term they will be visible. They are necessary. On the beginning Google had no troubles with documentation but as the codebase grows, the need for it becomes more and more visible. That is why, in case wqhen a company wants to grow and scale their products, it is necessary to have a good documentation from the beginning. Google’s history gives a little bit of insight:

New users discovering bad documents either couldn’t confirm that the documents were wrong or didn’t have an easy way to report errors. […] The way to improve the situation was to move important documentation under same sort of source control that was being used to track code changes.

Even if scaling is not taken into consideration, normal development can also hugely benefit from writing documentation. Developers need to know that writing documentation is not much different than writing the code.

Testing - why should we do it?

Shifting left concept.

What is it?

If you can catch it before the original developer commits the flaw to version control, it’s even cheaper.

If we have User Story development on the left side of the graph, then on the right side of it we have delivery to production. The X axis is time.

The closer to the left side of the graph we catch bugs, the cheaper it will cost us.

Catching bugs on production is very costly : revising legacy code, changing integrations, losing customer’s confidence in our products. All that affects the turnover. That is also why documentation plays an important part. It helps to shift towards left side. Helps to avoid introducing bugs.

There will always be bugs. No matter how good the developers and documentation. The most important part of shifting left is implementing testing process.

Testing - Google’s good standards for everyone

There is a few places where we can automate testing. At Google there is a culture that says:

Treat your tests like production code.

This means that a lot of testing is being done by the developers. They are responsible for writing tests. They can’t test everything. That is why there is a separate job position that for some scopes of testing.

The engineers who build systems today, play an active and integral role in writing and running automated tests for their own code.

Test scopes that are at Google are as follows:

Narrow scope - unit tests (80% of tests)
Mediums scope - integration tests (15% of tests)
Large scope - end to end tests (5% of tests)

The 5% that is left - exploratory, beta, smoke testing.

Test suite antipatterns are basically processes that have significantly different amounts of tests by percentage in a code base.

e.g an ice cream antipattern is the revers of the top example:

Unit tests (20% of tests)
Integration tests (55% of tests)
End to end tests (25% of tests)

Antipatterns should be avoided. Remember, this is the Google’s view of products so deviating from those rules could be possible. We need to remember that the goal of Google is to deliver quickly , well tested product.

If your company’s structure or process is not similar to Google’s - there are chances that the goal is. Everybody should have structure that allows for quick deliveries and high quality.

Even in companies where QA is a prominent organization, developer-written tests are commonplace. At the speed and scale that today’s systems are being developed, the only way to keep up is by sharing the development of tests around the entire engineering staff.

Testing - Automate where possible

You should be able to automate every test that you make. In case of failure or bug, a new test to cover the case should be created. This requires a very well structures architecture of the software product and according chose of tools.

It is important to remember that testing should always be done in order to increase quality assurance. You have to be able to trust your tests. That means that tests have to be good.

A bad test suite can be worse than no test suite at all.

Treating tests in this way, fixing and adjusting them immediately maintains (or increases) our level or certainty regarding test results.

Allowing failed tests to pile up quickly defeats any value they were providing, so it is imperative not to let that happen.

Google’s take on broken tests is simple:

Teams that prioritize fixing a broken test within minutes of a failure are able to keep confidence high and failure isolation fast, and therefore derive more value out of their tests.

Not adjusting and fixing test causes test flakiness. This can have negative impact on the team and quality.

If test flakiness continues to grow, you will experience something much worse than lost productivity; a loss of confidence in the tests. […] After that happens, engineers will stop reacting to test failures, eliminating any value tha test suite provided.

At Google % of failed tests is limited to 0.15%.

Testing - Bunch of advices for SDET

SDET - Among of other responsibilities that they have, Software Developer Engineer in Tests is a person that guards quality by automating testing procedures. They are usually responsible for implementing medium and larger tests where SDEs ( Software Developer Engineers ) develop smaller tests.

At Google they differentiate tests to 3 sets:

Small - No sleeping, no awaits, no blocking calls, they run as fast as possible.
Medium - localhost webdriver, slower, nondeterministic.
Large tests - no localhost, remote cluster, integrate wide assets.

The amount of small tests should be the highest due to it’s runtime and maintainability (that’s why SDEs are always responsible for covering their code with good unit tests).

Smaller and smaller tests tourned out to be faster, more stable and generally less painful.

All tests should strive to be hermetic. The larger the tests the more difficult to stick to this advice.

Reusability

Code is a liability. Using the Statical Code Analysis tools we could assess the amount of duplication that is in our codebase. Increasing duplication decreases the speed of delivery in a CI/CD pipe.

Duplicated code not only is a wasted effort, it can actually cost more in time than not having the code at all; changes that could be easily performed under one code pattern often require more effort when there is duplication in the codebase. Basically, “if you’re writing it from scratch you’re doing it wrong”.

Sum up

Google surprises me with the quality of their product. Their uptime, security measures, user interface and ease of access is amazing. If I love their products it means that they test it well. The quality that they maintain is a proof that their way works. If they say that it’s possible to scale testing by automating it or that developers should be testers, it’s difficult to disagree with this statement.

Tell me about your insights and leave a comment - you are most welcome to see more posts of this type just go to home page

<!-- [0]:

How does Google test software?”
fundamentals of testing 720 1.30 0.03
qa at amazon 590 0.31 0.06
google testing blog 320 1.41 0.02
how google tests software pdf 210 0.00 0.04
android automation testing tools 140 2.70 0.3
software testing new technologies 110 1.00 0.1
how we test software at microsoft 110 6.57 0.08
google cloud functions testing 70 0.00 0
how we test software at microsoft pdf 50 0.00 0.04
google qa salary 30 0.00 0.04
google cloud platform automation 20 7.69 0.22
google software quality assurance 10 0.00 0.1
how google tests software amazon 10 0.00 0.14
how google tests software part one 0 0.00 0
how google tests software blog 0 0.00 0
cloud test automation using selenium 0 0.00 0
devops test data management 0 0.00 0
gcp devops architecture 0 0.00 0
test data management java 0 0.00 0
effective ways in the test management. 0 0.00 0
crowdtesting platform 0 0.00 0
testing new features 0 0.00 0
learn android testing 0 0.00 0
ui android test 0 0.00 0
android studio unit test ui 0 0.00 0
google phone testing 0 0.00 0
how does facebook test new features 0 0.00 0
sdet conferences 0 0.00 0
google automation summit 0 0.00 0
screenster login 0 0.00 0
google's acc methodology 0 0.00 0
how to test google book 0 0.00 0
qa in google 0 0.00 0
google testing blog 320 1.41 0.02
how google tests software amazon 10 0.00 0.14
how we test software at google 10 0.00 0
how google test software free pdf 0 0.00 0
how google tests software 2020 0 0.00 0
how google tests software blog 0 0.00 0
how google tests software summary 0 0.00 0
how google tests software epub 0 0.00 0

Static code analysis.

One of the most important technological improvements regarding automation over the past few years is automatic static analysis of a given code change.

As a corollary to this, we also strongly discourage the use of control flow statements like conditionals and loops in tests.

When conducting and exploratory test, the specific problems to be found are unknown at the start. As with the detection of security vulnerabilities, as soon as an exploratory test discovers an issue, an automated test should be added to prevent future regressions.

-->

Geolocation tutorial project with Geodjango and GIS data to build using REST api.

george_pollock — Mon, 09 Nov 2020 00:00:00 +0000

Boilerplate with Geodjango and Leaflet - learn while using it and deploy to Heroku

The topic we will cover here. Click to go to code examples:

Get geolocation from the IP - we will get coordinates of the IP that we extract from the request itself. I mention loading IP data from other DB to not use external API.
Searching via Address - we will use q django query to search location fields.
Adding new objects on map - we will add a few objects using demo VueJS component with api from django rest framework.
Closest object - we will use geodjango point to find closest object

Plus useful snippets:

Drawing polygons on map - we check here geodjango area.
List of objects within radius
Calculate distance between two points - backend and frontend solution.

ExTRA :

Deployable boilerplate with all above - how to prepare PostgreSQL database for geodjango on Heroku.

Important: We will be using vuejs with geodjango (connecting django rest framework api with javascript) throughout this tutorial - django-rest-framework-gis. This django package is compatible with both currently used django versions - Django 2 and Django 3. For more compatibility check geodjango documentation.

Intercepting requests - using proxy and interceptor to hack your workflow ( no swagger? no problem! )

Get geolocation from the IP

There are 3 ways of obtaining geolocation based on IP address: (easy) ⚪ Call an external geocoding webservices API that returns location data (done from frontend or backend) (harder) 🟠 Use geo IP database with mapped IP.

First let’s get IP address from the django request. Let’s install django-ipware

pip install django-ipware

and use it in class views :

from rest_framework.generics import ListAPIView
from rest_framework.response import Response
from rest_framework import status
from ipware import get_client_ip

class ShowUsersIP(ListPIView):
    """
    Sample API endpoint that shows user's IP
    """
    queryset = Message.objects.all()
    serializer_class = UserSerializer

    def get(self, request):
        client_ip, is_routable = get_client_ip(request)
        print('IP_INFO: ',client_ip, is_routable)
        data = {
            'client_ip': client_ip,
            'is_routable': is_routable
        }
        return Response(data, status=status.HTTP_200_OK)

or function based views :

from django.http.response import JsonResponse
from ipware import get_client_ip

def show_users_ip(request):
    client_ip, is_routable = get_client_ip(request)
    data = {
        'client_ip': client_ip,
        'is_routable': is_routable
    }
    return JsonResponse(data)

(The easy one) ⚪

Now that we know how to get the IP of the user we can track it’s location. With ip.info we get 50k calls on freemium which is more than enough. In order to obtain the location you need to pass the IP to the request:

import sys
import urllib.request
import json

ACCESS_TOKEN = 'YOUR_TOKEN'

def get_lat_lng_from_ip(ip_address):
    """Returns array that consists of latitude and longitude"""
    URL = 'https://ipinfo.io/{}?token={}'.format(ip_address, ACCESS_TOKEN)
    try:
        result = urllib.request.urlopen(URL).read()
        result = json.loads(result)
        result = result['loc']
        lat, lng = result.split(',')
        return [lat, lng]
    except:
        print("Could not load: ", URL)
        return None

location = get_lat_lng_from_ip('208.80.152.201')
print('Latitude: {0}, Longitude: {1}'.format(*location))
# Prints: Latitude: 32.7831, Longitude: -96.8067

The response we get and transform to get lat and lng is this:

{
  "ip": "208.80.152.201",
  "city": "Dallas",
  "region": "Texas",
  "country": "US",
  "loc": "32.7831,-96.8067",
  "org": "AS14907 Wikimedia Foundation Inc.",
  "postal": "75270",
  "timezone": "America/Chicago"
}

Responses vary between providers. Some of other service providers are: Google, OpenCage

Now let’s save the latitude and longitude we got from user’s IP to his account ! All examples are shown in the boilerplate code. The User model we have there extends Abstract user.

# User model
class User(AbstractUser):
    #...
    coordinates = models.PointField(blank=True, null=True, srid=4326)

# Update view
# ...
from rest_framework_gis.pagination import GeoJsonPagination
from backend.accounts.api.serializers import UpdateLocationSerializer
from backend.utils.coordinates import get_lat_lng_from_ip
from rest_framework.response import Response
from django.contrib.gis.geos import GEOSGeometry
from django.contrib.gis.geos import Point
from rest_framework import status
import json

class UpdateLocation(UpdateAPIView):
    """ Updates location of a user using it's IP taken from request"""
    model = User
    serializer_class = UpdateLocationSerializer
    pagination_class = GeoJsonPagination

    def update(self, request, *args, **kwargs):
        """Here we:
        - get location from the IP
        - change coordinates to a randomly
        close one in order to anonymize users location.
        """

        # Using ipware library
        client_ip, is_routable = get_client_ip(request)
        # Using our function created previously in utils
        latitude, longitude = get_lat_lng_from_ip(client_ip)
        if not latitude:
            return Response({'message': 'IP or location was not found.'}, status=status.HTTP_406_NOT_ACCEPTABLE)
        # Anonimizing users location before saving
        latitude, longitude = self.anonymize_location(latitude, longitude)
        print('Users latitude {} and longitude {}'.format(latitude, longitude))

        # For GeoDjango PointField we define y and x
        # more in docs: https://docs.djangoproject.com/en/3.1/ref/contrib/gis/geos/#point
        point = Point(float(longitude), float(latitude), srid=4326)

        # Requires at least 1 user in DB (e.g. admin)
        user_obj = User.objects.all().first()

        # Using partial update to not create a new user obj
        anonymize_profile = UpdateLocationSerializer(user_obj, data={'coordinates': point}, partial=True)
        if not anonymize_profile.is_valid():
            return Response(anonymize_profile.errors, status=status.HTTP_400_BAD_REQUEST)
        anonymize_profile.save()

        pnt = GEOSGeometry(point)
        geojson = json.loads(pnt.geojson)
        return Response({'coordinates': reversed(geojson['coordinates'])}, status=status.HTTP_201_CREATED)

For your purposes there is a high chance you should use the first solution. There is many providers with free tiers offering many calls with high degree of accuracy.

(The hard one) 🟠

This option is based on downloading databases of IP and locations and querying it with the obtained IP of the user. You can check IP2location page to see how to do it. Install python package for querying the DB and download the BINs with data to query.

(for more loading dbsync data check out this realpython django tutorial)

Searching via Address

Here we talk about geocoding and reverse geocoding. You can imagine that although with IP we could have location data in our DB - here, with addresses it is not going to be possible because we need much higher accuracy and the amount of data that is needed to achieve it is a lot bigger. You will be forced to use external service’s API.

What if you don’t want to use Google Geocoding API? There is a geopy library that provides easy API access to Nominatim geocoding information that is open source. You just need to specify the name when using it. It can be name of your app.

Let’s see how easy it is:

>>> from geopy.geocoders import Nominatim
>>> geolocator = Nominatim(user_agent="mysuperapp")
>>> location = geolocator.geocode("Warsaw Culture Palace")
>>> print(location.address)
Pałac Kultury, 1, Plac Defilad, Śródmieście Północne, Warszawa, województwo mazowieckie, 00-110, Polska
>>> print((location.latitude, location.longitude))
(52.2317641, 21.005799675616117)

In order to implement it we need to follow good REST API practices! How to implement it into django search endpoint?

First it’s important to follow a good URL naming practices:

yourapp.io/coordinates?address=QUERY_ADDRESS

In Django we can implement it like this starting from urls.py:

# urls.py

from django.urls import path
from backend.accounts.api.views import GetCoordinatesFromAddress

app_name = 'accounts'

urlpatterns = [
    # ...
     path("coordinates", # the rest will be in views: ?address=QUERY_ADDRESS
          GetCoordinatesFromAddress.as_view(), name="get-coordinates"),
]

and in views.py:

from geopy.geocoders import Nominatim
# ...

class GetCoordinatesFromAddress(ListAPIView, GeoFilterSet):
    """ Shows nearby Users"""

    def get(self, *args, **kwargs):
        # We can check on the server side the location of the users, using request
        # point = self.request.user.coordinates
        # ?address=QUERY_ADDRESS
        # QUERY_ADDRESS is the information user passes to the query
        QUERY_ADDRESS = self.request.query_params.get('address', None)

        if QUERY_ADDRESS not in [None, '']:
            # here we can use the geopy library:
            geolocator = Nominatim(user_agent="mysuperapp")
            location = geolocator.geocode(QUERY_ADDRESS)
            return Response({'coordinates': [location.latitude ,location.longitude]}, status=status.HTTP_200_OK)
        else:
            return Response({'message': 'No address was passed in the query'}, status=status.HTTP_400_BAD_REQUEST)

Now if we call our app’s API with:

https://geodjango-rest-vue-boilerplate.herokuapp.com/api/accounts/nearbyusers?address=warsaw

We get in response body:

{
  "coordinates": [52.2319581, 21.0067249]
}

There is a front end example if you want to [try it][] or [see the code][].

Adding new objects on map

🚧🏗️👷 It will be here soon!

Closest object

🚧🏗️👷 It will be here soon!

Drawing polygons on map

🚧🏗️👷 It will be here soon!

List of objects within radius

from geopy.geocoders import Nominatim
from rest_framework_gis.filterset import GeoFilterSet
from rest_framework_gis import filters as geofilters
from django.db.models import Q
from django.contrib.gis.measure import Distance
# ...

class ListNearbyUsers(ListAPIView, GeoFilterSet):
    """ Shows nearby Users"""
    model = User
    serializer_class = NearbyUsersSerializer
    pagination_class = GeoJsonPagination
    contains_geom = geofilters.GeometryFilter(name='coordinates', lookup_expr='exists')

    def get_queryset(self, *args, **kwargs):
        QUERY_ADDRESS = self.request.query_params.get('address', None)
        queryset = []

        if QUERY_ADDRESS not in [None, '']:
            queryset = User.objects.all()
            # here we can use the geopy library:
            geolocator = Nominatim(user_agent="mysuperapp.com")
            location = geolocator.geocode(QUERY_ADDRESS)

            # Let's use the obtained information to create a geodjango Point
            point = Point(float(location.longitude), float(location.latitude), srid=4326)
            # and query for 10 Users objects to find active users within radius
            queryset = queryset.filter(Q(coordinates__distance_lt=(
                point, Distance(km=settings.RADIUS_SEARCH_IN_KM))) & Q(is_active=True)).order_by('coordinates')[0:10]
            return queryset
        else:
            return queryset

Calculate distance between two points

🚧🏗️👷 It will be here soon!

Deployable boilerplate with all above

🚧🏗️👷 It will be here soon!

Did you make any mistakes when using REPLACETHIS or you’ve seen one here? Tell me about your insights. Leave a comment with YOUR opinion.

LEGO Art Creator - Mosaic and Portraits.

george_pollock — Mon, 02 Nov 2020 00:00:00 +0000

LEGO Art creator

Everything on how to make mosaic art

I present to you LEGO custom art generator that you can use for making your artwork. This mosaic maker is pure online software (you can use it offline once the page is loaded). Use your photos in ratio 1x1 and generate pixel art. LEGO Art maker trusted by thousands of people.

Click Here to enter onlineLEGO Mosaic Art maker

read more below

How to make your own art

Select an image from your device. (You can use your face picture to make art from photo - the tool is on your computer, I don't store any files you upload.)
Select size of the board and color set (you can chose e.g. LEGO World Map 2021 set colors and others)
Select if you want to use rectangle tiles or circular.
Click Generate and Convert the picture to LEGO board
You can use the edit mode. Click on the color name and then on a stud you want to change the color of.

What's new?

LEGO World Map color set. It is the biggest LEGO set so far. It means you have plenty LEGO pieces to reuse!

Now you can upload for example a geo map of north america and generate things like this:

If you don´t quite like your generated art - you can always edit it!

LEGO Portrait color set. It is the set you get in photo-booths. I got one from the very LEGO house but did not quite like the outcome. Here is the outcome from the generator I made:

Remember that you can generate those mosaics and retouch them later in edit mode to make it the way you like!

Options

The color themes - you can use pieces from different LEGO sets to reuse them and have even more fun. The themes are: Andy Warhol, Star Wars Sith, Iron Man, Mickey Mouse, Harry Potter, World Map, Mickey Mouse and custom.
Studs numbered with BrickLink IDs so you can convert image to LEGO art and download the instructions immediately.

Requests

If you need more options don't hesitate to contact me via contact form.

How did it all started?

This LEGO mosaic software was made by me for a reason. I didn't really like the way that stud.io was doing mosaics. For example for a Marlyn photo it generates this:

Not quite for a portrait maker.

It was rendering to this (in stud.io):

I didn't really like the fact the lips are black and the hair is green. Besides there's a lot of black and dark pieces around. I know you can tweak the photo hue, saturation etc but what I want is a no hustle way of doing this. That is why I created my mosaic maker app.

With my generator the result of the first photo is this:

I invite you to use it!

Why was I doing mosaics?

Many creative people are finding that creativity doesn't grow in abundance, it grows from scarcity - the more Lego bricks you have doesn't mean you're going to be more creative; you can be very creative with very few Lego bricks. - Jorgen Vig Knudstorp

This quote from Jorgen Vig Knudstorp who was a LEGO Group president made me think about creativity. I am using just my laptop to create and make my ideas reality but when it comes to LEGO you can just use a single squares 1x1 and create an astonishing pieces of art.

Making mosaic is very rewarding and you are becoming best LEGO artist when making your own canvas. Trust me - I was doing this as art therapy with LEGO. It's very calming and more creative than puzzles. It does not matter to me how much money artists make, I do that for pleasure. My friends always like the art. It looks a bit like ASCII but just like ASCII mosaic it can be a favorite one to toy with. You can make really anything, Beatels set, Star Wars Sith or Elon Musk. I made Freddie Mercury face by default to show you.

Here you have an example with edit mode that I used to add the bottom left 'HVITIS' title .

Convert your picture with online mosaic maker art now here!

This is good LEGO art alternative because you can order your own pieces on Pick a Brick and be unique. The coolest thing is that this app is always here. Save this page to your favorite so you can use it anytime you want! In 2021 you can have a new art build.

You know what is also super awesome? Once you export the generated file, you can render it in stud.io and make it 3D. The quality and level of details is amazing:

Here is Freddie in zoomed 3D without editing to show the beauty of silver, fluorescent, transparent and other colors:

If you would like to see the whole process from a software perspective you can check my other post about how I was building it.

Otherwise - let me know if you like it!

Thank you!

Enter here Mosaic Art maker and convert picture to LEGO.

or listen to me talking about the software:

Do you need more functionality in the generator? Drop me a message here. Did you learn anything reading this post? Tell me about your insights. You are most welcome to see more posts of this type - just go to home page

How to make your own lego art

You need a board
You can use your face picture to make art from photo - the tool is on your computer, I don’t store any files you upload.
Convert the picture to LEGO board

I was doing this as art therapy with LEGO. Making mosaic is very rewarding and you are becoming best LEGO artist when making your canvas. It doesn’t matter how much artists make, I do that for pleasure.

It looks a bit like ASCII but it’s my favourite toy. You can make really anything, Beatels set, Rolling Stones, Star Wars Sith or Elon Musk. I made Freddie Mercury face by default to show you.

This is good LEGO art alternative because you can order your own pieces on Pick a Brick and be unique. The mosaic maker has exporting buttons so that you can order your pieces.

The coolest thing is that this app is always here. Online. I do not collect any images or results you get with the maker.

Save this page to your favourite so you can use it anytime you want!

In 2021 - the year of staying indoors - you can have a new art build. Convert picture to LEGO art and share your results!

Click Here to test out the maker.

Django Rest Framework & Filepond ReactJS

george_pollock — Tue, 20 Oct 2020 00:00:00 +0000

Integrating DRF with Filepond ReactJS

Why? Because it is amazing out-of-the-box solution for Pythonists. Just have a quick look:

There is a Django REST Framework plugin for uploading pictures via Filepond components. It works amazing! Filepond has everything you might need for uploading pictures. Everything neatly works together and I am really happy after using it. Works on different browsers, integrates with Amazin S3 and has powerful options.

Here you can check it live in action. (It’s on heroku so you need to wait a lot to boot it up (and refresh the page if you get timedout).)

Django is known to be cumbersome when uploading pictures. Thank god we have a ready to use solution that neatly integrates with Filepond. There has been some issues when understanding the docs in my case that is why I decided to make a tutorial.

Goal: Integrate django-drf-filepond with react-filepond

What we need to do:

In the previous tutorial on Filepond VueJS I wrote more on how to use DRF extension. Here I want to show you a piece of front-end code and give you boilerplate on github that you can discover on your own.

You can also deploy it directly to heroku by clicking on the button.

( for more useful repos ready to deploy check out post on heroku buttons )

React component settings

Generally it is pretty straight-forward. You can follow the official docs at the official repo. The most important part that is omitted there is that when installing and placing the VueJS component on your front-end remember to change name to filepond. In ReactJS case it is important also to change server to our DRF endpoints fp/process/ to POST an upload.

<FilePond
  ref={ref => (this.pond = ref)}
  files={this.state.files}
  allowMultiple={true}
  allowReorder={true}
  maxFiles={3}
  server="http://127.0.0.1:8000/fp/process/"
  name="filepond"
  oninit={() => this.handleInit()}
  onupdatefiles={fileItems => {
    // Set currently active file objects to this.state
    this.setState({
      files: fileItems.map(fileItem => fileItem.file),
    })
  }}
/>

Handling all the class, functions and setState you can check out in the repo’s file if you are having problems. 💁🏻

This is how it looks like when using on deployed heroku version:

Let me know if that was helpful or if you want me to explain it better / more.

Did you make any mistakes when using Filepond or you’ve seen one here? Tell me about your insights. Leave a comment with YOUR opinion.

TUTORIAL OAuth2 Django 2 and Djoser - part 1

george_pollock — Sat, 25 Jul 2020 00:00:00 +0000

OAuth2 Django 2 and Djoser - Tutorial pt 1 - Introduction

There is repo on GitHub in case you want to clone the code.

''The truth is rarely pure and never simple.''
    - Oscar Wilde

There is so much confusion when it comes to OAuth and Django! I wasted quite a lot of time on getting lost inbetween another forks of another package, app or a library. Sadly, maintanance issues or REST non-REST packages adds to the confusion.

After making a research I decided to choose Djoser library that works quite well. Is maintained, has nice docs and could be used for OAuth2 integrations. Many libraries use social_django (social-auth-app-django) under the hood. This tutorial was made due to BETA social implementation that does not have extended documentation ( yet!).

The goal : Use 1 library to get REST extension for Django Authentication with extended User model that work with Google OAuth2.

Basically to kill two birds with one stone ( this harsh translation of polish ‘cook two meals over one campfire’ always strikes me )

Let´s get to the chase! What we will do here:

Part 1

Start the project, install libraries and apps.
Explain all neccessary settings
Explain urls
Create Google OAuth credentials ( Facebook in Part 2 ).
Use POSTMAN to test the basic flow. ( by the way here are, POSTMAN tricks and POSTMAN for Jedi posts in case you’re interested )

Part 2

In contruction… 🚧🏗️

Explain the basic flow - architecture chart - how does it work?
Implement the basic flow in VueJS
Implement the Facebook OAuth2

1. Start the project, install libraries and apps.

Start your virtual environment using virtualenv, activate it and install django:

virtualenv venvSocialDjoserDRFJWT
cd venvSocialDjoserDRFJWT
source bin/activate (on Unix)
Scripts\activate.bat (on Windows)
pip install django==2.2

Create project and one app for extending User model:

django-admin startproject SocialDjoser
cd SocialDjoser
python3 manage.py startapp accounts

Now let’s install all python-backend libraries we will use throught this tutorial ( later we also use npm-vue-js libraries ):

pip install django-cors-headers
pip install djangorestframework
pip install djangorestframework_simplejwt
pip install djoser (version 2.0.3 as of writing this tutorial)
pip install social-auth-app-django
pip install Pillow (for adding user's thumbnail - not related to social oauth)
pip install psycopg2 (JWT time )

¡Important! :

Djoser version 2.0.3 uses social_django (social-auth-app-django).
social_django uses social_core (social-auth-core) under the hood.
We use rest_framework_simplejwt (djangorestframework_simplejwt) for customizing information that comes in the JSON Web Token.

2. Explain all neccessary settings

Let’s have a look at the placings is settings:

INSTALLED_APPS = [
    'corsheaders', # I always put it first here and in Middleware!
    'django.contrib.sites', # We add this as extra
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    # Project apps:
    'accounts', # The app we have created for extending User model. Needs to go before the DRF since it has User model that DRF uses.

    'rest_framework',
    'djoser',
    'social_django', # Not needed to add but pip install required. Adding it here will create additional acces to social user via admin
    'rest_framework_simplejwt',
    'rest_framework_simplejwt.token_blacklist' # Add it to avoid problems with migrations

]

Now let’s add CORS middleware and exceptions for developing purposes. Remember you shouldn´t whitelist everything in production!

MIDDLEWARE = [
    # IMPORTANT: CORS policies has to go before other entries
    'corsheaders.middleware.CorsMiddleware',

    # IMPORTANT: Essential when using django_social
    'social_django.middleware.SocialAuthExceptionMiddleware',
]

The extra corsheaders middleware is for allowing in-browser requests.

Ok!

Now Django REST Framework, JSON Web Token and Djoser settings.

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework_simplejwt.authentication.JWTAuthentication', # OAuth2, JWT
    ),
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.AllowAny', # Up to you to decide, depends on your project
    )
}

import datetime
from datetime import timedelta
from core import settings

SIMPLE_JWT = {
    ...
    'AUTH_HEADER_TYPES': ('Bearer', 'JWT',), # Adding Bearer for POSTMAN testing
    'USER_ID_FIELD': 'id',
    'USER_ID_CLAIM': 'user_id',

    'AUTH_TOKEN_CLASSES': (
        'rest_framework_simplejwt.tokens.AccessToken',
        # 'location.to.custom.token.CustomJWTToken' # This is optional - custom class where token could be manipulated e.g. enriched with tenants, UUIDs etc.
        ),
    ...
}

TEMPLATES = [
    {
        ...
        'OPTIONS': {
            ...
            'context_processors': [
                ...

                # Essential !!
                'social_django.context_processors.backends',
                'social_django.context_processors.login_redirect',
                ...
            ]
        }
    }
]

Now comes the best part: Djoser and OAuth2!

AUTHENTICATION_BACKENDS = (
    # We are going to implement Google, choose the one you need from docs
    'social_core.backends.google.GoogleOAuth2',

    # Crucial when logging into admin with username & password
    'django.contrib.auth.backends.ModelBackend',
)

# Client ID and Client Secret obtained from console.developers.google.com
SOCIAL_AUTH_GOOGLE_OAUTH2_KEY = 'YOUR_CLIENT_ID_KEY'

SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET = 'YOUR_SECRET_KEY'

SOCIAL_AUTH_RAISE_EXCEPTIONS = False
# SOCIAL_AUTH_POSTGRES_JSONFIELD = True # Optional, how token will be saved in DB

white_list = ['http://localhost:8000/accounts/profile'] # URL you add to google developers console as allowed to make redirection

DJOSER = {
    "LOGIN_FIELD": "email", # Field we use to login on extended User model
    'SERIALIZERS': {
        'user': 'location.to.custom.serializers.CustomUserSerializer', # Custom Serializer to show more user data
        'current_user': 'location.to.custom.serializers.CustomUserSerializer', # Custom Serializer to show more user data
    },
    'SOCIAL_AUTH_ALLOWED_REDIRECT_URIS': white_list # Redirected URL we listen on google console
}

Ok! We are done with settings. These are bare minimum to make Google OAuth2 work with DRF and JWT.

3. Explain urls

urlpatterns = [
    # Original Admin panel
    path('admin/', admin.site.urls),

    # Custom JWT implementation. That's optional. It's possible to do this with Djoser settings or just use the default JWT URL
    path('api/auth/jwt/create', CustomJWTToken.as_view()),

    # REST Implementation of Django Authentication system
    path('api/auth/', include('djoser.urls')),

    # Djoser Beta extension for social_django
    path('api/auth/social/', include('djoser.social.urls')),

    # The URL that you could use for testing and that later on can be used for Front-End app Authentication.
    path('accounts/profile/', RedirectSocial.as_view()),

    # The default Djoser endpoints for JWT.
    path('api/auth/', include('djoser.urls.jwt')),
]

4. Create Google OAuth credentials ( Facebook in Part 2 ).

Enter your account and here is how your console should look like.

Here you need to put the redirection allowed URL

5. Use POSTMAN to test the basic flow.

You can download POSTMAN tests to import them and have a better understanding on how those endpoints work.

They come with descriptions and scripted flow so you could have more insights. I will show you a quick test run to obtain a token.

First run the first GET request in order to obtain URL that opens you Google Login page. Copy the link nr 2. It’s an output of script that replaces HTTPS to HTTP that this endpoint returns ( you can do it manually )

This step is just a helper, don´t login via postman preview view. Just verify in preview of the second request that you are indeed seeing the login page ( could be skipped and done directly in the browser )

With the URL copied in step 1 open your browser and navigate to it. You should see a google select profile page or if you are using incognito mode ( or you’re not logged yet to any of google accounts ) you just see this:

Finally, once logged in or account selected, you will see the outcome of the endpoint:

    path('accounts/profile/', RedirectSocial.as_view()),

which just returns JSON view with the code for POSTMAN testing purposes since we are not implementing it with front-end yet.

from django.views import View
from django.http import JsonResponse

class RedirectSocial(View):

    def get(self, request, *args, **kwargs):
        code, state = str(request.GET['code']), str(request.GET['state'])
        json_obj = {'code': code, 'state': state}
        print(json_obj)
        return JsonResponse(json_obj)

Copy the code to 3rd POSTMAN endpoint (POST) and obtain TOKEN! 🎉🎊

With this token you can authenticate within DRF!

Now seriously, go and make yourself a drink. 🍹 You deserve it!

⏭️ In the second part:

we implement those endpoints into VueJS front-end app.
we extend the template with Facebook Sign In Button.

➡️➡️➡️

Did you implement social login with Django? How did you do it? How could I make this tutorial better for you? Share your insights and leave a comment with YOUR opinion.

TUTORIAL Django OAuth2 1-click Facebook login with VueJS - part 2

george_pollock — Thu, 23 Jul 2020 00:00:00 +0000

Django OAuth2 1-click - Tutorial pt 2 - Facebook login with Djoser

In order to obtain token to access out Django backend using Facebook Social Login we need to do the following:

Set up a new app in Developers Facebook (no need to change hosts etc. you can easily redirect to localhost for testing purposes)
Define settings and secret information in settings.py
Extend the Djoser Facebook views class for customization (not necessary but might be useful for explainint purposes)
Add customized view to AUTHENTICATION_BACKENDS
Whitelist view that we will be redirected to (FrontEnd) in Djoser´s SOCIAL_AUTH_ALLOWED_REDIRECT_URIS
Make sure that urls.py are set up in a right way.
Test!

0. Setting up the app

In order to set up the app follow steps from this amazing tutorial on django_core

1. Defining scopes (!)

In order to make it work you need email address from Facebook, otherwise Django will throw an error due to registering issues (email required).

SOCIAL_AUTH_FACEBOOK_KEY = 'YOUR_FACEBOOK_APP_ID'
SOCIAL_AUTH_FACEBOOK_SECRET = 'YOUR_APPS_SECRET_KEY'

# IMPORTANT! :
SOCIAL_AUTH_FACEBOOK_SCOPE = ['email']
SOCIAL_AUTH_FACEBOOK_PROFILE_EXTRA_PARAMS = {
    'fields': 'email'
}

2. Extending views.py

# djoser/facebook.py

from social_core.backends.facebook import FacebookOAuth2

class CustomFacebookOAuth2(FacebookOAuth2):
    REDIRECT_STATE = False

3. Add view to AUTHENTICATION_BACKENDS

AUTHENTICATION_BACKENDS = (
    # Important for accessing admin with django_social
    # 'social_core.backends.google.GoogleOAuth2',
    'core.restconf.djoser.facebook.CustomFacebookOAuth2',
    'django.contrib.auth.backends.ModelBackend',
)

4. Whitelist your redirection URL

# where your DJOSER settings are
DJOSER = {
    # "LOGIN_FIELD": "email",
    # 'SEND_ACTIVATION_EMAIL': False,
    'SOCIAL_AUTH_ALLOWED_REDIRECT_URIS': [
    'http://localhost:8000/temporary-redirect-for-testing/',
    ]
}

5. Make sure about urls.py

# main/settings.py

# I made my URLs custom:
urlpatterns = [
    # ...
    path('api/auth/social/', include('djoser.social.urls')),

    # This is URL we will use for future testing in pt. 5
    # path('temporary-redirect-for-testing/', RedirectSocial.as_view()),
]

This will give us 2 endpoints to test:

GET http://localhost:8000/api/auth/social/o/facebook/?redirect_uri=http://localhost:8000/temporary-redirect-for-testing/

that returns us URL we have to enter in a browser: (note api version that you can customize via social_core settings SOCIALS_AUTH_FACEBOOK_API_VERSION - that is optional)

{
  "authorization_url": "https://www.facebook.com/v3.2/dialog/oauth?client_id=YOUR_FACEBOOK_APP_ID&redirect_uri=http://localhost:8000/temporary-redirect-for-testing/&state=THIS_IS_GENERATED_BY_DJANGOf&return_scopes=true&scope=email"
}

and

POST http://localhost:8000/api/auth/social/o/facebook/

This one requires 2 parameters:

code

state

both we obtain after calling redirect_uri from first call and giving Facebook credentials for any Facebook user.

Important to remember that this call has to be of:

Content-Type: application/x-www-form-urlencoded

Calling this endpoint with a code and state will work only for the first time. Every next time renders:

{
  "non_field_errors": ["Authentication process canceled"]
}

But successful call gives us:

{
  "access": "eyJ0eXAiOiJK...",
  "refresh": "eyJ0eXAiOiJ...",
  "user": "FACEBOOK_USER_WE_LOGGED_WITH@gmail.com"
}

5. How do we test it?

This is where a Testing endpoint would come in handy. We need the redirect view that will be called by facebook after executing authorization_url

Let´s create the view that will show us the code and state returned by facebook:

# core/restconf/djoser/views.py
from django.http import JsonResponse
from django.views import View

class RedirectSocial(View):

    def get(self, request, *args, **kwargs):
        code, state = str(request.GET['code']), str(request.GET['state'])
        json_obj = {'code': code, 'state': state}
        print(json_obj)
        return JsonResponse(json_obj)

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        print(context)
        return context

Let’s add it to urls.py

# main/settings.py

# Import it from wherever you create the view.
from core.restconf.djoser.views import RedirectSocial

# I made my URLs custom:
urlpatterns = [
    # ...
    path('api/auth/social/', include('djoser.social.urls')),
    path('temporary-redirect-for-testing/', RedirectSocial.as_view()),
]

Now, the final test:

We execute the 1st GET url
We copy authorization_url from the response to the browser.
We log in to the app
We get redirected to our testing RedirectSocial url:

🎉🎉🥳🥳 You did it!

Now you can use your code and state to call the POST url. It can be done from your Django app (if you are not using JS frameworks) or it can be called from your Front End ap using e.g. axios or fetch

var myHeaders = new Headers()
myHeaders.append('Content-Type', 'application/x-www-form-urlencoded')
myHeaders.append('Cookie', 'sessionid=m4c2oc0ahj7dibomn4a2d8uo8wmjrcy0')

var urlencoded = new URLSearchParams()
urlencoded.append('code', 'YOUR_CODE_OBTAINED_FROM_REDIRECT')
urlencoded.append('state', 'YOUR_STATE_OBTAINED_FROM_REDIRECT')

var requestOptions = {
  method: 'POST',
  headers: myHeaders,
  body: urlencoded,
  redirect: 'follow',
}

fetch('http://localhost:8000/api/auth/social/o/facebook/', requestOptions)
  .then(response => response.text())
  .then(result => console.log(result))
  .catch(error => console.log('error', error))

Dont’t forget to change redirect urls for front-end app

Did you make any mistakes when using this tutorial or you’ve seen one here? Tell me about your insights. Leave a comment with YOUR opinion or QUESTION.

The Easiest, Quickest and Most elegant way to set up Django 2 Gallery

george_pollock — Mon, 20 Jul 2020 00:00:00 +0000

How to make beautiful, responsive, gallery in Django

There is repo on GitHub in case you want to clone the code.

I was struggling many times with making my own custom Gallery or with just adding different options for photos. What is the reason of reinventing the wheel? I did not know this solution and I was tired of reviewing apps and repos that are not maintained anymore.

The lightbox itself is an old technology but it’s maintained and alive. Works very well and out of the box. You can see some examples here on how smoot it works.

The photologue is also maintained and it has amazing functionalities (see below). It also works with AWS Amazon S3 Bucket.

This is how it looks like in the Template. It’s possible navigate with keyboard between the images and the transition is great.

This is how it looks like in the Admin:

Let’s code it

To implement it in your app you need to do a few things:

Download lightbox and unzip it. I would not recommend adding CDN since there are different versions and some of them require initializing the galleries separately.
Load JS and CSS as static files. You can load a lightbox+jquery file from zipped package or you can load just lightbox JS in case you are already using JQuery.

    <!--Lightbox CSS -->
    <link href="{% static 'lightbox.css' %}" rel="stylesheet">
</head>

...
    <!--Lightbox and JQuery-->
    <script src="{% static 'lightbox-plus-jquery.js' %}"></script>
</body>

Don’t forget about

python3 manage.py collectstatic

Install photologue and extend your model that has photos/gallery. In my case it´s a blog post and FOreignKey. Your case might be different. Everything depends on DB relations you need to apply.

from django.db import models
from photologue.models import Gallery
# Create your models here.
class Post(models.Model):
    gallery = models.ForeignKey(Gallery, on_delete=models.SET_NULL, null=True)

    title = models.CharField(max_length=199)
    content = models.TextField()

    ...

Pass your models to views.

from django.shortcuts import render
from .models import Post

def homepage(request):
    post_list = Post.objects.all()
    context = {}
    context['post_list'] = post_list
    return render(request, 'base.html', context)

Add the view to URL

urlpatterns = [
    path('', homepage, name='homepage'),
]

Iterate through gallery loaded into Template context.

<div class="container">
  {% for post in post_list %}
  <h1>{{post.title}}</h1>
  <p>{{post.content}}</p>

  <!--Gallery-->
  <div class="row">
    {% for photo in post.gallery.sample %}
    <a
      class="align-self-center"
      title="{{ photo.title }}"
      href="{{ photo.get_display_url }}"
      data-lightbox="gallery-example"
      max-height="150px"
    >
      <img src="{{ photo.get_thumbnail_url }}" />
    </a>
    {% endfor %}
  </div>
  {% endfor %}
</div>

The Photologue has get_display_url for big photos and get_thumbnail_url for thumbnails. You can customize their sizes , effects and more in your Django admin!

The only thing to make lightbox 2 work is part of HTML tag:

data-lightbox="NAME_OF_YOUR_GALLERY"

so don´t miss it! Also in case you want to have more than one galleries remember to change those tags e.g. :

<!--first galery with 3 photos:-->

<a href="{{ photo.get_display_url }}" data-lightbox="FIRST_GALLERY" max-height="150px"><img src="{{ photo.get_thumbnail_url }}" ></a>
<a href="{{ photo.get_display_url }}" data-lightbox="FIRST_GALLERY" max-height="150px"><img src="{{ photo.get_thumbnail_url }}" ></a>
<a href="{{ photo.get_display_url }}" data-lightbox="FIRST_GALLERY" max-height="150px"><img src="{{ photo.get_thumbnail_url }}" ></a>

<!--second galery with 2 photos:-->

<a href="{{ photo.get_display_url }}" data-lightbox="SECOND_GALLERY" max-height="150px"><img src="{{ photo.get_thumbnail_url }}" ></a>
<a href="{{ photo.get_display_url }}" data-lightbox="SECOND_GALLERY" max-height="150px"><img src="{{ photo.get_thumbnail_url }}" ></a>

I hope this 🖼️ Gallery solution came in handy!

Did you make progress when using this post? Tell me about your insights. Leave a comment with YOUR opinion.

REST API Best practices - Why's and How's

george_pollock — Wed, 24 Jun 2020 00:00:00 +0000

Why your API should be REST ?

REST helps. A lot. Let’s focus on what REST gives your team. The picture I’ve chosen respresents the idea very well. In a crowd of backenders, front-enders, clients, users and testers - well design API increases productivity and efficiency, decreases frustration, let’s people cooperate better. In case you’re a bit confused on what’s REST try improving your knowledge before continuing.

If you are a front end developer you may want to save yourself writing custom messages for bad error requests. You may want to have a reusable functions for services and getting data, this shines when things are consistent. When requests you make are predictible, when their codes represent the actual state.

If you are a back end developer you may want to check your services by calling them while developing. It’s easier to debug if you can actually read in a message what did just happen instead of seeing empty body. Coding is not guessing, requests responses should be self explanatory.

If you are a tester then you should make everybody follow the REST practices, otherwise you will go nuts. Backend API testing tools like POSTMAN could ease your work, especially if you know some tricks. If you want to perform server side automation testing and write set of tests for the API - it will be hard if all you get is unconsistency in HTTP responses. Even if your API testing resume says: automation tester - it only makes sense if the API is well designed.

If you are a user - you have a limited contact with developers, maybe the only thing that could help you is documentation. Well designed API could benefit from rest api documentation tools like SWAGGER, Sphinx or POSTMAN but the best documentation won’t have all the possible error you may encounter when using API. That is why for the sake of the future and potential integrations or exposing your API to customers, please, design well!

How to design well ?

Here is the list of must haves:

Error handling
Methods
Naming consistency
Resources
Returning objects
Slash redirect
Status codes
Query parameters
Versioning

Error handling

In order to provide more specific machine-readable messages with an error response, the API clients can react to errors more effectively and eventually it makes the API services much more reliable from the REST API testing perspective. Clients could benefit a lot from a good error handling since they can use the message strings comming directly from backend ( that eases translation issues ).

For non-success conditions, developers SHOULD be able to write one piece of code that handles errors consistently across different Microsoft REST API Guidelines services.

This quote comes from Microsoft’s REST API design best practices (it’s a great place to learn although it often shows design examples of api architecture for a huge project).

The quote says you need a wrapper. Some class or function that will be responsibe for showing a consistant error message for non-success requests. One important reminder: Error responses MUST use standard HTTP status codes in the 400 or 500 range to detail the general category of error.

In case you are developing a huge system then go back and take a look at Microsoft’s example. My take on this for smaller projects would be as follows:

Error : Object

Property	Type	Required	Description
`code`	String	✔	One of a server-defined set of error codes.
`target`	String		The target of the error.
`message`	String	✔	A human-readable representation of the error.
`details`	details[]		An array of details about specific errors that led to this reported error.

Details : Object

Property	Type	Required	Description
`code`	String	✔	A more specific error code than was provided by the containing error.
`target`	String		The target of the error.
`message`	String	✔	A human-readable representation of the error details.

The simple design example example:

{
  "error": {
    "code": "WrongMediaType",
    "message": "Data types accepted: csv, json."
  }
}

(this message should be returned along with status code 413)

Other extended example:

{
  "error": {
    "code": "BadArgument",
    "target": "GetAllCatsService",
    "message": "Data introduced has 1 error.",
    "details": [
      {
        "code": "NullValue",
        "target": "CatWeigth",
        "message": "Cat's weight must not be null"
      }
    ]
  }
}

The way you name your key&value pairs in returned object is totally up to you. The way you construct your wrapper is also your choice although you may want to use one of existing libraries.

SUM_UP: Error handling is extremely important for backend automation testing and usability, it also could bring more profits for clients. It is possible to start small with a simple error / message object that could be extended with time and complexity.

Methods

There is many HTTP methods but you should use at least those:

GET
POST
PUT
DELETE
PATCH (You should know what is the difference between POST/PUT/PATCH)

Using methods is strictly connected with naming consistency because the method themselves are verbs which is a direct reason why you should avoid using verbs in your URLs. Instead of duplicating code and making it unmaintainable and unintuitive - use methods well.

GET: https://yourdomain.com/getCats ⬅️🛑Wrong
GET: https://yourdomain.com/cats ⬅️✅ Good

POST: https://yourdomain.com/getCats ⬅️🛑Wrong
GET: https://yourdomain.com/cats ⬅️✅ Good

GET: https://yourdomain.com/getCats ⬅️🛑Wrong
GET: https://yourdomain.com/getCat/{id} ⬅️🛑Wrong
POST: https://yourdomain.com/postCats ⬅️🛑Wrong
DELETE: https://yourdomain.com/deleteCats ⬅️🛑Wrong

GET: https://yourdomain.com/cats ⬅️✅ Good
GET: https://yourdomain.com/cats/{id} ⬅️✅ Good
POST: https://yourdomain.com/cats ⬅️✅ Good
DELETE: https://yourdomain.com/cats ⬅️✅ Good

There is more methods and more ON methods. Read-up before continuing!

Difference between POST/PUT/PATCH

It’s pretty simple:

POST creates a new resource that did not exist before:

POST:
URL: https://yourdomain.com/cats/
REQUEST:

{
  "name": "George",
  "color": "black",
  "is_adopted": true
}

RESPONSE CODE: 201
RESPONSE:

{
  "id": 1,
  "name": "George",
  "color": "black",
  "is_adopted": true
}

PUT updates resource previously created with a new object information:

PUT:
URL: https://yourdomain.com/cats/{catId}
REQUEST:

{
  "name": "Bob",
  "color": "white",
  "is_adopted": false
}

RESPONSE CODE: 200
RESPONSE:

{
  "id": 1,
  "name": "Bob",
  "color": "white",
  "is_adopted": false
}

PATCH updates resource previously created with partial information:

PUT:
URL: https://yourdomain.com/cats/{catId}
REQUEST:

{
  "name": "Zoey"
}

RESPONSE CODE: 200
RESPONSE:

{
  "id": 1,
  "name": "Zoey",
  "color": "white",
  "is_adopted": false
}

Naming consistency

It’s simple:

URIs resources as nouns - do not use verbs in your URI

GET: domain.com/getCats ⬅️🛑Wrong
GET: domain.com/getCat/{id} ⬅️🛑Wrong
POST: domain.com/postCats ⬅️🛑Wrong
DELETE: domain.com/deleteCats ⬅️🛑Wrong

GET: domain.com/cats ⬅️✅ Good
GET: domain.com/cats/{id} ⬅️✅ Good
POST: domain.com/cats ⬅️✅ Good
DELETE: domain.com/cats ⬅️✅ Good

Pluralized resources

GET: domain.com/cat ⬅️🛑Wrong
GET: domain.com/cat/{id} ⬅️🛑Wrong

GET: domain.com/cats ⬅️✅ Good
GET: domain.com/cats/{id} ⬅️✅ Good

Lowercase letters and dashes

GET: domain.com/Cats ⬅️🛑Wrong

GET: domain.com/cats ⬅️✅ Good

---

GET: domain.com/cat_races ⬅️🛑Wrong

GET: domain.com/cat-races ⬅️✅ Good

(American) English

GET: domain.com/gatos ⬅️🛑Wrong

GET: domain.com/cats ⬅️✅ Good

No file extensions

GET: domain.com/cats.pdf ⬅️🛑Wrong

GET: domain.com/cats ⬅️✅ Good

Do not use trailing slashes

GET: domain.com/cats/ ⬅️🛑Wrong

GET: domain.com/cats ⬅️✅ Good

Snake_casing for query parameters

GET: domain.com/cats?skinColor=black ⬅️🛑Wrong

GET: domain.com/cats?skin_color=black ⬅️✅ Good

Resources

They should be flat, no nesting.

GET: domain.com/animals/{id}/family/{id}/colors/{id}/cats ⬅️🛑Wrong

GET: domain.com/cats ⬅️✅ Good

If nested is required it should be shallow.

GET: domain.com/animals/{id}/family/{id}/colors/{id}/cats ⬅️🛑Wrong

GET: domain.com/animals/{id}/cats ⬅️✅ Good

Returning objects

It depends on your client and architecture but in most cases it’s good to return a body on creating resources.

{
  "result": {
    "cat": {
      "id": 1,
      "name": "Georgina"
    }
  }
}

You could think about wrapping the responses with additional info but you need to remember that certain requests should return empty body e.g. DELETE requests. Once the resource has been deleted and status code 204 returned then body should be empty. In most cases the decision on body of successful requests is up to you or your front-end collegue.

Slash redirect

As part of naming consistency: do not use trailing forward slashes. In case of using slashes e.g. by your users (mistakes / no docs etc) you should try to redirect to original resource. It’s an interesting topic, try reading more about it.

“The trailing slash must not have specific semantics. Resource paths must deliver the same results whether they have the trailing slash or not.” - Zalando’s RESTful API guidelines

Status codes

Like with HTTP methods there is many of them. Try starting from the basics:

200s

HTTP_200_OK
HTTP_201_CREATED
HTTP_204_NO_CONTENT

300s

HTTP_301_MOVED_PERMANENTLY
HTTP_302_FOUND

400s

HTTP_400_BAD_REQUEST
HTTP_401_UNAUTHORIZED
HTTP_403_FORBIDDEN
HTTP_404_NOT_FOUND
HTTP_405_METHOD_NOT_ALLOWED
HTTP_408_REQUEST_TIMEOUT
HTTP_415_UNSUPPORTED_MEDIA_TYPE

Important to know the difference between 401 and 403 :

401 - No credentials, no token, unauthenticated

402 - No authorization for the resource, role not correct, not enough eprmissions. Basically e.g. you may want to access admin resources as a simple user.

Query parameters

Query resources via url parameters:

domain.com/cats?name=georgina&color=black

Stick with conventional query parameters.

Versioning

If you are not working on a mobile app or in a bigger team you can skip it for now. This is very important if you are planning an API for a mobile app because app updates take long time to develop or deliver and it’s difficult to synchronise the changes with the backend. In case you are starting a project in a bigger team or a company, versioning is also extremly important since it allows more agile development - backenders don´t have to wait for front-enders. You can deploy your new API without worrying that you’ll break the client’s app. There is many REST API verioning strategies, at the end the goal is to maintain backward compatibility and avoid code duplication.

There is multiple possibilities of versioning schemes for your API:

Versioning via URL

urlpatterns = [
    url(
        include('v1/registration',
        register_via_email
    ),
    url(
        include('v2/registration',
        register_via_oauth
    )
]

Versioning via hostname:

https://v1.yourdomain.com/registration/
https://v2.yourdomain.com/registration/

Versioning via parameter:

https://yourdomain.com/registration/?version=1
https://yourdomain.com/registration/?version=2

Restful API best practices when it comes to way of versioning API - doesn’t exist. Remember that your case might be different and your needs differ too - choose what is the best solution for you depending on the circumstances. Google has a great source on versioning API, you should check it out before laying first lines of code.

Once you use versioning you can benefit from it like this:

def registration_type(self):
    if self.request.version == 'v1':
        return register_via_email(self.body.email, self.body.password)
    return register_via_oauth(self.body.token)

SUM_UP: API Versioning is a great way to implement changes to code but it should be used if other methods are unavailable, use it conciously to not introduce unnecessary complexity.

Did you make any mistakes when using REST API or you’ve seen one here? Tell me about your insights. Leave a comment with YOUR opinion.

How to implement reCAPTCHA v3

george_pollock — Fri, 12 Jun 2020 00:00:00 +0000

reCAPTCHA v3 - prepare yourself for bots!

I will just quickly explain you what is reCAPTCHA, give you some explanation on how it works and we will go to examples and usage.

What and how ? 👨‍🎓

reCAPTCHA is a technology that assesses probability that the entity that uses your web-code (page, app, portal etc) is a human and not a bot ( or the other way around ). Grabbing information of behaviour (of a user or a bot) it encapsulates it in the token that gets send to your server. On your server, the token is being send again to Google for returning the assessment on how probable is that the token was generated by a human. Part of response returned from Google to your server:

  "score": number // the score for request (0.0 likely bot - 1.0 likely human)

Having this information you could use it to later in your logic e.g. to return 404 for suspicious requests.

It uses AI and information gathering under the hood which may be disliked by certain people. You can learn more on that matter and check the history & usage of reCAPTCHA to get better knowledge on how to use it and have your own opinion. Maybe you don´t really want to use it?

Last but not least, quote from docs on where to use it and where it can get usage data from:

For this reason, we recommend including reCAPTCHA verification on forms or actions as well as in the background of pages for analytics.

Let´s implement it - example and usage 🛠️

Have a quick look at the flow we will go through:

We will be following the programmatic steps from the official guide

GENERATE

To generate the token first you need to generate the SECRET_SITE_KEY and SECRET_KEY. Then the steps are straightforward:

Add the JS script to the page (index.html). Front end frameworks have a folder called public that has index.html file inside. Open it and put the script on the bottom of the body, just like you would be adding bootstrap.

<body class="h-100">
  ...
  <script src="https://www.google.com/recaptcha/api.js?render=YOUR_SIT_KEY"></script>
  <div id="app"></div>
</body>

Add the function to methods available in the component.

 methods: {
    verifyCaptcha() {
      e.preventDefault();
      grecaptcha.ready(() => {
        grecaptcha
        // Don't forget about putting your unique SECRET_SITE_KEY.
          .execute("SECRET_SITE_KEY", {
            action: "submit"
          })
          .then(token => {
            // Try loggin the token first time to see if it got generated.
            console.log(token);
            // Add your logic to submit to your backend server here.
            // Here we will send the token to our backend using axios (install and import it to the component).
            // You could also use a Vue built in $http
            const body = { token: token };
            axios
            // We are POSTing to a dedicated URL we have on backend (it can be any URL that will need the reCAPTCHA token)
              .post("http://localhost:8000/api/verify-captcha/", body)
              .then(response => {
                // THe is_human part is our custom parameter comming from our backend. You can call it other name or just return something completly different. This is 100% to developer how will the reCAPTCHA returned probability be used.
                if (response.data.is_human) {
                 // Your front-end logic fired-up on success.
                } else {
                 // Your front-end logic fired-up on error.
                }
              })
              .catch(error => {
                console.log("Error : ", error);
              })
              // Boolean that could e.g. stop page loader.
              .finally(() => (this.loading = false));
          });
      });
    }
  }

Add the method to an element on your page (the method where you execute reCAPTCHA token generation - verifyCaptcha - in our case)

<b-button @click="verifyCaptcha">Submit</b-button>

( Here I use Buefy library for VueJS but your button tag may be called different way or just )

VERIFY

Verification is happening on backend, since we want to use the assessment to decide what to do on that particular request. Should we let the request add e-mail address to DB or not etc., any kind of server-logic.

Let’s have a look at the quote from docs:

reCAPTCHA tokens expire after two minutes. If you’re protecting an action with reCAPTCHA, make sure to call execute when the user takes the action rather than on page load.

What does it mean? It means that we can generate token on Front End > console.log it in developer tools > copy it and use in POSTMAN using Google’s endpoint ( or our backend but this makes sense only if axios or communication front-end==back-end doesn´t work for some reason ).

Let’s start to test the token generated on front-end. The URL for checking it is:

METHOD: POST
https://www.google.com/recaptcha/api/siteverify

and we need to provide a following body:

{
  "response": "TOKEN_GENERATED_ON_FRONT_END_ACTION",
  "secret": "YOUR_SECRET_KEY"
}

or we could put those values as encoded form:

This very request is the request you have to send from your backend to later use the score value in your logic.

I wanted to start with POSTMAN because it has very useful feature that let’s you export request in any language you want. I will export it in python´s requests:

If you execute this script in Python e.g. in CMD / Bash you will get this:

{
  "success": true,
  "challenge_ts": "2020-06-13T22:58:18Z",
  "hostname": "localhost",
  "score": 0.9,
  "action": "submit"
}

Let´s jump into Django now.

PROCESS & RETURN

I will show you that in one code snippet. For learning purposes everything shown here is as simple as possible. In real life you would have a process function somewhere in a separate file and custom return afterwards.

As I mentioned, verifying will be done in a custom, dedicated endpoint but you can do it in any of your endpoints (as long as you pass the token there)

# urls.py

urlpatterns = [
    # ...
    path('api/verify-captcha/', VerifyCaptcha.as_view(), name='verify-captcha'),
]

# api.py

from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status

import json
import requests

YOUR_SECRET_KEY = "YOUR_SECRET_KEY"
GOOGLE_URL = 'https://www.google.com/recaptcha/api/siteverify'

class VerifyCaptcha(APIView):
    """Calls Google endpoint to verify front-end TOKEN and returns information whether user is a human.
    """
    # ...

        # This is POST endpoint that we were calling with axios in the first front-end example.
    def post(self, request, *args, **kwargs):
        # Decoding the payload
        request = json.loads(request.body.decode('utf-8'))
        # Taking out the token from the payload
        TOKEN_GENERATED_ON_FRONT_END_ACTION = request['token']
        # Creating body to send to Google for verification. You could also pass user's IP as an optional parameter but I never do that.
        body = {
            "secret": YOUR_SECRET_KEY,
            "response": TOKEN_GENERATED_ON_FRONT_END_ACTION
        }
        # Sending the request
        r = requests.post(GOOGLE_URL, data=body, json=body,)
        # Receiving the response
        google_response = r.json()

        # Analyzing the response
        if google_response['success'] == True:
            # Preparing our response that will be send to our front-end
            response = {"is_human": True}

            # This is our custom logic in case the request was initiated by a bot.
            if google_response['score'] < 0.5:
                response['is_human'] = False
            return Response(
                data=response, status=status.HTTP_200_OK, content_type="application/json")
        else:
            return Response(
                data={"is_human": None, "message": "Validation of FE token went wront."}, status=status.HTTP_404_NOT_FOUND, content_type="application/json")

In this case official docs give you all the info on possible responses from Google

You did it! 👏

Now you can avoid bots 🤖

Did you make any mistakes when using reCAPTCHA or you’ve seen one here? Tell me about your insights. Leave a comment with YOUR opinion.

5 ways to improve your automated POSTMAN tests.

george_pollock — Sat, 23 May 2020 00:00:00 +0000

Improve your POSTMAN tests.

The methods I am sharing in this article allow me to prepare more robust tests. Some example use cases would be:

Wondering how to create random number or random name for a variable? Need to use some jQuery for parsing HTML? Try using 3rd party libraries in POSTMAN sandbox.
Executing POST with body that needs to be changed? No problem! Try to make some changable body conent by preparing it in pre-requests.
Your requests just return a rendered page HTML? Try parsing it and scrapping for the data you need ( tokens for example ).
Your request response makes the next request redundant? You want to execute previous request? Queue execution of specific request.
You are recieving big objects of important data? Verify just its format by validating with schema.

Using 3rd party libraries in pre scripts

You want to create random numbers? use lodash library? jQuery? Chai assertions?Maybe you need some high level time operations for your scripts? MomentJS? No problem.

Passing, parsing URLs
Validating Schemas
Decoding, encoding keys, tokens with SHA256 (atob, btoa)

No problem. POSTMAN allows you to use 3rd party libraries in its sandbox. Next tips will be tightly connected with this information, so have as look before proceeding!

Passing prerequest to body

Let’s say SWAGGER tells you that in body you need to pass string as title:

You can programaticaly change your body content in pre-request scripts so that e.g. each time you make a request you have different Job title.

First open your pre-request scripts tab, prepare your body and stringify it.

(code version in case you need to copy)

const title = "Boss' Assistant nr " + _.random(0, 10)

const preparedBody = {
    "title" : title
}

pm.environment.set("job_object", JSON.stringify(preparedBody))

You have to set this body object as an environment, global or collection variable. Once you do it, before executing the script you just need to use it in body like that:

Now each time you execute this request you will get new Job title:

"title": "Boss' Assistant nr 5"
...
"title": "Boss' Assistant nr 2"
...
"title": "Boss' Assistant nr 6"

Parsing data from HTMLs forms

General use case: There is some visible or hidden data within the HTML tags on the rendered page that you need to pass / verify / check.

Specific use cases: You have a page that has a login fields with generated token that you need for next request.

Let’s say that you make a request to some API and it redirects you to a page with input fields that have tokens assigned. That’s a common use case with SAML and SSO. You can practice here on a real use case. This login field has a token.

On requesting the URL of this page we will get just normal HTML:

As you’ve read in a previous tip, we can use libraries provided in POSTMAN SandBox. jQuery is replaced by CheerioJS which emulates jQuery core API.

Let’s load the library and see if it’s working:

const $ = cheerio.load(pm.response.text())
console.log('Title of the page: ', $('title').text())

Now you just need to use your selectors magic knowledge. Let’s query for the TOKEN we need:

console.log($('input[name=authenticity_token]').attr('value'))

( TIP 1: If you are not good with selectors, click right mouse button on element you are looking for and then copy selector )

( TIP 2: If you need other information from HTML elements, check the Cheerio documentation for more. )

And finally let’s write the test and export the TOKEN:

pm.test('input field it should have a token', () => {
  let TOKEN = $('input[name=authenticity_token]').attr('value')
  pm.expect(TOKEN).to.not.be.empty
  pm.environment.set('TOKEN', TOKEN)
})

Now you can use the TOKEN in other requests.

SetNextRequest

I’m assuming you know what is a postman collection. It’s a set of requests or folders with requests, depending how you order your work. The basic workflow that POSTMAN uses is checking request one after the other. It goes from the top to bottom basically. Most of this is well described already but I will just mention that with

postman.setNextRequest('REQUEST_NAME_HERE')

You can go back to previous requests or skip some requests or loop through requests 7. It’s IMPORTANT to remember that naming convention in your requests that follows e.g. CamelCase and has consistent information helps tremendously because it makes your code cleaner.

Instead of using default naming:

Try naming your request in a way that will be easier to write them in the scripts when passing their names into your setNextRequest() e.g. using request method and no spaces:

Validating schema

Imagine you make a test for checking if your API returns a Bank Account number.

const jsonData = pm.response.json()
pm.test('returns IBAN as a number', () => {
  const IBAN = jsonData.IBAN
  pm.expect(IBAN)
    .to.be.an('number')
    .that.does.not.include('ES')
})

You assert that it should be a number and it works fine but what if you would need to be sure as per data types of an entire response of object with 50 keys? Using assertions would be cumbersome.

That’s a use case for schema validation. POSTMAN allows us to use

tv4 - Tiny Validator (for v4 JSON Schema) and / or
Ajv - Another JSON Schema Validator

The documentation for both libraries shows all the API you could use. I will just show you the basic idea behind it:

// We create a schema for our response
var schema = {
  IBAN: {
    type: 'number',
  },
}

and write a test for it:

const jsonData = pm.response.json()
pm.test('Schema is valid', () => {
  pm.expect(tv4.validate(jsonData, schema)).to.be.true
})

We could verify properties, whether field is required, nested objects, referencing and many more. Always remember to read documentation before starting to know the full potential of your tools!

Did you make any mistakes when using POSTMAN or you’ve seen one here? Tell me about your insights. Leave a comment with YOUR opinion.

Your Web presence with Django.

george_pollock — Sat, 11 Apr 2020 17:26:32 +0000

Have you ever made a curriculum vitae? If you did then probably you’ve started with a template. In most of the templates there is always a ‘web’ place for your website. Setting up may be cumbersome for many but there is many services that solve that. Here you can use ( and work on! ) a template to set up with literally 2 clicks.

We are focusing here mostly on people who either want to start a podcast/blog or want to start programming in Python. In case you just want to start a podcast and are looking for a website then we are here for you.

In case you code — stay and read along!

Your Junior CV references can be better

Great way to boost your curriculum is to start working on a project. To go out from “while True: do online courses” loop. I’m not saying another to-do list. I’m saying something that can be useful, used and visible.

What experience do you have, Sir?

Well, […] and I was contributing to an open-source project that you can see on my website.

Contributing to open-source is a great way to learn, leave your knowledge out there and let people use it.
You can search for good first issues directly on GitHub or you can use apps like this CLI one or this one — that aim to help you find those. Both solutions have their advantages but in case you didn’t find anything or your level is not that high yet (or you simply like my idea) then you can find some open issues for Django beginners and start using your knowledge!

Your online presence counts

Recruiters are very well aware of the fact that people lie on their resume. Fortunately in IT the easiest way to verify actual knowledge is to click on a link in your resume and see with their own eyes. Amount of contributions, users or just the look of your page. Nowadays personal website is essential in your online presence.
Even if you don’t like Podcaster idea and you’re not intending to contribute, there are many other platforms where you can start your page or blog in seconds. Try them out or build your website from scratch. You won’t regret it!

What you lose by not trying

Coding together ( pair programming ) as well as testing together ( peer testing ) can be profitable. Peer activities are profitable not only according to our experience but to studies. As long as you don’t have other important side project or you’re not exchanging your time for money, reading other people’s code can bring you value.
Give it a shot and search for some Good First Issue today. In the worst case scenario you can find an interesting, free software that can ease your life.