DEV Community: Kanthaliya

Validating and Sanitizing user inputs on python projects REST api

Kanthaliya — Tue, 06 Oct 2020 06:18:09 +0000

Validation

User input data validation is one of the most important things while developing a project. It not only keeps the data clean but also helps with somewhat malicious data being sent with requests using intercept tools like burp suite.

One of python package which helps in validating Api request is Schema

from schema import Schema, And, Use, Optional

schema = Schema([{'name': And(str, len),
                  'age':  And(Use(int), lambda n: 18 <= n <= 99),
                   Optional('gender'): And(str, Use(str.lower),
                                           lambda s: s in ('Male', 'Female'))}])

data = [{'name': 'Pritesh', 'age': '29', 'gender': 'Male'},
        {'name': 'Alisha', 'age': '26', 'gender': 'Female'},
        {'name': 'Atul', 'age': '28'}]

validated = schema.validate(data)

If validation fails It raises SchemaError else it would return filtered payload based on schema validation.

There are many features of Schema we can use, few of them are -

Optional keys can also carry a default, to be used when no key in the data matches: eg:

Schema({Optional('best_songs', default='blues'): str, 'best_movie': str}).validate({'best_movie': 'shawshank redemption'})

In a dictionary, you can combine two keys in a “one or the other” manner. To do so, use the Or class as a key

Schema({  Or("key1", "key2", only_one=True): str })

The Schema(...) parameter ignore_extra_keys causes validation to ignore extra keys in a dictionary, and also to not return them after validating.

Schema({'movie': str}, ignore_extra_keys=True)
print(schema.validate({'movie': 'tenet', 'review': '4'}))
{'movie': 'tenet'}

You can pass a keyword argument error to any of validatable classes (such as Schema, And, Or, Regex, Use) to report the error instead of a built-in one.

Schema(Use(int, error='Invalid year')).validate('2020')

Sanitization

Once user inputs are validated, data needs to be sanitized with an HTML sanitizing library that escapes or strips markup and attributes. Bleach
Adding sanitization helps in eliminating XSS attacks on application.

import bleach
bleach.clean('an <script>evil()</script> example')
u'an &lt;script&gt;evil()&lt;/script&gt; example'

Advance Mongodb queries - cheatsheet

Kanthaliya — Sat, 04 Jul 2020 16:27:40 +0000

Here are few less used but important Mongodb queries I use now and then. Will keep updating this page If I get to use different less used queries.

Copy database

db.copyDatabase("dbname", "dbnamenew")

Drop database

use dbname
db.dropDatabase()

Active operations

db.currentOp() or db.adminCommand( { lockInfo: 1 } )

Kill operation

db.killOp(opid)

Query stats

db.collection.find({"status":"single"}).explain( "executionStats")

DB stats (in GB)

db.stats(1024*1024*1024)

Create index

db.collection.createIndex( { status: 1 } )

Get indexes on collection

db.collection.getIndexes()

Create read only access for user on mongodb

db.createUser({user: "example_read",pwd: "12345", roles: [{role:"read", db: "dbname"}]})

Get user roles on mongodb

db.getUser("user_name")

Export collection

mongoexport --host=10.0.0.1:27017 --username=example --authenticationDatabase=admin --collection=example_mapper --db=dbname --out=example_mapper.json

Import from collection

mongoimport --host=10.0.0.1:27017 --username=example --authenticationDatabase=admin --collection=example_mapper --db=dbname --file=example_mapper.json

Dump mongodb

mongodump --host=10.0.0.1:27017 --username=example --authenticationDatabase=admin --db db_example --out dumps/

Restore mongodb from dump

mongorestore --host=10.0.0.1:27017 --username=example --authenticationDatabase=admin  dumps/

Shell connect with ssh

mongo --ssl --host docdb.amazonaws.com:27017 --sslCAFile rds.pem --username example --password asddasdasd

Build Intelligent Chatbot using Dialogflow

Kanthaliya — Sun, 28 Jun 2020 17:57:22 +0000

Today chatbots have become a very important part for most websites, not only to answer FAQs or provide customer service but also to give intelligent service for website navigation. There are lots of chatbot tools available in the market from Amazon Lex, Google Dialogflow, to open source RASA, etc. We are going to look at how we can create chatbot using Google's Dialogflow and python cloud function.

Dialogflow is an end-to-end, build-once deploy-everywhere development suite provided by Google for creating conversational interfaces for websites, mobile applications, popular messaging platforms, and IoT devices.

Dialogflow Setup -

Agent - First create a new Agent on dialogflow. Provide general information related to your project requirement under settings tab.
Intent - It categorizes an end-user's intention for one conversation turn. When a user says something, dialogflow matches with best intent in the agent.
Paraphrases - Training phrases are example phrases for what end-users might type or say. Each intent can have multiple Paraphrases.
Action and Parameters - The action field is a simple convenience field that assists in executing logic in your service. When an intent is matched at runtime, Dialogflow provides the extracted values from the end-user expression as parameters, which we can use in service triggered by using the above action field.
Response - Each Intent has a response handler that can return responses after the intent is matched. we can set text or different response formats provided by dialogflow.

Detect Intent -

Deploy small python program on cloud function which detect intents for user queries. you also have to get access key of dialogflow to run it from cloud function. Download it from service accounts in IAM and Admin.

requirements.txt

Flask
dialogflow
functions-framework

main.py

import dialogflow
import json, logging
from flask import request, jsonify
from google.protobuf.json_format import MessageToJson
from google.protobuf.struct_pb2 import Struct
from google.protobuf.struct_pb2 import Value

def detect_intent(request):
    response = {
        'status_code': 500,
        'message': 'Failed to fetch data from dialogflow.',
        'data': []
    }

    try:
        request_json = request.get_json(silent=True, force=True)

        project_id = "<project id>"
        session_client = dialogflow.SessionsClient.from_service_account_json('key.json')

        payload = {'project_id': project_id, 'session_id': request_json.get("session_id")}
        struct_pb = Struct(fields={
            key: Value(string_value=value) for key, value in payload.items()
        })
        params = dialogflow.types.QueryParameters(
            time_zone="IST", payload=struct_pb)
        session = session_client.session_path(project_id, request_json.get("session_id"))

        text_input = dialogflow.types.TextInput(
            text=request_json.get("text"), language_code='en-US')
        query_input = dialogflow.types.QueryInput(text=text_input)
        df_response = session_client.detect_intent(
            session=session, query_input=query_input, query_params=params)

        # Convert proto object and serialize it to a json format string.
        json_obj = MessageToJson(df_response)

        result = json.loads(json_obj)
        response['status_code'] = 200
        response['data'] = result
        response['message'] = "Successfully fetched response from dialogflow."
    except Exception as e:
        logging.error(e)

    return jsonify(response)

Webhook Call -

For each Intent if you want to process some business logic using parameters extracted from query, you enable webhook call under fulfillment in Intent. Add webhook api endpoint in the fulfillment tab.

requirements.txt

pydialogflow-fulfillment
Flask
functions-framework

main.py

import json
from flask import request
from pydialogflow_fulfillment import DialogflowResponse, DialogflowRequest
from pydialogflow_fulfillment.response import SimpleResponse, OutputContexts
import logging

def webhook(request):
    dialogflow_response = DialogflowResponse()
    try:
        dialog_fulfillment = DialogflowRequest(request.data) 
        action = dialog_fulfillment.get_action()
        if action == "welcome":
            dialogflow_response.add(SimpleResponse("This is a simple text response","This is a simple text response"))
        else:
            dialogflow_response.add(SimpleResponse("This is a fallback text response","This is a fallback text response"))

    except Exception as e:
        logging.error(e)
        dialogflow_response = DialogflowResponse()
        dialogflow_response.add(SimpleResponse("Something went wrong.","Something went wrong."))
        return json.loads(dialogflow_response.get_final_response())

    return json.loads(dialogflow_response.get_final_response())

Download code from github

Nodejs cloud function deployment with private npm packages

Kanthaliya — Sat, 13 Jun 2020 17:38:29 +0000

Cloud Functions is an event-driven serverless compute platform from Google. You can run your code locally or in the cloud without having to provision servers. There are many ways to run nodejs application on cloud function. we are going to look for zip upload and deploy code from terminal.

On Google cloud platform select cloud function and create function.

Fill the required information

Name - function name for your program.
Memory allocated - as required by function processing.
Trigger - select Http
Source code - zip upload.
Runtime - nodejs 10
Function to execute - It is a name of a function exported by the module specified in the directory with source code.
Check Advance Options if you want more control.

and finally upload a zipped nodejs application.

To deploy nodejs application with private npm package, include .npmrc file at root level -

//npm.private.com/:_authToken="<token>"
@ng-test:registry=https://npm.private.com/

you can read private npm package auth token from ~./npmrc or login with npm login --registry=https://registry.company-name.npme.io and npm token list.

In package.json add private package name in dependencies and Google functions-framework in dev dependencies.

{
    "name": "notification-service",
    "version": "1.0.0",
    "description": "Notification Service",
    "main": "index.js",
    "author": "Pritesh Kanthaliya",
    "license": "UNLICENSED",
    "scripts": {
        "start": "npx @google-cloud/functions-framework --target=notificationService",
        "deploy": "npx gcloud functions deploy notificationService --runtime nodejs10 --trigger-http",
    },
    "dependencies": {
        "@ng-test/hello-world": "1.0.0",
    },
    "devDependencies": {
        "@google-cloud/functions-framework": "~1.5.1",
    }
}

With npm start you can run nodejs application at local machine and test. With npm deploy you can deploy your code on Google cloud function. If a cloud function is already present with the same name, it overrides the current one else it would create a new one.