DEV Community: sivakon

Easy concurrent scraping with Scala with Akka streams

sivakon — Sat, 29 Feb 2020 11:15:26 +0000

Step 1: Download ammonite

# Download Ammonite
$ curl -L https://github.com/lihaoyi/ammonite/releases/download/2.0.4/2.12-2.0.4-boostrap > amm212 && chmod +x amm212

Step 2. Start ammonite

$ amm212 --class-based

Step 3. We need akka-streams

import $ivy.`com.typesafe.akka::akka-stream:2.6.3`
// standard imports
import akka.stream._
import akka.stream.scaladsl._
import akka.{ Done, NotUsed }
import akka.actor.ActorSystem
import akka.util.ByteString
import scala.concurrent._
import scala.concurrent.duration._

// Json parsing
import ujson._

// Handle files
import java.nio.file.Paths


implicit val system = ActorSystem("QuickStart")

val hotelids = scala.io.Source.fromFile("source.txt") // we need to scrape these urls or some ids for an endpoint

def parser(i: Int): (Int, String) = {
      val url = s"ENDPOINT/{i}"
      val urlResponse = Try(ujson.read(requests.get(url).data.toString))
      val result = urlResponse match {
        case Success(res) => "PARSED_RESULT"
        case Failure(e) => "null"
      }
      return (i, result)
    }

Step 4. We create file sink to store the data (you can print to console as well)

def lineSink(filename: String): Sink[(Int, String), Future[IOResult]] =
    Flow[(Int, String)].map(s => ByteString(s._1 + "," + s._2 + "\n")).toMat(FileIO.toPath(Paths.get(filename)))(Keep.right)

Step 5. Introduce throttling behavior because some websites disallow too many requests. Here, we make 25 requests in 10 seconds (5 requests in 2 seconds)

val resultFile = source.map(_.toInt).throttle(25, 10.second).map(parser).runWith(lineSink("output.txt"))

All done from console with Ammonite shell!!

Twitter with Swift

sivakon — Sat, 29 Feb 2020 11:03:08 +0000

Twitter search API

import Foundation


struct Endpoint {
    let path: String
    let queryItems: [URLQueryItem]
}


extension Endpoint {
    static func searchTwitter(_ query:String, _ count: Int) -> Endpoint {
        return Endpoint(
            path: "/1.1/search/tweets.json",
            queryItems: [
                URLQueryItem(name: "q", value: query),
                URLQueryItem(name: "count", value: String(count)),
                URLQueryItem(name: "result_type", value: "mixed"),
                URLQueryItem(name: "lang", value: "en")

            ]
        )
    }
}

extension Endpoint {

    var url: URL? {
        var components = URLComponents()
        components.scheme = "https"
        components.host = "api.twitter.com"
        components.path = path
        components.queryItems = queryItems

        return components.url
    }
}

Extension methods allows good abstraction over URLRequest in Foundation. We generate Endpoint from static function. Use Endpoint.url to get the URL.


let endpoint = Endpoint.searchTwitter("#coronavirus", 10) // static factory method to construct the Endpoint


let twitterURL = endpoint.url

// Construct URLRequest from URL
var ghRequest = URLRequest(url: twitterURL!)


ghRequest.httpMethod = "GET"
ghRequest.setValue("Bearer AAAATOKEN", forHTTPHeaderField: "Authorization")

// semaphore required for yielding async tasks

let semaphore = DispatchSemaphore(value: 0)

let taskGh = URLSession.shared.dataTask(with: ghRequest) { (data, response, error) in
    if let error = error {
        print("error \(error)")
        return
    }

    guard let data = data else { return }

    let json = try? JSONSerialization.jsonObject(with: data, options: []) as? [String: Any]

    if let statuses = json!["statuses"] as? [[String: Any]] {
        for i in statuses {
            print(i["text"]!)
        }
    } else {
        print("Wrong with the OAuth token Authentication")
    }


    semaphore.signal()
}

taskGh.resume()

_ = semaphore.wait(wallTimeout: .distantFuture)

Packaging python using Poetry on Google Cloud

sivakon — Thu, 07 Nov 2019 04:54:12 +0000

Intended workflow

Create/edit code
Write tests
Commit changes
Cloud build trigger (automated testing step)

Preferred way of creating packages

Step 1: Create docker image with poetry and the app code. Look at the sample dockerfile in the repo

FROM python:3.7-alpine3.7

ENV PIP_DISABLE_PIP_VERSION_CHECK=on

RUN pip install poetry

WORKDIR /app
COPY . /app

RUN poetry config settings.virtualenvs.create false && \
    poetry install -v --no-interaction --no-ansi

RUN python -m poetry_tutorial

You can push this image to Google Container Registry for futher use. But in this example, we don't need to. We will only use this image for creating the package
Step 2: Use this created image to create the package. All this does is to run poetry build inside the container in the working directory. Look at Dockerfile WORKDIR. This creates two files, tar.gz and whl files in the dist directory
We can upload these files directly to Cloud Storage bucket for distribution. Look at the artifacts section in the build file.

    steps:
    - name: 'docker'
      args: ["build", "-t", "gcr.io/<PROJECT_ID>/poetry_tutorial_builder:latest", "."] #step1
    - name: 'gcr.io/<PROJECT_ID>/poetry_tutorial_builder:latest'
      args: ['poetry', 'build'] #step2
    artifacts:
      # images: ['gcr.io/<PROJECT_ID>/poetry_tutorial_builder:latest']
      objects:
        location: 'gs://your-bucket/python_packages/'
        paths: ['dist/*']

Create an additional step (step 2) that only builds when all the tests have passed like this

...
    - name: 'gcr.io/<PROJECT_ID>/poetry_tutorial_builder:latest'
      args: ['poetry', 'run', 'pytest','tests/'] #testing before building
...

For deploying python app in a Kubernetes cluster

Add an extra step at the end of cloud build file

...
    - name: 'gcr.io/cloud-builders/gke-deploy'
      args:
      - run
      - --filename=gke_deployment.yaml
      - --image=gcr.io/<PROJECT_ID>/poetry_tutorial_builder:latest
      - --location=<region>
      - --cluster=<cluster-name>
...

Publishing to Private PyPI

Create a private PyPI hosted on Google App Engine using this
In the build step, add several commands that should run in the same container (or use a bash script). This is required for using Poetry to upload your built package to private PyPI

    poetry config settings.virtualenvs.create false
    poetry config repositories.gcp https://<PROJECT_ID>.appspot.com/
    poetry config http-basic.gcp <USERNAME> <PASSWORD>
    poetry publish -r gcp --build

Put this in bash file and run this as an extra step, to publish to PyPI instead of Kubernetes deployment

...
    - name: 'gcr.io/travelmlog/poetry-tutorial-publisher:latest' #publish step
      args: ['sh', 'publish.sh']
...

Username, password of your PyPI should be setup accordingly. Refer to this link. Remember to use sha1 hashed password here

Caveats:

If you setup Cloud build trigger on commit, make sure to always run poetry version to bump the version up. Otherwise, private PyPI does not accept same version package twice.

Using Poetry to create Python packages

sivakon — Fri, 01 Nov 2019 08:09:36 +0000

How to make a Python package using Poetry

Create a pyproject.toml file in a directory (with comments)

[tool.poetry]
name = "siva-poetry"
version = "2.0.0"
description = "Simple project"
authors = ["siva"]
packages = [
    { include = "siva_poetry", from = "src" },
]
# provide packages from `src` directory

[tool.poetry.dependencies]
python = "^3.7"
pendulum = "^2.0"
pyyaml = "^5.1"
# `poetry add pendulum` to add a dependency to current project

[tool.poetry.dev-dependencies]
pytest = "^3.0"

[tool.poetry.scripts]
# You can directly call the symbol `get_time` because it is exported using __init__.py.
start_pendulum = "siva_poetry:get_time"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Every python script you write should be ideally a package
Poetry makes it easy to create and distribute packages
Use __init__.py to tell if it's a package or not
When the module is imported by other packages, __init__.py starts first.
To run the module as a script, create a __main__.py file in the root folder of your module
To publish the package to PyPI, poetry publish after authenticating poetry with PyPI credentials

Directory structure

Use src folder and put packages in that folder
Put config inside the src/siva_poetry folder
Add os.chdir(os.path.dirname(__file__)) to package initialization script. Otherwise, config files are not read
Surprisingly, no config change for tests are required. They work as is.

Installation

Build using poetry build
pip install dist/siva_poetry-1.0.0-py3-none-any.whl to install to your local machine
Start a new terminal session and execute python -m siva_poetry. You can look at the code location and verify that all the required files are present.
poetry publish to publish to PyPI

Testing

Install before testing using poetry install -v
Run the tests using poetry run pytest tests/
poetry run before any command runs the package in current pyenv environment created by poetry.

Run the script

To run the module as a script (before installing), go to the root folder containing pyproject.toml and run python -m siva_poetry. This is the best way for debugging (before installing)
Because the __main__.py is present in the code files, you can to run the module as a script using poetry run python -m siva_poetry

Additional files in the tarball/whl

Put all the relevant files into the model folder inside the package, then all the files will be put together.

Directory structure

.
├── README.rst
├── dist
│   ├── siva-poetry-2.0.0
│   │   ├── PKG-INFO
│   │   ├── pyproject.toml
│   │   ├── setup.py
│   │   └── src
│   │       └── siva_poetry
│   │           ├── __init__.py
│   │           ├── __main__.py
│   │           ├── config
│   │           │   └── cities.yaml
│   │           ├── model
│   │           │   └── data.pickle
│   │           └── simple.py
│   ├── siva-poetry-2.0.0.tar.gz
│   └── siva_poetry-2.0.0-py3-none-any.whl
├── poetry.lock
├── pyproject.toml
├── siva_poetry.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   ├── entry_points.txt
│   ├── requires.txt
│   └── top_level.txt
├── src
│   └── siva_poetry
│       ├── __init__.py
│       ├── __main__.py
│       ├── config
│       │   └── cities.yaml
│       ├── model
│       │   └── data.pickle
│       └── simple.py
└── tests
    ├── __init__.py
    └── test_siva_poetry.py