DEV Community: Avi Kaminetzky

Scala - On Classes & Objects

Avi Kaminetzky — Wed, 11 Jul 2018 15:00:53 +0000

I've put together a semi-beginnerish guide for some of the pesky parts of classes and objects in Scala, ideal for someone coming from from languages like JavaScript & Python, and has barely encountered compiled, strongly typed languages.

I attempt to
1. Give the simplest examples
2. Never ever dare compare Scala to Java
3. Avoid selling you on functional programming or Scala

Each section builds on the previous one.

Recommended Resources

These resources are not directly related to this tutorial, but are very useful for learning Scala and functional programming. without making assumptions about your
[insert from here: functional programming/type theory/object oriented programming/category theory/math phd/actor model/Java 8/reactive programming] prowess.

Creative Scala (eBook)
Essential Scala (eBook)
Tour of Scala
Gitter - beginner question welcome!
Mostly Adequate Guide to Functional Programming (eBook)
Haven't watched this, but I'm sure it's great: Professor Frisby Introduces Composable Functional Javascript (video course)

Traits

Traits are used to share interfaces and fields between classes... Classes and objects can extend traits but traits cannot be instantiated and therefore have no parameters. (docs)

Traits can have both abstract and concrete methods.

trait Person {
  // concrete method, can only be overridden by class if it's a def, not a var or val
  def name = "Tim"
  // abstract method, must be implemented in class
  def greeting: String
}

class Tim extends Person {
  def greeting: String = s"Welcome, ${name}"
}

val timClass = new Tim
timClass.greeting
// res0: String = Welcome, Tim

You can override name in the Person trait:

class Tim extends Person {
  // override trait name.
  override val name: String = "Mr. Tim"
  def greeting: String = s"Welcome, ${name}"
}

val timClass = new Tim
timClass.greeting
// res0: String = Welcome, Mr. Tim

Plain Objects

An object on its own is simply a way to package together data and functions. Technically, it's a singleton instance of an anonymous class.

trait Person {
  val name: String
}

object Tim extends Person {
  val name = "Tim"
}

Tim.name
// res0: String = Tim

Another example from the App trait:

The App trait can be used to quickly turn objects into executable programs." (docs)

The verbose way:

object HelloWorld {
  def main(args: Array[String]): Unit = {
    val name = args(0)
    println(s"Hello, $name")
  }
}
// "Hello, dude"

Without the boilerplate:

object HelloWorld extends App {
    val name = args(0)
    println(s"Hello, $name")
}

To execute, run scala HelloWorld.scala dude from command line.

Option/Some

Any value that could potentially contain a null/empty value, has an option type, and when assigned, must be wrapped in Some.

val name: Option[String] = Some("Tim")
name.getOrElse("Anon")
// res0: String = Tim

The getOrElse method will provide a default for when name is None. It will also reduce name from a Some("Tim") value, to a plain "Tim" value.

A nice article on idiomatic handling of Option values.

The introduction of Maybe can cause some initial discomfort. Users of Swift and Scala will know what I mean as it's baked right into the core libraries under the guise of Option(al). When pushed to deal with null checks all the time (and there are times we know with absolute certainty the value exists), most people can't help but feel it's a tad laborious. However, with time, it will become second nature and you'll likely appreciate the safety. After all, most of the time it will prevent cut corners and save our hides.

Writing unsafe software is like taking care to paint each egg with pastels before hurling it into traffic; like building a retirement home with materials warned against by three little pigs. It will do us well to put some safety into our functions and Maybe helps us do just that. (Mostly Adequate Guide to Functional Programming Ch 8)

Pattern Matching

Pattern matching in Scala, is like a switch statement on steroids.

val name: Option[String] = Some("Bob")

name match {
  // for an exact match on a Tim name
  case Some("Tim") => "Hi, Tim!"

  // for any other Some(value)
  case Some(name) => s"Hi ${name}"

  // default case for any other value
  case _ => "Anonymous has no name!"
}

// res0: String = Hi Bob

Object's apply() and unapply() Methods

The way I think about these functions is that apply() is an object's built in constructor, while unapply() is how an object deconstructs, which particularly useful for pattern matching. These two functions are mutually exclusive, so you can define one without the other. (StackOverflow) Example from the docs.

object Person {
  def apply(name: String) = {
    s"Welcome--$name"
  }
  def unapply(name: String): Option[String] = {
    val nameSplit = name.split("--")
    // we can't be sure the value exists in the object
    if (nameSplit.tail.nonEmpty) Some(s"Goodbye--${nameSplit(1)}")
    else None
  }
}

// using apply() to construct the object
val dudePerson = Person("dude")
println(dudePerson)
// res0: Welcome--dude

// using unapply() to assign a variable
val Person(dudeName) = dudePerson
println(dudeName)
// res1: Goodbye--dude

// using unapply() for pattern matching
val matchTheDude = dudePerson match {
  case Person(name) => name
  case _ => "Goodbye--anonymous dude"
}
println(matchTheDude)
// res2: Goodbye--dude

Class Companion Object

When an object shares the same name with a class, it is commonly referred to as a companion object. It is useful for storing class functionality which isn't tied to an instance of the class (think of them as static or class methods).

For a class/companion object named Person with a constructor of name: String, the object can access anything from the class by calling new Person("Tim").value, while the class can access anything from its object by calling Person.value. Even if the values are set to private.


class Person(val name: String) {
  // Person(Tim) instead of the default memory address Person@3018b152
  override def toString = s"Person($name)"
}

object Person {
  // static method to track number of people objects instantiated
  var count = 0

  def apply(name: String): Person = {
    count = count + 1
    // using the Person object as the constructor, allows you to call Person("Tim")         
    // without the need to write "new Person("Tim")"
    new Person(name)
  }

  def unapply(person: Person): Option[String] = Some(person.name)
}

// calling apply() to instantiate the Person class
val bob = Person("Bob")

// our custom toString function
println(bob.toString)
// res0: Person(Bob)

// the apply() increments the count variable for each new Person
val tim = Person("Tim")
Person.count
// res1: 2

// unapply() with pattern matching
val matchTim = tim match {
  case Person(name) => name
  case _ => "Sorry, no match"
}
matchTim
// res2: String = Tim

Case Classes

case class is a type of class, which is useful for storing immutable data with minimal boilerplate. case object has similar functionality, but for a singleton class.

case class Person(title: Option[String] = None, name: String)

// comes with a companion object apply() method, no need for new Person("Dude")
val dude = Person(name="Dude")

// name parameter is set to an immutable val
dude.name
// res0: String = Dude

// sane toString method, as opposed to Person@721d791f weird hex stuff
dude.toString
// res1: String = Person(None,Dude)

val tim = Person(name="Tim")

// convenient copy method if you only need to change specific values
val timCopy = tim.copy(title=Some("Mr"))
// timCopy: Person = Person(Some(Mr),Tim)

// comes with a companion object unapply() method, useful for pattern matching
timCopy match {
  case Person(title, name) => title
  case _ => "No title"
}
// res2: Some(Mr)

// sane equals method which compares the content of case classes
dude.equals(tim)
// res3: Boolean = false

// class equality is usually done by comparing the memory address of each class 
// instance, that's why you get this weird behavior
class Guy(name: String)

val guy1 = new Guy("guy")
// guy1: Guy = Guy@5f46825d

val guy2 = new Guy("guy")
//guy2: Guy = Guy@17be95d

guy1 == guy2
//res4: Boolean = false

Implicits

A nice example from here:

// probably in a library
class Prefixer(val prefix: String)
def addPrefix(s: String)(implicit p: Prefixer) = p.prefix + s

// then probably in your application
implicit val myImplicitPrefixer = new Prefixer("***")
addPrefix("abc")  // returns "***abc"

The Play Framework controllers use implicits, so that any function call from an Action will gain access to the http request variable:

def action = Action { implicit request =>
   anotherMethod("Some para value")
   Ok("Got request [" + request + "]")
 }

def anotherMethod(p: String)(implicit request: Request[_]) = {
 // do something that needs access to the request
 }

implicit comes in handy when converting case classes to and from Json with the Play Json library. This is simple way to get type-safe JSON when creating an API:

import play.api.libs.json.Json
case class Totals(
                   subTotal: Double = 0,
                   estimatedShippingAndInsurance: Double = 0,
                   estimatedTax: Option[Double] = Some(0),
                   orderTotal: Double = 0
                 )
// withDefaultValues is required if you want your case class defaults to be applied // to the Json conversion
object Totals { implicit val writeTotals = Json.using[Json.WithDefaultValues].format[Totals] }

Class Structuring

In object oriented programming, we make a distinction between is-a and has-a. is-a is a candidate for inheritance.

A Product has-a salePrice and a regularPrice:

trait Product {
  val salePrice: Double
  val regularPrice: Double
}

A computer is-a product in a store:

case class Computer(
                     salePrice: Double,
                     regularPrice: Double,
                     processor: String,
                     operatingSystem: String
                   ) extends Product

Clothes is-a Product as well:

case class Clothes(
                     salePrice: Double,
                     regularPrice: Double,
                     size: Int,
                     material: String
                   ) extends Product

Another point to consider, Product is of type Computer or Clothes.

A common pattern for is-or (from Essential Scala):

sealed trait TrafficLight {
// pattern matching on case objects
  def next: TrafficLight = this match {
    case Red => Green
    case Green => Yellow
    case Yellow => Red
  }
}
final case object Red extends TrafficLight
final case object Green extends TrafficLight
final case object Yellow extends TrafficLight

val myLight = Red.next
// res0: TrafficLight = Green
myLight.next
// res0: TrafficLight = Yellow
myLight.next.next
// res1: TrafficLight = Red

sealed trait is used to tell the compiler to only expect TrafficLight of types Red, Green and Yellow. If we were to leave out Yellow while pattern matching, the compiler would complain that match may not be exhaustive. final is to forbid anyone form extending the Red, Green and Yellow classes.

Solr + Python — A Tutorial

Avi Kaminetzky — Tue, 24 Apr 2018 19:40:52 +0000

Update: I have pushed my Python code to GitHub (repo is here). My implementation is a tad more advanced than this tutorial. See the Readme file and code comments.

Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene (Solr website).

Goal

My goal is to demonstrate building an e-commerce gallery page with search, pagination, filtering and multi-select that mirrors the expectations of a typical user. See this article for a nice explanation of the multi-select filtering I am trying to implement.

Search should work for phrase queries like “mens shrt gap” or “gap 2154abc”, factoring in typos, various word forms (stemming) and phonetic spelling.

Solr Setup

Solr 7 is installed locally on my computer with an active connection to a database. Solr is using the deltaQuery feature (in db-data-config.xml) to detect changes in my database and import those records into Solr.

Web Development Setup

I have a basic Django/React app with Python 3. See this article for ideas on how to integrate Django with React. I recommend following these instructions to create your own Solr client.

I was considering using pySolr as a client, but it lacks good documentation and seems to have been neglected since 2015 (like most Solr libraries). Nevertheless, pySolr can work if you are ready to comb through the GitHub issues and codebase.

If you are using pySolr:

Paste export DEBUG_PYSOLR=’true’ into your terminal before running your server, and you will be able to view the URL generated by pySolr.
The URL you see in your terminal doesn’t seem to be clued in about URL encoding issues, so a query like Dolce & Gabbana will work on your website, but break when you paste the URL into a browser.

Facets & Facet Pivots

Facets are synonymous with product categories or specs. Solr has an option to return the available facets with their respective counts for a specific query. You can control the minimum number of products required in a facet by setting facet.mincount=<number>.

For example, if you are selling brand named clothing, facets might refer to gender, style and material. If the search was for “mens casual gap”, the facets returned would look like this (notice the constraints on gender and style):

"facet_fields" : {
   "gender" : [
       "Men" , 25,
       "Women" , 0 
    ],
   "style" : [
       "casual", 10,
       "dress", 0
    ],
   "material" : [
       "wool", 15,
       "cotton", 10
    ],
   }

Example Query

Let’s run through an example:

from urllib.request import urlopen
import urllib.parse
import simplejson

def gallery_items(current_query):

    solr_tuples = [
        # text in search box
        ('q', "mens shirt gap"),
        # how many products do I want to return
        ('rows', current_query['rows_per_page']),
        # offset for pagination
        ('start', current_query['start_row'] * current_query['rows_per_page']),
        # example of a default sort, 
        # for search phrase leave blank to allow 
        # for relevancy score sorting
        ('sort', 'price asc, popularity desc'),
        # which fields do I want returned
        ('fl', 'product_title, price, code, image_file'),
        # enable facets and facet.pivots
        ('facet', 'on'),
        # allow for unlimited amount of facets in results
        ('facet.limit', '-1'),
        # a facet has to have at least one 
        # product in it to be a valid facet
        ('facet.mincount', '1'),
        # regular facets
        ('facet.fields', ['gender', 'style', 'material']),
        # nested facets
        ('facet.pivot', 'brand,collection'),
        # edismax is Solr's multifield phrase parser
        ('defType', 'edismax'),
        # fields to be queried
        # copyall: all facets of a product with basic stemming
        # copyallphonetic: phonetic spelling of facets
        ('qf', 'copyall copyallphonetic'),
        # give me results that match most fields
        # in qf [copyall, copyallphonetic]
        ('tie', '1.0'),
        # format response as JSON
        ('wt', 'json')
    ]

    solr_url = 'http://localhost:<port>/solr/<core>/select?'
    # enocde for URL format
    encoded_solr_tuples = urllib.parse.urlencode(solr_tuples)
    complete_url = solr_url + encoded_solr_tuples
    connection = urlopen(complete_url)
    raw_response = simplejson.load(connection)

Phrase search will be discussed in the next section — Schema Modeling.

I would suggest using tuples for each key-value pair as it will be easier to urlencode. It will also be easier to manipulate, particularly when you have a complicated fq with a ton of AND, OR logic (which will happen very soon if you are doing filtering).
Each facet group will have its own fq field. This ensures that AND logic is applied across filter groups. Here is code for applying OR logic within a facet group:

    def apply_facet_filters(self):
        if self.there_are_facets():
            for facet, facet_arr in self.facet_filter.items():
                if len(facet_arr) > 0:
                    new_facet_arr = []
                    for a_facet in facet_arr:
                        new_facet_arr.append("{0}: \"{1}\"".format(facet, a_facet))
                    self.solr_args.append(('fq', ' OR '.join(new_facet_arr)))

facet.pivot.mincount allows you to control the minimum number of products required for a facet.pivot, but beware, if you set it to 0, your server will likely crash.
I’ve found that field values needed to be formatted in quotes: ‘fq’: “brand: \”{0}\””.format(current_query[‘current_brand’])
facets are returned in arrays like [‘brand’, ‘gap’], not a dict() which I find inconvenient. Here is one way to format them:

import more_itertools as mit
facets = {}

for k,v in raw_response['facet_counts']['facet_fields'].items():
    spec_list = [list(spec) for spec in mit.chunked(v, 2)]
    spec_dict = {}
    for spec in spec_list:
        spec_dict[spec[0]] = spec[1]
    facets[k] = spec_dict

raw_response['facet_counts']['facet_fields'] = facets

By default, if a user selects a facet in a facet group, Solr will return that facet group with only the selected facet, since the search has been narrowed down. But many times, a user would like still like to view the unselected facets and associated counts, to enable multi-select. To allow this functionality, use tagging and excluding. See my repo for a possible implementation.
To create price ranges as a filter with custom intervals, copy price to a new field with one of Trie fieldTypes. The new field should have indexed and stored set to false, and docValues set to true. Then follow the instructions to add custom ranges. See the next section on schema modeling. See my repo for a possible implementation.

Schema Modeling

If you can get past the idea that fields exist simply to store properties of data, and embrace the idea that you can manipulate data so it can be found as users expect it, then you can begin to effectively program relevance rules into the search engine. (Relevant Search, Chapter 5)

We are ready to modify fields in our document schema to conform to the users’ perception of our products.

Take a look at the documentation about how to update the schema, particularly the sections on tokenizing and filtering. Learn about stemming filters. Ask yourself which tokens/filters are relevant for your situation, and whether it should be apply at query or index time.

I will be following a recommendation in the documentation to copy all fields a user might be interested in into a single copyall field. This solves the albino elephant issue, as well as signal discordance:

As we’ve stated, when users search, they typically don’t care how documents decompose into individual fields. Many search users expect to work with documents as a single unit: the more of their search terms that match, the more relevant the document ought to be. It may surprise you to know that search engine features that implemented this idea were late to the party. Instead, Lucene-based multifield search depended on field-centric techniques. Instead of the search terms, field-centric search makes field scores the center of the ranking function. In this section, we explore exactly why field-centric approaches can create relevance issues. You’ll see that having ranking functions centered on fields creates two problems:

The albino elephant problem — A failure to give a higher rank to documents that match more search terms.

Signal discordance — Relevance scoring based on unintuitive scoring of the constituent parts (title versus body) instead of scoring of the whole document or more intuitive larger parts, such as the entire article’s text or the people associated with this film. (Relevant Search, Chapter 6)

We will be using the Schema API through the Admin UI. You cannot edit the schema file manually (why). Here is the recipe for creating the copyall field:

Step 1: Create a fieldType for the field. I am using the same fieldType for both index and query time. I have kept the stemming light to ensure that brand names stay intact.

      <fieldType name="facets" class="solr.TextField">
        <analyzer>
          <tokenizer class="solr.ClassicTokenizerFactory"/>
          <filter class="solr.LowerCaseFilterFactory"/>
          <filter class="solr.StopFilterFactory"/>
          <filter class="solr.ClassicFilterFactory"/>
          <filter class="solr.EnglishMinimalStemFilterFactory"/>
        </analyzer>
      </fieldType>

Step 2: Create a copyall field with a facets as the fieldType. Set multiValued=true to allow multiple values in the field (as an array). Set omitNorms=true since users don’t care about the length of each field (docs), and we don’t want Solr to care either.
Step 3: Create copyFields for every field in the data source that you want to be copied. Remember, there is no chaining of copyField’s.

    {
       "add-copy-field":{
           "source":"brand",
           "dest":"copyall"
        }
    }

Step 4: Repeat steps 1–3 if you want to create a copyall for phonetic spelling. Use an appropriate fieldType. I am using the Beider-Morse Filter.
Step 5: Add a tie breaker of 1 to get a most-fields functionality. The docs provide a nice explanation.

Some ideas:

Add index time boosts for products that are more popular and you want them to rank higher in the search results. You could also do a query time boost, something to the effect of bf='div(1,popularity)'.
Use function queries to customize anything about your the relevancy scoring of your search results.
Consider the N-Gram filter for typo tolerance.
Consider the Edge-N-Gram filter for autocomplete.
Consider using the text_en fieldType for regular English words (it is one of the many fieldTypes which come with Solr):

  <fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
    <analyzer type="index">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.EnglishPossessiveFilterFactory"/>
      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
      <filter class="solr.PorterStemFilterFactory"/>
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
      <filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.EnglishPossessiveFilterFactory"/>
      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
      <filter class="solr.PorterStemFilterFactory"/>
    </analyzer>
  </fieldType>

Debugging and Workflow

Check analysis in the Admin UI for how particular terms are analyzed at index or query time.
Add a console.log in your code to print the url for every query. Set debugQuery=true and read the parsedQeury and explain. All the math fun is lurking in the explain (see Relevant Search, Chapter 2).
After re-configuring the schema, make sure to delete all docs in your index and do a fresh full-import from your database. This can be done in the Admin UI.
If you need to debug the database import, use the Debug-Mode with verbose output.