DEV Community: Anders Björkland

Optimizing Travel Routes: Applying Genetic Algorithms to the Traveling Salesman Problem

Anders Björkland — Thu, 11 Apr 2024 13:15:19 +0000

Have you ever watched a delivery driver crisscrossing your neighborhood and wondered if they were taking the most efficient route? That's the Traveling Salesman Problem in action! This classic challenge has entertained mathematicians and coders alike for ages. In this article, we'll see how to tackle it using the power of Elixir and genetic algorithms, mimicking nature's process of evolution to find surprisingly good routes.

I was looking out my window and saw a delivery truck passing by on the street the other day. This got me thinking about a classic algorithm challenge: The Traveler Salesman Problem (TSP for short).

The challenge goes like this: A salesman has a list of cities they need to visit. They must visit each of them and return to their starting position. What is the shortest route they can take?

This is of course a problem that has been solved in multiple ways, thousands upon thousands of times over. The delivery driver has a route already preprogrammed - they don't need my help mulling it over. So why am I thinking about it?

One could say such everyday problems are more fun when applying algorithms to solve them, and I am looking for a reason to pull out my algorithmic toolbox. It's a fair statement! I recently put a new tool in that toolbox and I've been on the lookout for a reason to go nuts with it.

There are some interesting tools I can use, but there is a specific one I'm on the lookout for. So let me just quitcklu rifle through it:

Brute force: The hammer of all hammers, brute forcing is not elegant but it gets the job done... eventually. If we were to use brute force to solve TSP we would be calculating each possible solution and then select the best one. The number of possible solutions to TSP is given by the formula (n-1)!/2. It's a factorial representing each possible combination of visits to the cities, and as it loops back to the beginning, the direction of the loop does not matter. So if the salesman had to visit 18 cities, this would be (18-1)!/2) which comes out to 1.8 × 10¹⁴ possible solutions. It's entirely possible, but I rifle on.
Nearest neighbor: A screwdriver in comparison, this algorithm is focused on the here-and-now with disregard for the larger picture. With a nearest neighbor the salesman would always travel to whichever city is closest to the city they are currently in. While this might see them cover a larger distance than brute forcing, it is much faster and can be good enough. But if you've read my previous article, you might suspect that I'm on the lookout for another tool.
Genetic Algorithm: Now this is what I'm talking about! Genetic Algorithms (GA) might find the best possible route, but they might also find a solution that is just good enough. This is my hammer, saw, and ruler - all combined. It might look a bit freakish, but it's so versatile and fun to use! GA is inspired by evolutionary processes. Survival of the fittest and mutations will play a role in it finding a good enough solution. As long as we can define the solution as a "chromosome" and a way to measure the "fitness", GA will be possible to use.

So how would I use a Genetic Algorithm (GA) to find a short enough route for the salesman? Let's consider this; is it possible to describe a possible solution as a "chromosome"? I will say an emphatic yes! A route can be described as a series of cities in the order they will be visited. If we have the 3 cities 1, 2, and 3, then a chromosome could be a list: [1, 2, 3] or [2, 1, 3], and so on. As long as there are coordinates for each city it will be possible to calculate the distance between them, a.k.a: their "fitness" (where shorter is better).

Knowing that I have a reasonable enough representation of a possible solution and a way to measure its fitness, let me share with you the plan!

The Plan

There is a classic movie series here in Sweden called "The Jönsson Gang". It's about a gang of mainly 3 thieves aiming for heists beyond their competence. Their leader, Charles Ingvar Jönsson, had an iconic line: "I have a plan!" His plans were very detailed, and would probably work most of the time. That is, if he and his gang weren't distracted by anything from a hotdog to a beer. Well, let me tell you: I have a plan and I'm all out of sausages!

We are going to need:

Chromosomes and a way to generate them for the initial population.
Fitness function that will be able to give a measure to the chromosomes' "fitness".
Which means we need a function to measure the distance between each city in the chromosome, and not forget to measure the distance between the last city in the chromosome with the first.
Parents and how we select them. OK, that sounds weird, but if we are going to have a new generation of chromosomes we need something to base them off of.
And we are not going to select them completely at random. This is not a singles' cruise!
Crossover (and I'm so glad that this is the term chosen) between pairs of parents' chromosomes.
Each pair of parents will produce 2 offsprings, where first offspring will select from parent a first, and second offspring will select from parent b first.
Mutation of chromosomes. This is a natural part of the parents producing offspring. This will introduce a unique trait to an offspring that is not necessarily found in the parent. If we are lucky we might see some Super Mutants!
Pruning of the population. The least fit chromosomes will leave place to the next generation. This dispassionately called process is less on the nose than "The Purge", and I can't ask for more than that.
The Evolutionary Loop! Each part that I've described is part of evolution. All that is left is passing on the new generation to the next and let nature have its fun. Since I'm not as elegant as dear Charles Ingvar Jönsson I will set a generational limit instead of a goal-oriented threshold for the loop.

Chromosomes are potential solutions

With the Travelling Salesman Problem the solution is a route, and a route is nothing but a series of places to visit. And by randomly selecting cities to visit, there is a potential that these routes collectively hold the key to the shortest route. And eventually, through evolution, the collection of randomly generated routes, survival of the fittest, and mutation, a generation will produce a good enough route.

I think I've teased you enough. It's time to bring in the code (using Elixir)! I will represent the cities as a list of coordinates, and the chromosomes shall be the indices for each city.

  @cities [
    %{x: 10, y: 20}, %{x: 5, y: 15}, %{x: 12, y: 1},
    %{x: 20, y: 10}, %{x: 7, y: 14}, %{x: 15, y: 3},
    %{x: 15, y: 12}, %{x: 15, y: 10}, %{x: 20, y: 8},
    %{x: 2, y: 14},  %{x: 3, y: 11}, %{x: 14, y: 7},
    %{x: 5, y: 11}, %{x: 5, y: 23}, %{x: 20, y: 23}, 
    %{x: 7, y: 23}, %{x: 16, y: 12}, %{x: 25, y: 12}
  ]

A chromosome is the length of the list, with each gene representing the index to one of the cities. The foundation is laid when I initialize the population. Let me show you:

  def initialize_population(population_size) do
    chromosome_template = Enum.to_list(0..(Enum.count(@cities) - 1))


    Enum.map(1..population_size, fn _ -> Enum.shuffle(chromosome_template) end)
  end

Not to toot my own horn here, but doesn’t that look very simple? I introduce the chromosome_template that contains all the indices and shuffle it until I have created enough chromosomes.

While we are talking about population and chromosomes, let me just warn you that we will see an addition on top of this. I will store a chromosome along with its distance (introduced just below this paragraph) in a map so we get a chromosome_data variable with a value like %{chromosome: [0,1,2], distance: 4.321}.

The shorter the better - the fitness function edition

Just like Bilbo, it's not enough to go "there" - you need to have "and back again" too! The traveling salesman will be visiting each city and then return to their starting city, making a complete loop. The fitness function will sum up all the distances between each city and return that.

A city's coordinate is represented by a map: %{x: 10, y: 20}. When we have two cities they can be: %{x: 10, y: 20} and %{x: 5, y: 15}. The distance between the cities requires a touch of math (or magic if you just want to skim this). The distance between two coordinated in a 2D field can be calculated with the help of Pythagoras: a^2^ + b^2^ = c^2^ (or 2D Euclidean). And reframed for the two cities it would be the same as (x1 - x2)² + (y1 - y2)² = c². Then the distance is d=√(c²). And when we insert the values: d=√((10-5)² + (20-15)²)≈7.

Representing this calculation in code:

  # Calculate the distance between two cities 
  def distance(%{x: x1, y: y1}, %{x: x2, y: y2}) do
    # Using 2D Euclidean distance
    dx = abs(x1 - x2)
    dy = abs(y1 - y2)


    (dx ** 2) + (dy ** 2)
    |> :math.sqrt()
  end

This calculation should be done between each city in a chromosome. If you like some recursion and pattern matching, you will like this bit:

  def distances([first, second]) do
    city1 = Enum.at(@cities, first)
    city2 = Enum.at(@cities, second)

    distance(city1, city2)
  end
  def distances([first, second | tail]) do
    city1 = Enum.at(@cities, first)
    city2 = Enum.at(@cities, second)

    distance(city1, city2) + distances([second | tail])
  end

Oh, the pattern matching - unpacking values to variables right there in the function definitions! The first distances function will catch when a list is passed to it that contains only two elements, so it will return only the distance between the two cities in the list. But the second function will continue calling itself as long as there are more elements in the list.

And if you are thinking that this does not measure the loop-back distance, you are correct! I saved that fun tidbit for the fitness_function:

  def fitness_function(chromosome) do
    last_to_first_elements = Enum.take(chromosome, 1) ++ Enum.take(chromosome, -1) 

    there = distances(chromosome)
    and_back = distances(last_to_first_elements)

    there + and_back
  end

On top of taking the distance between the cities in a chromosome, the fitness_function will also take the last and first element and get its distance. Finally it will add up the distance there and back again and return the sum on a silver platter.

With this tidbit, each chromosome can be neatly packaged together with its distance in a data-package. A bit like this:

    population = initialize_population(population_size)


    chromosome_datas = 
      population
      |> Enum.map(
        fn chromosome -> %{chromosome: chromosome, distance: fitness_function(chromosome)} end
        )

As initialize_population returns a list of chromosomes, it can be enumerated on and return new elements that package the chromosome with the distance returned from the fitness_function. Doing it this way means we will have the chromosome's fitness readily at hand. I do not consider distance or fitness a part of the chromosome, but rather additional data about the chromosome.

The not-a-singles-cruise way of selecting parents

"So you want offspring but want someone to do it for you?" Good news, I’ve got the recipe for you! Given that there is a population of random chromosomes, it would be neat if the next generation were a bit better than the one previous. While I personally would like to say that millennials have it all figured out, we can still improve upon this generation. But for this to work we want to give a higher chance to chromosomes with shorter routes to produce offspring. This will mean that the offspring inherits the parents' good qualities (or bad, but that's why we want to prefer the parents with shorter routes).

There is this thing called roulette wheel selection. To give a higher chance for the chromosomes with shorter distances to be selected I will just give them a larger pie of the wheel than the chromosome with longer distances. When I spin the wheel, every chromosome has the chance of being the winner but the likelihood that it falls on a chromosome with a longer route is smaller.

My foolproof plan for this is to summarize every route's distance and make an inverse ratio for them. On each spin of the wheel, there is a chance that a chromosome will be selected. The higher the inverse ratio is, the higher the chance that the wheel will stop at the chromosome.

In Elixir, I would do it like this:

  def general_inverse_fitness(population_data) do
    Enum.reduce(population_data, 0, fn data, acc -> acc + 1 / data.distance end)
  end

  def select_parents(population_data, num_parents \\ 2) do
    general_inverse_fitness =  general_inverse_fitness(population_data)

    Enum.map(1..num_parents, fn _ -> # Select n parents
      random_value = :rand.uniform() * general_inverse_fitness
      select_chromosome(population_data, random_value, 0)
    end)
  end

  defp select_chromosome(population_data, target_fitness, acc_fitness) do
    [chromosome_data | tail] = population_data

    # The greater value from the distance function the less the new distance will be, 
    # so we promote the shorter routes before the longer routes
    new_fitness = acc_fitness + 1 / chromosome_data.distance

    if new_fitness >= target_fitness do
      chromosome_data
    else
      select_chromosome(tail, target_fitness, new_fitness)
    end
  end

The select_parents function is the entry-point. It will first summarize the inverse distances. In general the sum will become larger the shorter the distances are but it will depend on population size (meaning as a standalone metric it is not useful for comparing between different population sizes, but is meaningful within the population that it is used).

With the general_inverse_fitness, the function can continue with selecting parents. Whenever a new parent is selected, a new random value will be the target that a chromosome should achieve. If a chromosome does not achieve the target it will add to an accumulator that is added to the next chromosome's calculation. This is like the slowing of the roulette wheel. So selecting parents is not as much a cruise fare as it is a casino!

Crossover and mutation: The Offsprings

Before we let the parents do what defines them in the evolutionary process, let's consider who should be pairing up with who? In the previous step I let the roulette wheel select parents. I will let the order in which they were selected define the pair of parents. Though it could be any kind of parental constellation - we are not limited to two parents, really - I feel like I want to keep it simple.

Each pair of two parents will combine their chromosomes with each other in a crossover process. This part is important! So if you've skimmed this far, focus-up!

In a crossover I will take a random length of genes from one parent starting from a random position. The genes will be a subarray of cities, preserving the order from the parent chromosome. Then I will assemble a new chromosome and fill in the blanks for each element that is missing from the first subarray.

Example
A chromosome has length 6, the crossover will have a starting index of 2, and subarray length of 3.

Parent Chromosome A: [2, 0, 1, 3, 4, 5]

Parent Chromosome B: [1, 2, 3, 5, 4, 0]

Subarray 1: [_, _, 1, 3, 4, _]

Available genes from Chromosome B: [2, 5, 0]

Fill in blanks with available genes: [2, 5, 1, 3, 4, 0]

By combining the chromosomes this way I can preserve some of the route information and find new ones. But whichever parent's chromosome is selected first will have a higher impact for the new chromosome. For this reason I will crossover each pair of parents twice. Both parents' chromosomes will get to go first. It will be less fuss that way!

But an offspring is not only the combination of parents' chromosomes. Each offspring has the potential of being unique: a beautiful snowflake, if you may. Or a Super Mutant!

This is the role of mutation. To randomly introduce new characteristics to a chromosome. What my mutation function will do is to select at least one gene from the crossover chromosome and insert it at a random position.

  def crossover(chromosome_1, chromosome_2) do

    # 1. We select the parts of chromosome_1 randomly (the order is important)
    #    It can be [_, _, B, C, A, _]
    #    which means first selection will be [B, C, A], leaving D, E, F 

    cut_point_1 = :rand.uniform(Enum.count(chromosome_1) - 1) 
    cut_point_2 = :rand.uniform(Enum.count(chromosome_1) - 1) 
    [start_point, end_point] = [cut_point_1, cut_point_2] |> Enum.sort()
    selection = Enum.slice(chromosome_1, start_point..end_point) 

    # 2. We want to get the relationships that is present in chromosome_2
    #    It can be [B, C, D, F, E, A]
    #    Be mindful that we do not want to select either A, B, or C again. 
    #    This means that we should filter out every used gene: [D, F, E]
    #    and start filling the missing pieces with the genes that are left.

    available_genes = Enum.filter(chromosome_2, fn gene -> !Enum.member?(selection, gene) end)
    {prefix, suffix} = Enum.split(available_genes, start_point)

    # 3. And then we assemble the chromosome from the different parts, 
    #    combining chromosome_1 and chromosome_2 genes

    prefix ++ selection ++ suffix
  end

  def mutation(chromosome) do
    chromosome_length = Enum.count(chromosome)
    portion_of_mutation = 0.1
    least_num_of_mutations = 1

    num_of_mutations = 
      round(chromosome_length * portion_of_mutation) 
      |> max(least_num_of_mutations)

    # For each number of mutations
    # - Take random element
    # - Insert at random position

    mutating_genes = Enum.take_random(chromosome, num_of_mutations)

    template = Enum.filter(chromosome, fn gene -> !Enum.member?(mutating_genes, gene) end)

    Enum.reduce(mutating_genes, template, fn mutating_gene, acc -> 
      random_pos = Enum.count(acc) - 1 |> :rand.uniform()
      List.insert_at(acc, random_pos, mutating_gene)
    end)

I would be amiss if I didn't point out that changing the position of just one element in the chromosome will have a huge impact on the overall chromosome information. One element contains not only the order a city is being visited, but it's also information about a connection from one city via the element to the next city. So keeping the mutation to minimal changes is good, as it otherwise would be resulting in a chromosome that is far different from either parent.

Making space for the new generation

Charles Ingvar Jönsson would probably know about the Swedish proverb: "Finns det hjärterum så finns det stjärterum". It means that one can always make space for anyone that wants to join, even if it is crowded: the crowd just needs to squeeze together more. So dear Charles would be pretty terrified if he knew what I am about to do.

In nature the population size would continuously fluctuate. But in my synthetically created world the size will be constant. That means: for any number of new offsprings, as many of the old population need to scoot over, and out!

There are many ways this can be done. Some are more imaginative than others. The Purge might have an idea on how this should be done. But I will just sort the old population by their fitness and keep just so many as are needed to keep a constant population size when the new generation is added.

  def prune(sorted_population_data, prune_count) do
    preserve_count = Enum.count(sorted_population_data) - prune_count

    Enum.take(sorted_population_data, preserve_count)
  end

Prune. That sounds pretty cold. But that's the process where I simply "cut off" some branches that, in this case, are too long. This allows our population to become ever more fit for its task.

The snapshot of the process: prune(sorted_population_data, num_parents) ++ offsprings

Evolving a better fit

The goal is to find a good-enough chromosome, or route, for our salesman. Each component plays a crucial part to get to that goal. But it's a process that requires many trial and errors. That's why there is randomness and pruning involved. The randomness allows the algorithm to try new and innovative routes never before considered. And while many of them are utterly poor, somewhere a nugget of gold is found.

The evolutionary loop will take in a population of chromosomes, select a cadre of parents and let them reproduce, followed by pruning the population to make space for the next generation. This new population will be passed on to another go at evolution. I reason that it is possible to have this loop continue until a good enough fit has been reached (such as when improvements from one generation to the next are stagnating). However, I've decided to use a hard generational limit instead because I'm feeling lazy.

Once the generational limit is reached, the chromosome with the shortest route in the current population will be returned.

  def evolve(
      sorted_population_data, 
      generation_n,  
      %{generation_limit: generation_limit}
    ) when generation_n == generation_limit do

    [chromosome_data] = Enum.take(sorted_population_data, 1)

    total_fitness = general_inverse_fitness(sorted_population_data)

    %{
      distance: chromosome_data.distance, 
      chromosome: chromosome_data.chromosome, 
      generation: generation_n, 
      population_fitness: total_fitness
    }
  end
  def evolve(sorted_population_data, generation_n, memo) do
    %{num_parents: num_parents} = memo

    parents = select_parents(sorted_population_data, num_parents)
    offsprings = 
      parents 
      |> Enum.chunk_every(2)
      |> Enum.flat_map(fn [parent_1, parent_2] -> 
        [
          TSP.crossover(parent_1.chromosome, parent_2.chromosome) |> TSP.mutation(),
          TSP.crossover(parent_2.chromosome, parent_1.chromosome) |> TSP.mutation()
        ]
      end)
      |> Enum.map(fn chromosome -> 
        %{chromosome: chromosome, distance: fitness_function(chromosome)}
      end)

    new_population_data = 
      prune(sorted_population_data, num_parents) ++ offsprings
      |> Enum.sort_by(
        fn chromosome_data -> chromosome_data.distance end,
        :asc
      )

    evolve(new_population_data, generation_n + 1, memo)
  end

  def run(population_size \\ 100, generation_limit \\ 250, reproduction_rate \\ 0.3) do
    population = initialize_population(population_size)

    population_data = 
      population
      |> Enum.map(
        fn chromosome -> %{chromosome: chromosome, distance: fitness_function(chromosome)} end
        )

    sorted_population = Enum.sort_by(
      population_data,
      fn %{distance: distance} -> distance  end,
      :asc
    )

    num_parents = num_parents(population_size, reproduction_rate)

    evolve(
      sorted_population, 
      0, 
      %{
        num_parents: num_parents, 
        generation_limit: generation_limit
      }
    )
  end

Let's say that someone like me created all these components and put them in a module called TSP (Traveling Salesman Problem in short). The way I would initiate the evolution would be to call the run function like this:

TSP.run(aggregator_pid, 100, 1000, 0.3)

The response would be:

%{
  distance: 87.68567015836446,
  chromosome: [10, 9, 1, 4, 13, 15, 0, 14, 17, 3, 8, 16, 6, 7, 11, 5, 2, 12],
  generation: 1000,
  population_fitness: 0.946812150303661
}

To me, this looks like a functional algorithm. But don't take my word for it. Take this graphs instead:

In general, a population's (inverse) fitness will increase over time, but it will fluctuate as new and untested offsprings are introduced to the mix. A shortest distance may remain the current best alternative for generations upon generations, until the seeming equalibirum is punctuated by a burst of improvement. By combining these views, we can see that the population is ever improving - even if we can't detect it right away by the currently shortest route.

Here's a peak at the routes (and at which generation) the algorithm discovered them during one run of the code:

population_size=100, generation_limit=1000, reproduction_rate=0.3

Reflecting on the generations past

Genetic Algorithms are useful when we might want a solution that is "good enough", or if we don't know what a good solution might even look like. I was lucky today that I withstood the temptation of chasing a hotdog and managed to arrive at a solution (Charles Ingvar Jönsson would be proud, as well as the traveling salesman). I can't say if you managed to withstand it, but I'm happy that you allowed me to draw a route for you.

I find Genetic Algorithms incredibly fun to work with. This has been the second time that I'm exploring the subject and I would like to do it again. Besides finding solutions to interesting problems I also find these additional benefits:

Trying different data representations and structures can unlock huge perfomance improvements - which is fun!
Recursion, pattern matching, and (if you check out the Livebook document) concurrency! Working with Genetic Algorithms is a good exercise to deepen Elixir skills.
Fitting each different component together in a recursive loop that makes sense is very satisfying!

I hope you had a good time reading this, and hopefully learned something new. I sure did!

Intro to Genetic Algorithms with Elixir

Anders Björkland — Mon, 25 Mar 2024 12:34:28 +0000

Genetic Algorithms (GA) are algorithms inspired by the evolutionary process found in nature. Sometimes these are called Stochastic Algorithms as they make use of randomness to find an optimal solution, but with features such a "natural selection" to remove less suitable solutions while generating new possible ones.

Genetic Algorithms are found in route optimzation problems, in engineering (e.g. evolved antennas), in epidemic mitigation, and much more.

In this article we will solve a simple problem: "what is the target phrase?". This is not a real-world problem, but suitable enough to showcase each component of a Genetic Algorithm; like a "Hello World" of Genetic Algorithms. With the foundation provided by this, I hope this can give you a better understanding of GAs, and an appreciation for using Elixir with them.

Anatomy of a Genetic Algorithm

A genetic algorithm will try to promote the possible soultions (chromosomes) with the best "fitness" and combine them into a new possible solution, which will inherit features of both. While it does this, it may introduce some mutations (small changes that were not part of its "parents" chromosomes).

In essence, these are the genetic algorithm's components:

Chromosomes possible solutions
Initial population random chromosomes
Fitness evaluation
Selection better fitness score are more likely to be chosen as "parents"
Crossover "mating" of parents to produce an "offspring", a new chromosome
Mutation small random changes in a chromosome
Evolution Loop repeating the steps of **selection* -> crossover -> mutation to arrive at a chromosome that is the best fit for the problem at the hand*

We will add an "elitism" component to this which will make sure that the best possible solutions are never discarded. This is a component that may act dynamic depending on where we are in the evolution loop. But initially we will keep it static.

For anyone who has set up a Livebook application and want to experiment, just grab the source from my gist (copy) and click Open/From source.
If you haven't set up a Livebook application, read about how to do that in my article Interactive documents with Livebook. You can set it up on free hosting via Hugging Face.

The target phrase

Our goal is via random chromosomes start selecting the best fit ones to crossover with each other. They will be selected based on how well they fit with the target phrase. It will be a phrase based on characters from 'a' to 'z', with space included. Let's define this:

defmodule GeneticString do
  # We define the target phrase here but it might as well be passed in via the evolve function.
  @target_phrase "The solution is yet to emerge"

  # Defines which characters can be present in the target phrase
  defp possible_characters() do
    Enum.to_list(?a..?z) ++
      Enum.to_list(?A..?Z) ++
      [?_, ?\s]
  end

end

The first chromosome

The first step towards arriving at a solution is a function to generate a random chromosome. It will produce a chromosome of length length by random characters present in possible_characters. This function will be used to generate the initial population.

defmodule GeneticString do
  # ...

  # Generate a random chromosome (potential solution)
  defp random_chromosome() do
    Enum.map(
      1..String.length(@target_phrase),  # We will help the GA along the way by letting it set correct length
      fn _ -> possible_characters() |> Enum.random() end
    )
    |> List.to_string()
  end

end

Evaluating fitness

The fitness evaluation is an important part in the genetic algorithm. It will tell the algorithm when it is moving closer towards a solution. In our case, we will do a fairly simple evalutaion. The fitness score will represent how many characters are correctly placed in the chromosome when compared to target_phrase. A more evolved fitness evaluation might take into account its length as well as maybe its Levenshtein distance, and so on.

We will see that our fitness evaluation will also take a map, a "memo" of already evaluated chromosomes. This way an already evaluated chromosome will not have to be calculated again. This would be a trade-off between memory/retrieval and calculation cycles.

defmodule GeneticString do
  # ...

  # Calculate fitness based on if a character is placed in the correct n:th place.
  defp fitness_calc(chromosome) do
    chromosome
    |> String.graphemes()
    |> Enum.zip(String.graphemes(@target_phrase))
    |> Enum.count(fn {char1, char2} -> char1 == char2 end)
  end

  # Either retrieve fitness score for a chromosome or calculate it and place it in the map
  defp fitness(chromosome, memo \\ %{}) do
    case Map.get(memo, chromosome) do
      nil ->
        fitness = fitness_calc(chromosome)
        updated_memo = Map.put(memo, chromosome, fitness)
        {fitness, updated_memo}

      fitness ->
        {fitness, memo}
    end
  end

  # Updates the fitness map for all chromosomes
  defp memoize_fitness([], memo), do: memo
  defp memoize_fitness([chromosome | tail], memo) do
    {_, new_memo} = fitness(chromosome, memo)
    memoize_fitness(tail, new_memo)
  end

end

Parents, offsprings and mutations

We have now defined fitness and chromosomes. Let's go to the next part, which is selecting parents to produce offsprings via the components crossover and mutation. The idea is to get at least one parent from the ones with the best fitness score, and the other parent from the broader population. Once selected, these will be passed to the crossover function to produce the offspring, and a mutation function that will introduce a bit of variation. Currently we will only make a mutation in one randomly chosen spot.

defmodule GeneticString do
  # ...

  # Randomly select two "parents" from the provided list of chromosomes
  defp select_parents([]), do: []
  defp select_parents(possible_parents) do
    possible_parents
    |> Enum.shuffle()
    |> Enum.take(2)
  end

  # Combine first part of one chromosome to the second part of another. 
  # The resulting chromosome will have features from both "parents"
  defp crossover(chromosome1, chromosome2) do
    crossover_point = Enum.random(1..(String.length(chromosome1) - 1))

    String.slice(chromosome1, 0, crossover_point) <>
      String.slice(chromosome2, crossover_point, String.length(chromosome2) - crossover_point)
  end

  # Introduce a random mutation to a chromsome. 
  # The spot where the mutation will happen is randomly chosen, as is the possible character introduced.
  defp mutation(chromosome) do
    mutation_point = Enum.random(0..(String.length(chromosome) - 1))

    String.slice(chromosome, 0, mutation_point) <>
      List.to_string([possible_characters() |> Enum.random()]) <>
      String.slice(
        chromosome,
        mutation_point + 1,
        String.length(chromosome) - (mutation_point + 1)
      )
  end

Evolution - the iterative process to produce best fit chromosome

We will be using a evolve mechanism where we will initialize a starter population with a radom chromosomes and set an elitism rate. These initial values will be passed into a recursive evolve mechanism (our evolution loop). Once we've either gotten a perfect fitness score, or we have hit the generation limit, we will return the best suited chromosome.

(While we are at it we will also bundle in which iteration - n:th generation - we were at when the chromosome was produced.)

defmodule GeneticString do
  # ...

  # Recursive function that will continue evolving the chromosomes until a perfect fitness score is reached, 
  # or the generation limit has been reached.
  defp evolve_mechanism(generation, %{population: [best_match | _]}, _elitism, _fitness_memo)
      when generation.limit == generation.i,
      do: %{chromosome: best_match, generation: generation.i}

  defp evolve_mechanism(generation, population_data, elitism, fitness_memo) do
    memoized_fitness = memoize_fitness(population_data.population, fitness_memo)

    # Sort population based on fitness
    sorted_population =
      population_data.population
      |> Enum.sort_by(fn chromosome -> Map.get(memoized_fitness, chromosome) end, :desc)

    # Select number of population that will have a higher chance to survive, and produce offspring
    elite_population =
      sorted_population
      |> Enum.take(elitism.count)

    rest_population = Enum.drop(sorted_population, elitism.count) |> Enum.shuffle()

    possible_parents = case length(elite_population) do
      x when x < 2 -> elite_population ++ Enum.take(sorted_population, 2)
      _ -> elite_population
    end
    [parent1, parent2] = select_parents(possible_parents)

    # Produce offspring by combining parents and introduce a mutation
    offspring = crossover(parent1, parent2) |> mutation()

    # Keep an elite-num of chromosome, and drop less fortunate chromomse before appending the offspring
    new_population =
      (elite_population ++ rest_population)
      |> Enum.drop(-1)
      |> Kernel.++([offspring])

    [elite | _] = new_population
    fitness_score = Map.get(memoized_fitness, elite)
    max_score = String.length(@target_phrase)

    # If perfect fitness score, then return the elite chromosome.
    case fitness_score do
      x when x == max_score ->
        %{chromosome: elite, generation: generation.i}

      _ ->
        evolve_mechanism(
          %{i: generation.i + 1, limit: generation.limit},
          %{population: new_population, size: population_data.size},
          elitism,  # Might be switched for adaptive elitism
          memoized_fitness
        )
    end
  end
end

The evolve_mechanism function is recursive in that it will call itself as long as we either haven't gotten a perfect fitness or we have hit a generation limit.

Now it's finally time to define our entry-point to this genetic algorithm. evolve will set up some initial values and start the evolve-mechanism:

defmodule GeneticString do
  # ...

  #  Our ENTRY POINT to the evolution
  def evolve(population_size, generation_limit \\ 100) do
    population = Enum.map(1..population_size, fn _ -> random_chromosome() end)
    elitism_rate = 0.01

    evolve_mechanism(
      %{i: 0, limit: generation_limit}, # generation data
      %{population: population, size: population_size}, # population data
      %{rate: elitism_rate, count: floor(elitism_rate * population_size)}, # elitism data (suitable for adding an adaptive mechanism)
      %{} # fitness map
    )
  end
end

We can call our Genetic Algorithm with the following command: GeneticString.evolve(50, 20000). This will set the initial population to 50 and sets the generation limit to 20000. The result will be a map like this: %{generation: 6367, chromosome: "The solution is yet to emerge"}

Our Genetic Algorithm has multiple points where we can experiment with its different parameters. Instead of a static elitism_rate we can involve a more dynamic or adaptive format. We can also experiment with number of offsprings and parents as well. There are a lot of ways to tweak just this simple Genetic Algorithm, but for now, let's explore how it would be like to have elitism that is adjusted according to how far along the generations has come towards the generational limit.

First we will add a new function that will update the elitism:

defmodule GeneticString do
  # ...

  defp adjust_elitism(elitism, generation, population_size) do
    progress = generation.i / generation.limit

    rate = case progress do
      # Less of population considered for elite (aka, in this case, available for reproduction)
      x when x < 0.2 ->
        max(elitism.rate - 0.05, 0.01)

      # In later generations, a larger proportion of population will be included in elite population
      _ -> 
        min(elitism.rate + 0.01, 0.25)
    end

    %{rate: rate, count: floor(rate * population_size)}
  end
end

Then we will update our evolution_mechanism so that in its recursive call will be updating the elitism:

defmodule GeneticString do
  # ...

  defp evolve_mechanism(generation, population_data, elitism, fitness_memo) do
    # ...
    case fitness_score do
      x when x == max_score ->
        %{chromosome: elite, generation: generation.i}

      _ ->
        evolve_mechanism(
          %{i: generation.i + 1, limit: generation.limit},
          %{population: new_population, size: population_data.size},
          adjust_elitism(elitism, generation, population_data.size),  # We change this row to update elitism
          memoized_fitness
        )
    end
  end
end

I test ran our genetic algorithm first with a static elitism, then with an adjusted elitism. Each was run 1000-times. I plotted their density curves and gathered their core descriptive statistics. Let's see the result:

n:th generation (static elitism)

min	max	mean	median	stdev
2391	20000	7170	6763	2497

n:th generation (adjustable elitism)

min	max	mean	median	stdev
2149	15078	6297	5991	1847

With this adjustable elitism, we got a more narrow distribution and a lower median. We can argue that the inclusion of the adjustable elitism improved our algorithm. But still, there are so many ways we can tinker with it. The elitism can have multiple ways it can be adjusted; different cut-offs, rate changes, minimum and maximum rates, and so on. But there are also many other adjustments we can make. How many parents we will allow getting offsprings, and how many? Perhaps the fitness-evaluation is too simple and need more nuance? Should the crossover be more of a diffusion where one parent's feature is scattered in the chromosome?

The Genetic Algorithm is versatile! We have seen its core components and got to tinker with them. We have explored one way to make it more dynamic and what impact that may have. Even a simple GA as we have built here can be adjusted in many different ways. I therefor welcome you to do so. I have created a GitHub Gist with a Livebook document that you may use in your own Livebook session: genetic-algorithm.livemd

Interactive Documents With Livebook

Anders Björkland — Sun, 10 Mar 2024 10:36:59 +0000

Have you ever wrestled with confusing documentation while trying to learn a new library or language? The explanations might be sparse, the code examples outdated, or the concepts too advanced for a beginner. It's a common frustration! Take the powerful R visualization library ggplot2. Its complex subject matter and documentation make it a challenge for newcomers.

The documentation for ggplot2 clearly explains this:

If you are new to ggplot2 you are better off starting with a systematic introduction, rather than trying to learn from reading individual documentation pages.

But what if learning about complex subjects could be easier? What if examples were always up-to-date and explanations plentiful? That's where the Elixir ecosystem shines! Imagine a world where package documentation is interactive, intuitive, and beginner-friendly - even when the subject matter is difficult.

Let’s talk about documentation in Elixir-land!

Imagine if you’re visiting your common dependency registry, where you usually would search and find a package and some instructions for how to install it with your project, and of course a link to the package’s main webpage. In Elixir-land, this is where hexdocs.pm enters.

hex is the ordinary Package Manager, but its documentation site is more than a place for installation instructions. Here you can find documentation for Elixir as well as every package available via hex. You would not need to leave hexdocs.pm for reading about more than installation instructions. It contains the API for each package, and in many cases the guides/tutorials for how to use them. And as everything with hexdocs.pm is managed on GitHub, anyone can be on top of any issues with documentation (such as rectifying poor explanations or adding or fixing examples) by issuing PRs.

Some of these API documents also have guides connected to them; occasionally written in a format called livemd. These are markdown files that also works as live documents; meaning they are able to be launched with Livebook (either locally hosted or in cloud) and experimented with. This way, anyone can see how each function and struct can be used.

What is Livebook

Livebook is a format for interactive Elixir notebooks. Like Jupyter, but specialized for Elixir (and Erlang). They are run as web applications. And when you have a Livebook application, you may open any livemd-document in it. You will see a "Run in Livebook" button on these pages.

Clicking it will open a new page where you can enter a URL to your local or cloud hosted Livebook application. You may also configure hexdocs.pm so it remembers your URL.

Hosting Livebook on Hugging Face

Hugging Face offers many alternatives for hosting. All it requires is that you've registered an account. Then you can onboard to their free hosting alternative. I'm registered as beercan, and let's see how onboarding would look like for me.

Create a new Space: Fill in the basic information for the Hugging Face Space.
Select the Space SDK: Possibilities are endless, as Hugging Face offers a Docker hosting alternative. Select Docker and the Livebook template.
Select hardware: We will run a basic setup, which offers 2 virtual CPUs with 16GB RAM. Other, priced, alternatives exists. For most cases, vCPUs are enough. If you want to be training your own models or use LLMs, then the need for the GPU alternatives are there for your choosing. You may also upgrade/downgrade at any point.
Secrets: Select a password for your Livebook application, and then let XLA_TARGET be cpu. (This should reflect the type of hardware that you are running your application on.)
Make your Livebook Private or Public: Select whatever you are comfortable with. Start with Private if you don't plan to make it visible to anyone.
Click on Create Space. This will build the docker image and spin it up. Per default, the container will sleep after 48 hours of inactivity. Any of the paid alternatives allows to change this.

When the image is done building, it will be run, and you will be able to see this screen:

Clicking on "Dashboard" will lead you to an authentication page where you will enter the password that you set earlier. Once that is passed, you can start to explore what Livebook is. It has multiple livebook documents teaching the basics of Elixir, Livebook, and some common Livebook components.

Explore a Livebook document - Bumblebee

Let's head to hexdocs.pm/bumblebee/examples.html#image-classification. Click the cog-icon in the upper right corner to access the settings and check "Run in Livebook". Paste your Livebook URL in the field. It should be something like https://beercan-liveboook.hf.space/, which is https://{username}-{projectName}.hf.space.

Now when we click on "Run in Livebook" it will initiate a new Livebook document for us in our Huggning Face space. Now we can configure and execute (or evaluate) each Elixir code element in our document. The first Elixir element is where we will install all the dependencies that will be covered in the document. It can be executed by hovering the element and click the Evaluate button that will appear.

Clicking it will start the code execution. Each element will have access to variables declared in earlier elexir elements, so you may define variables that will be used later in the code.

With Livebook we have access to a library of Livebook components in the form of the Kino dependency. In this example we will see how it renders an Image input-field:

And to wrap this gift that is the Bumblebee library in Livebook, let's run an inference model on the image that was imported and see how it classifies the badger laying on the ground.

In this case, the model that was used was pretty sure that it indeed was a badger, with a .3% uncertainty about it.

In closing

Remember how easy it was to set up a Livebook environment on Hugging Face? And how the interactive examples in Bumblebee's documentation made understanding the library a breeze? This is just a taste of how Elixir and Livebook can streamline your learning of complex topics. Whether you're an Elixir beginner or a seasoned developer, there's always something new and exciting to discover. Go forth and explore – you might be surprised by the creative possibilities that await!

Resize Image Uploads with Phoenix LiveView

Anders Björkland — Thu, 01 Feb 2024 07:00:17 +0000

In this article we will see how to resize an image before uploading it to a Phoenix backend. We will do this by adding a profile image to the user in a basic Phoenix setup. This will take us through the process of modifying an existing schema, create and run a migration, write custom hooks, add file validation, and ultimately create a new file. So if you are up for the task and would like to follow along, here’s the resize/upload flow:

Prerequisites and Setup

Docker
Elixir
Mix

Run a postgres container:
docker run --name phoenix_db -e POSTGRES_PASSWORD=phoenix_pwd -e POSTGRES_USER=phoenix_usr -d -p 5432:5432 postgres

If you do not have a Phoenix application, run:
mix phx.new my_app

If phx.new is not a recognized mix-command, install the command with mix archive.install hex phx_new. At the time of writing, this will install version 1.7.10.

Setup the application to use the correct database credentails in ./config/dev.exs. Adjust in the top of the file so it corresponds with your docker container setup:

# …
# Configure your database
config :my_app, MyApp.Repo,
  username: "phoenix_usr",
  password: "phoenix_pwd",
  hostname: "localhost",
  database: "my_app_dev",
  stacktrace: true,
  show_sensitive_data_on_connection_error: true,
  pool_size: 10

# …

With database configuration ready we can set up the database with mix ecto.create. We will then add a Phoenix LiveView authentication system by first running mix phx.gen.auth Accounts User users. And then run mix ecto.migrate.

Modify the User schema

We will have a profile image added to our new User model. The user will only ever have one of these and we will not preserve any kind of history beyond what the user currently has as their profile image. We plan to add a string field to the User model and schema to store the path for the image. Let’s create a migration file for this.

From the CLI we run mix ecto.gen.migration add_profile_image.

This creates a new file priv/repo/migrations/[timestamp]_add_profile_image.exs with the backbone for a migration-change. We will alter the table for users by adding a new field:

defmodule ArchAngler.Repo.Migrations.AddProfileImage do
  use Ecto.Migration

  def change do
    alter table(:users) do
      add :profile_image, :string
    end
  end
end

Next we will run this migration with mix ecto.migrate.
Having updated the database schema for the users table, we now turn to update our model accordingly. The model is defined in ./lib/my_app/accounts/user.ex. Let’s adjust the schema:

defmodule MyApp.Accounts.User do

  # …
  schema "users" do
    # add the profile_image field here
    field :profile_image, :string

    timestamps()
  end
  #...

  # add a new changeset function for the new field by the end of the module:
  @doc """
  A User changeset for updating Profile Image
  """
  def profile_image_changeset(user, attrs, _opts \\ []) do
    user
    |> cast(attrs, [:profile_image])
  end

end

With the updated User model, we should update the context as well. We find it in ./lib/my_app/accounts.ex. Add the following function to the module:

 @doc """
  Returns an `%Ecto.Chageset{}` for updating the user profile_image.
  """
  def change_profile_image(user, attrs \\ %{}) do
    User.profile_image_changeset(user, attrs)
  end

  def update_profile_image(%User{} = user, attrs) do
    user
    |> User.profile_image_changeset(attrs)
    |> Repo.update()
  end

This will be used to create a form for the LiveView component.

Update User Settings page

With updates to the User model, we can now update the LiveView component for the User Settings page. Currently the view has two forms: One for changing Email, and another for changing Password. We will add a new form for uploading and changing the Profile Image.

We will find the LiveView component in ./lib/my_app_web/live/user_settings_live.ex. In its render function, we will add our new form. Insert the following code above the email-form.

      <div>
        <div id="image-uploader-container">
          <label>
            <div class="block text-sm font-semibold leading-6 text-zinc-800">Profile Image</div>
            <input
              id="profile-image-uploader"
              type="file"
              accept="image/png, image/jpeg, image/jpg, image/webp"
              phx-hook="Resize"
            />
          </label>

          <%= if @image_error do %>
            <p class="text-red-500"><%= @image_error %></p>
          <% end %>
        </div>

        <.simple_form
          for={@image_form}
          id="profile_image_form"
          phx-submit="update_profile_image"
          phx-change="validate_profile_image"
        >
          <%= if @profile_image do %>
            <img src={@profile_image} alt="Profile Image" />
          <% end %>
          <div class="">
            <div id="image-preview-container">
              <%= if @resized_image_src do %>
                <img src={@resized_image_src} alt="preview resized image" />
              <% end %>
            </div>
          </div>

          <.input
            id="profile-image-src-input"
            field={@image_form[:profile_image]}
            type="hidden"
            value={@resized_image_src}
            required
          />

          <:actions>
            <%= if @allow_image_upload do %>
              <.button phx-disable-with="Changing..." id="submit-image-btn" class="">
                Submit image
              </.button>
            <% end %>
          </:actions>
        </.simple_form>
      </div>

      <%!-- below is Email-form --%>

Couple of things to note here: we are currently referring to Phoenix functions and socket assignments that currently do not exist. We are missing these functions in the LiveView component:
validate_profile_image, which will be called for every change in the form.
update_profile_image, which will be called upon form-submit.

The following socket assignments are missing:
image_form, a form-map that holds the field we want to parse.
image_error, a variable that we will add an error message to, if needed.
profile_image, a variable that will hold the current profile image.
resized_image_src, will hold a data URL of a resized image if validated.
allow_image_upload, will be false until validated.

On top of these Elixir-related changes, we will also make use of LiveView’s JavaScript Interoperability capabilities via its hooks. We will need a Resize hook, which will be responsible to resize an image to within specified parameters, and once that is done, it will update the value of the hidden input field.
Let’s start with the Elixir updates in the LiveView component (user_settings_live.ex). We will add the assignments first. These are added into the second mount function of the component:

def mount(_params, _session, socket) do
    user = socket.assigns.current_user
    email_changeset = Accounts.change_user_email(user)
    password_changeset = Accounts.change_user_password(user)

    # Let’s get the changeset for the image
    profile_image_changset = Accounts.change_profile_image(user)

    socket =
      socket
      |> assign(:current_password, nil)
      |> assign(:email_form_current_password, nil)
      |> assign(:current_email, user.email)
      |> assign(:email_form, to_form(email_changeset))
      |> assign(:password_form, to_form(password_changeset))
      |> assign(:trigger_submit, false)

     # Our new assigns are added here
      |> assign(:image_form, to_form(profile_image_changset))
      |> assign(:image_error, false)
      |> assign(:profile_image, user.profile_image)
      |> assign(:resized_image_src, false)
      |> assign(:allow_image_upload, false)

    {:ok, socket}
  end

Next up would be adding the event handlers for validate_profile_image and update_profile_image. But for the validation part it would be best to have a separate module that can validate by file size and type. This module’s validate function can then be called in our LiveView component. Create the file ./lib/my_app/image_validator.ex with the following content:

defmodule MyApp.ImageValidator do
  def validate_image(data_url, allowed_file_size, allowed_file_types)
      when is_binary(data_url) and is_integer(allowed_file_size) and is_list(allowed_file_types) do
    # Extract the media type and the base64 part from the data_url
    [meta_data, base64] = String.split(data_url, ",")

    [media_type, _] = String.replace(meta_data, "data:", "") |> String.split(";")

    # Decode the base64 part and get the binary data
    binary_data = Base.decode64!(base64)

    # Get the file size in bytes
    file_size = byte_size(binary_data)

    # Get the file extension from the media type
    file_ext = String.split(media_type, "/") |> Enum.at(1) 

    # Check if the file extension is allowed
    case Enum.member?(allowed_file_types, file_ext) do
      true ->
        # Check if the file size is less than or equal to 2MB
        case file_size <= allowed_file_size do
          true ->
            {:ok, data_url}

          false ->
            # Return an error message for invalid file size
            {:error, "Image size must be less than or equal to #{allowed_file_size}"}
        end

      false ->
        # Return an error message for invalid file extension
        {:error, "Image file type must be jpg, jpeg, png, or gif"}
    end
  end
end

Let’s write out our event handler for validate_profile_image back in user_settings_live.ex:

  def handle_event(
        "validate_profile_image",
        %{"user" => %{"profile_image" => profile_image}},
        socket
      ) do
    case MyApp.ImageValidator.validate_image(profile_image, 2 * 1024 * 1024, ~w(jpg jpeg gif png webp)) do
      {:ok, data_url} ->
        {:noreply,
         socket
         |> assign(resized_image_src: data_url, allow_image_upload: true, image_error: false)}

      {:error, message} ->
        {:noreply, socket |> assign(allow_image_upload: false, image_error: message)}
    end
  end

We can see how we are using pattern matching to unpack profile_image from the incoming parameters variable. This data is passed into the validate_image function, along with the maximum allowed file size (2MB) and allowed file types. As the function returns either an ok atom along with the image data, or an error and message, we will adjust the socket assignments accordingly.

Now, let’s add the event handler for update_profile_image in the same user_settings_live.ex, along with two helper functions:

  def handle_event(
        "update_profile_image",
        %{"user" => %{"profile_image" => profile_image}},
        socket
      ) do

    # Make sure that the image data is still valid
    case MyApp.ImageValidator.validate_image(
           profile_image,
           2 * 1024 * 1024,
           ~w(jpg jpeg gif png webp)
         ) do

      {:ok, data_url} ->
        handle_update_profile_image(data_url, socket)

      {:error, message} ->
        {:noreply, socket |> assign(allow_image_upload: false, image_error: message)}
    end
  end

  defp handle_update_profile_image(profile_image, socket) do
    [meta_data, base64] = String.split(profile_image, ",")

    file_name = generate_file_name()

    image_binary = Base.decode64!(base64)

    destination =
      Path.join([:code.priv_dir(:my_app), "static", "uploads", Path.basename(file_name)])

    File.write!(destination, image_binary)

    src_path = ~p"/uploads/#{Path.basename(destination)}"

    prev_file_name = socket.assigns.current_user.profile_image

    case Accounts.update_profile_image(socket.assigns.current_user, %{profile_image: src_path}) do
      {:ok, _} ->
        # If update was ok, then remove previous image
        case String.length(prev_file_name) > 0 do
          true ->
            previous_file_path =
              Path.join([
                :code.priv_dir(:my_app),
                "static",
                "uploads",
                Path.basename(prev_file_name)
              ])

            File.rm!(previous_file_path)
        end

        # Send back the updated image path and reset upload assignments
        {
          :noreply,
          socket
          |> assign(:profile_image, src_path)
          |> assign(:resized_image_src, false)
          |> assign(:allow_image_upload, false)
        }

      {:error, _} ->
        {:noreply, socket |> assign(image_error: "Could not update new image to user.")}
    end
  end

  defp generate_file_name do
    # Get the current timestamp in milliseconds
    timestamp = DateTime.utc_now() |> DateTime.to_unix(:millisecond)

    # Generate a random string of 8 characters
    random_string = :crypto.strong_rand_bytes(8) |> Base.encode64(padding: false)

    # Concatenate the timestamp and the random string with a dash
    "#{timestamp}-#{random_string}"
  end

The new function will create a file in the directory at ./priv/static/uploads. Create the directory if it does not exist. Then define it as one of the allowed static directories with the function static_paths in ./lib/my_app_web.ex:

  def static_paths, do: ~w(assets fonts images uploads favicon.ico robots.txt)

We will be making use of a JavaScript LiveView Hook here as well. It will be mounted on a file-input and resize any incoming images and set the resulting data URL (including base64 bytestring) to the hidden input-field. When that is done, it will trigger a change-event on the form which in turn will trigger the validate-change hook in Phoenix.

Lets first create a JavaScript module that can resize an image. Create the file ./assets/js/image_resizer.js with the following content:

export function resizeImage(image, maxWidth, maxHeight) {
    let scale = 1
    let width = image.naturalWidth
    let height = image.naturalHeight

    if (width > maxWidth) {
        scale = image.width / maxWidth
        width = maxWidth
        height = image.height / scale
    }

    if (height > maxHeight) {
        scale = image.height / maxHeight
        height = maxHeight
        width = image.width / scale
    }

    width = Math.floor(width)
    height = Math.floor(height)

    const canvas = document.createElement("canvas")
    canvas.width = width
    canvas.height = height

    const context2D = canvas.getContext("2d")
    context2D.drawImage(image, 0, 0, width, height)

    return canvas.toDataURL();
}

The let’s create the Resize hook and use this function in ./assets/js/app.js:

// …
// add import for the resizeImage function
import { resizeImage } from "./image_resizer"

// Then create our hook
let Hooks = {}
Hooks.Resize = {
    mounted() {

        const maxWidth = 250
        const maxHeight = 250

        let imageInput = document.getElementById("profile-image-uploader")

        // Inspired by blog-post https://imagekit.io/blog/how-to-resize-image-in-javascript/
        imageInput.addEventListener('change', function (e) {
            if (e.target.files) {

                let imageFile = e.target.files[0];

                var reader = new FileReader();
                reader.onload = function (e) {
                    const image = new Image()
                    image.onload = () => {
                        const dataUrl = resizeImage(image, maxWidth, maxHeight)

                        if (dataUrl.length === 0) {
                            return;
                        }

                        let resizedImage = document.getElementById("image-preview") ?? new Image()
                        resizedImage.setAttribute("id", "image-preview")
                        resizedImage.setAttribute("alt", "Preview")
                        resizedImage.src = dataUrl

                        document.getElementById("image-preview-container").appendChild(resizedImage)

                        hiddenInput = document.getElementById("profile-image-src-input")
                        hiddenInput.value = dataUrl
                        hiddenInput.dispatchEvent(
                            new Event("input", {bubbles: true})
                        )
                    }

                    image.src = e.target.result
                }

                reader.readAsDataURL(imageFile);
            }
        })
    }
}

// and let’s make it available to our LiveView socket:
let liveSocket = new LiveSocket("/live", Socket, {params: {_csrf_token: csrfToken}, hooks: Hooks})

// …

We see how we can set a preview image from the dataUrl produced by the resizeImage function and append it into a container in the DOM. We are also setting it as a value on the hidden input element. This alone won’t trigger the form to emit a change event, so we explicitly do that with a dispatchEvent on the element. That in turn will trigger the phx-change instruction which will emit the validate_profile_image event, and if all is valid then a “Submit image” button will appear.

In summary

Phoenix LiveView is all about reacting to events. We can attach hooks and event handlers on most elements we choose. In this scenario we upload a file indirectly via a LiveView form, which has hooks and events that takes care of resizing and validating uploads whenever a new file has been chosen.

We take care of resizing on the client-side. We emit a change-event on the form to initialize a server-side validation on the resized image data. If everything appears OK, the upload is greenlit on the client and the user may submit the image. Upon submitting a new event is taking place and we will again reaffirm that the image data is valid before we create a file, store the updated image path to the user and then remove the old image.

Here’s our updated User Settings page:

Schroedinger's Snack Stash

Anders Björkland — Thu, 28 Dec 2023 15:59:11 +0000

In the heart of Beakerage, where sea serpents sunbathed and pocket-sized black holes were as common as gossip at the local bakery, Keith - the Quantum Quokka - awoke with a start! “May your macarons never vanish into the void.” The ethereal voice was still echoing in Keith’s mind. He couldn’t recall anyone making such an astute statement, yet here it was: haunting him! His pocket-blackhole had been in a binging phase lately, often pulling each of his emergency snacks beyond the event horizon, leaving barely a crumb for sustenance.

Beyond Beakerage - Elixir conundrums and learnings

Module Definition

Aliasing modules

Struct Definition

Struct Manipulation

Pipelining

Elixir Documentation

Function Definition

Private Function

Pattern Matching

DateTime Manipulation

String Interpolation

Enum Mapping

Writing tests

Test for errors

It had been a conundrum, one which he thought he had solved with the help of his sunspot friend, Sylvie. Keith put on his slippers and waddled over to his desk while pondering. He always carried his pocket-blackhole in his ‘emergency’-pocket, and could not make use of his ‘safekeeping’-pocket for his snacks. That would violate the quantum coherence keeping his blackhole functional. But a third pocket would keep a snack in superposition in perpetuity, until observed! He had deftly solved it by adding a QuantumPocket, but feared that it was not as robust as he would like. These thoughts kept bugging his dreams.

Carefully sorting away his notes on “the great disco ball multiverse theory” (this is a story for another time), he laid down a fresh sheet of paper and sat down. “No sleep tonight, but a solution to behold I hope”, Keith thought to himself as he leaned over the desk and grabbed a pair of safety goggles and put them on. His tiny hands started scribbling furiously. His quokka brain buzzed with ideas. He needed a pocket that defied both entropy and existential uncertainty: a pocket that would not collapse into a probability haze!

As the night turned to dawn, the first glimpses of a solution appeared on the horizon. Keith had his blueprint ready. He found that his pocket would need three components:
Stardust Silk, a robust and hardy material that would withstand quantum fluctuations by weaving memories together.
Time-sprinkled Velcro, would stick to itself across parallel dimensions, fastening past with the future.
Uncertainty Thread, would surely leave snacks in a delightful superposition by threading memories within a bubble beyond space-time.

While many would need to cross the Severely Sinister Sea in search of moon spiders and pixieworms, Keith had all he needed in his right pocket: a blackhole with an Elixir interface!

Setting out to construct the weave, Keith pulled out an otherworldly silk by his fingertips:

defmodule StardustSilk do
  @doc """
  Weave the Stardust Silk, capturing cosmic memories
  """
  def weave(memories) when is_list(memories) do
    memories
    |> Enum.map(&capture_memory/1)
    |> Enum.join
  end

  defp capture_memory(memory) do
    "🌟 #{memory} 🌟"
  end
end

To make sure to keep the nightmares at bay, Keith realized he would need to construct tests for each module as well:

defmodule StardustSilkTest do
  use ExUnit.Case

  test "weave memories into Stardust Silk" do
    memories = ["Galaxy memories", "Comet sighs", "Black hole laughter"]
    expected_result = "🌟 Galaxy memories 🌟🌟 Comet sighs 🌟🌟 Black hole laughter 🌟"

    assert StardustSilk.weave(memories) == expected_result
  end
end

Keith threaded his tiny needle, its tip sharper than a quasar’s wit. The Stardust Silk shimmered, each strand holding a memory: the birth of a galaxy, the sigh of a comet, the laughter of a black hole. Keith’s quokka brain buzzed with anticipation. He was ready to weave eternity. Safe, knowing it would hold a cosmic weight worth of snacks - as long as it withstood the pressure and tearing of the space-time-continuum.

Looking over his blueprint again; time-sprinkled velcro would hold the snack secure over past, present, and future - up until the time the bubble would be burst by Keith’s nimble fingers searching for a macaron. Knowing that the velcro will require a datetime, he deftly made use of pattern matching to structure the time-cohesive:

defmodule TimeSprinkledVelcro do
  @doc """
  Attach the Velcro to moments, bridging past and future
  """
  def attach(%DateTime{} = moment) do
    past = DateTime.add(moment, -1, :day)
    future = DateTime.add(moment, 1, :day)

    "🕰️ (from #{past} to #{future}) 🕰️"
  end
end

And the accompanying tests:

defmodule TimeSprinkledVelcroTest do
  use ExUnit.Case

  test "attach Velcro to moments" do
    datetime = ~U[2023-12-26T07:51:11Z]
    past = ~U[2023-12-25T07:51:11Z]
    future = ~U[2023-12-27T07:51:11Z]

    # Test the function with a specific moment
    expected_result = "🕰️ (from #{past} to #{future}) 🕰️"

    assert TimeSprinkledVelcro.attach(datetime) == expected_result
  end

  test "raises an error for invalid input" do
    assert_raise FunctionClauseError, fn ->
      TimeSprinkledVelcro.attach("2023-12-26T07:51:11Z")
    end
  end
end

Having pulled both the stardust silk and the time-sprinkled velcro from the abyss, the Quantum Quokka contemplated how the uncertainty thread best would keep its characteristics once it was drawn from the blackhole as well. It all would come down to how the thread was observed! So letting something else do its magic with the thread would solve the dilemma:

defmodule UncertaintyThread do
  alias StardustSilk, as: Silk
  alias TimeSprinkledVelcro, as: Velcro

  @doc """
  Thread each possible outcome together with rigorous uncertainty
  """
  def thread(unknown, %Silk{} = silk_struct) when is_function(unknown, 1) do

    memories_with_result =
      Enum.map(silk_struct.memories, fn memory ->
        amount = unknown.(:rand.uniform(42))
        "#{memory} (#{amount})"
      end)

    silk = Silk.weave(memories_with_result)
    velcro = Velcro.attach(DateTime.utc_now)

    %{silk: silk, velcro: velcro}
  end
end

And testing for the expected behavior:

ExUnit.start()

defmodule UncertaintyThreadTest do
  use ExUnit.Case

  test "threads memories and results" do
    silk_struct = %StardustSilk{memories: ["Galaxy memories", "Comet sighs"]}
    result = UncertaintyThread.thread(fn _ -> 42 end, silk_struct)

    assert result.silk == "🌟 Galaxy memories (42) 🌟🌟 Comet sighs (42) 🌟"

    expected_pattern = ~r/🕰️ \(from \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{6}Z to \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{6}Z\) 🕰️/
    assert Regex.match?(expected_pattern, result.velcro)
  end
end

StardustSilk and TimeSprinkledVelcro with UncertaintyThread weaves a handsomely new pocket under Keith’s nimble fingers. When he attached the new pocket to his coat he already felt a light tingle of anticipation. Schroedingers Snack Stash 2.0! While he could never be certain of the number of snacks he could retrieve, he now knew that the combination of the memories of the universe and the time spent in the past would fix some amount of snacks into the present. No longer was the dilemma whether there would be a snack or not, but whether there would be plenty or a few.

And the Quantum Quokka reached into his pocket and found a fistful of probability-flavored macarons. A comet far-away sighed with relief, and on a beach much closer a sea serpent blessed the sun as it turned over to bathe in its rays. Who knew when a day would come when tampering with the space-time continuum would break Beakerage, but today was not that day. The Quantum Quokka looked at the fistful of macarons and thought to himself; “may you never vanish into the void”, and stuffed his mouth full of the delightful confections.

Beyond Beakerage - check your Elixir knowledge

How do you define a module?

Why would you use a struct instead of a map?

Did you see any use of pipelining?

Would you prefer using pattern matching or a “when” guard for making sure a function gets the correct type of argument?

How can you test that an error will be thrown?

The Macaron Paradox

Anders Björkland — Thu, 21 Dec 2023 09:25:58 +0000

It was a bustling town tearing by its threads. It was on the cusp of being called a city by the travelers arriving by boatloads each day, were it not for the inhabitants wielding their favorite pastry in a threatening way on even the slightest suggestion that their town was not just a little hamlet by the harbor. Once upon a time it was! And it was the way that many residents would like to keep it; a place separated from the world and the succession of time. But then the sunbathing sea serpents started dwelling on its shores, word spread of its many characters and possibilities, and so the world became aware of the quaint community by the Severely Sinister Sea: Beakerage!

On a sunny afternoon, Keith reclined on a beach towel. His coat unbuttoned to reveal his small frame as his fur absorbed the sun’s rays like a solar-powered calculator. Many knew him affectionately as the Quantum Quokka. Next to him was one of the sea serpents; Sylvie. Her scales shimmering in the sun in shades of red and purple. They would often find themselves sharing the sunspots and the occasional algebraic equation. Having just ended a lengthy discussion on the application of Schrödinger Equations for slowing down the time in Beakerage, Keith found himself in a sticky situation: his stomach had started growling. His emergency snack radar pinged - it was time for macarons!

Jumping up on his two feet, Keith quickly put his hand down his right pocket, the one designated for emergency but found only void - literal void!

Arching a non-existing eyebrow, Sylvie gave a questioning glance. “The pocket-sized black hole, again?”

“Curse you, entropy!” Keith muttered, shaking his tiny fist at the void. “Why must you devour my macarons?”

“Entropy?” Sylvie questioned. “Are you sure it’s not just a hungry singularity?”

“Same difference.” Keith sighed. “But it’s a conundrum I can’t seem to escape: I can’t put my emergency macarons in my left ‘just for safekeeping’ pocket - that would violate the principles of quantum coherence”

Sylvie blinked. “You don’t make much of a coherent sense right now!”

“Quantum coherence,” Keith started. “You see, in our world, pockets exist in a delicate superposition. They’re simultaneously ‘emergency’ and ‘safekeeping.’ If I put an emergency item in my left pocket, it collapses the waveform, and suddenly it’s no longer safekeeping!”

“That’s not how it is in our world. I think it’s only you. But you would need a third pocket?” Sylvie wondered.

Keith jumped at the idea. “Exactly! I need a ‘Schroedinger’s Snack Stash’. It’ll exist in both states until observed. That way, my macarons won’t vanish.”

Opening his pocket-sized black hole (carefully avoiding its event horizon), Keith coded an Elixir module:

defmodule QuantumPocket do
  defstruct snacks: []

  def add_snack(pocket, snack) when is_list(snack) do
    %{pocket | snacks: pocket.snacks ++ snack}
  end
end

Keith summoned his freshly baked pocket and stored a little macaron crumb in it:

keiths_pockets = %QuantumPocket{}
safekeeping_pocket = QuantumPocket.add_snack(keiths_pockets, ["Emergency Macarons (crumb)"])

“Are you sure that the little crumb is still there?” Sylvie inquired.

Keith looked in the pocket to verify:

IO.inspect(safekeeping_pocket.snacks)  # ["Emergency Macarons (crumb)"]

With a sigh of relief Keith looked over at Sylvie. “Yeah, it’s all good. Just wish it was the whole set, a crumb won’t carry me over until lunch.”

“That’s the beauty of sunspot friends then. It’s not a probability flavored macaron, but have some of my milkshake that probably tastes of strawberry”.

Keith grinned. “Looks like I’ve cracked the code, but you sure saved the afternoon”. And so, Keith, the Quantum Quokka, reclined once more. Sylvie nodded approvingly and followed suit.

And thus, the legend of Schroedinger's Snack Stash was born - a tale of quantum pockets, sweet paradoxes, and the eternal struggle between hunger and coherence.

Challenge: Your ugliest code

Anders Björkland — Sun, 11 Dec 2022 08:29:33 +0000

I work with web development daily. It is one of my joys, but can also be a source of great frustration. A recent frustration tracking down some logic in a third-party library led me to a discovery. It was a method declaring void as its return-value. Reasonably, one would assume that it wouldn't return any value as such. Maybe it would mutate a value, but that was all. Was this the case though? NO.

The method would parse some input value and then it would throw its calculated value out into the abyss, hoping there would be a catch-statement somewhere that would handle it.

I don't know if this is a common pattern and if it has any name. I suggested to one of my colleagues that we name it "the anti prop-drill": it isn't a prop, it isn't passed through step-by-step, and it is thrown in an opposite direction. I'm open to hear your suggestions, or maybe make me understand if this is a reasonable pattern.

Anyway, this inspired me to write the ugliest (and funniest) code I have ever written. This is exceptional coding!

<?php

function sum(array $array)
{
    $sum = 0.0;

    try {
        $i = 0;
        while(true) {
            $value = $array[$i] ?? throw new \Exception('' . $sum);
            $sum += $value;
            $i++;
        }
    } catch (Exception $e) {
        return (float)$e->getMessage();
    }
}

$values = [3, 2.5, 0.5];
echo 'Sum of ' . implode(' + ', $values) . ' = ' . sum($values) . PHP_EOL;

Output: Sum of 3 + 2.5 + 0.5 = 6

PHP doesn't automatically throw an "Array Index out of Bound" Exception. Instead it returns null. So $array[$i] ?? throw new \Exception checks if the value is null and if so it will throw a new exception with our $sum wrapped in its message.

What is the ugliest code you have achieved?

Configure Email over SMTP with SilverStripe

Anders Björkland — Thu, 27 Jan 2022 14:47:59 +0000

If you have used SilverStripe CMS, a lot of it may seem intuitive to you. But as a newcomer there may be a few things you might wish there were more documentation on. For me, one of those things was how to configure the CMS to use my mailtrap.io account for my development environment. For today, that is what we will do.

Mailtrap is a service that catches all outbound mails from your application and let you view them on their website. You will see a mail as in most mail clients, with the addition of switching views between different outputs: HTML, HTML-source, Text, and Raw. Suffice to say; this is not something that would be a good idea to use on a production-server. Mailtrap uses SMTP, so when we are configuring here for Mailtrap - the same configurations can be used for your SMTP service with updated values.

We will configure the following environment variables (be it in .env or "first class environment variables"):

SS_SEND_ALL_EMAILS_FROM="example@smtp.mailtrap.io"
APP_SMTP_USERNAME="example"
APP_SMTP_PASSWORD="password"
APP_SMTP_HOST="smtp.mailtrap.io"
APP_SMTP_PORT="2525"

If you are using Mailtrap, you will find your SMTP credentials at https://mailtrap.io/inboxes/ and then click on "My Inbox" or the inbox you want to use for your specific project if you have multiple inboxes. Under "SMTP Settings" -> "Integrations", click on PHPMailer to get a pretty good idea on which values are which.

Having configured your environment variables, we will have to update our configurations as well. Open up app\_config\email.yml (or create it if it doesn't exist). We will update our email service to use SMTP:

---
Name: project-emailconfig
After:
  - '#emailconfig'
---

SilverStripe\Core\Injector\Injector:
  Swift_Transport:
    class: Swift_SmtpTransport
    properties:
      Host: '`APP_SMTP_HOST`'
      Port: '`APP_SMTP_PORT`'
      Encryption: tls
    calls:
      Username: [ setUsername, ['`APP_SMTP_USERNAME`'] ]
      Password: [ setPassword, ['`APP_SMTP_PASSWORD`'] ]
      AuthMode: [ setAuthMode, ['login'] ]

Having updated SilverStripe like this, it may be a good idea to visit the dev/build URI (or run ?flush) to update the application to use it. This was appareantly a step that I missed.

How I stumbled onto the Reflection API

Anders Björkland — Fri, 21 Jan 2022 09:52:23 +0000

Coding for fun can sometime lead you into unexpected neighbourhoods. This is what happened to me this week. I've written previously about the micro-framework Framework-X and this week I was building another project with it and was in need of hooking it up with a database. Framework-X supports async and non-blocking execution. For this reason, while it is possible, the project recommends going for non-blocking database access. So it is absolutely possible to use Doctrine but it is not the reactive way that Framework-X promotes. So this had me leave the known grounds of Doctrine to explore a more "hands-on" approach of communicating with the database.

So I though, anyway.

Being me, I like to explore new subjects and see how far I can go with my current proficiency. I come from a background where all I've ever done has almost exclusively been "object oriented", but now being left without Doctrine (a popular PHP ORM) I saw an opportunity to build similar (but way more simplistic) functionality on my own.

I didn't think I'd set out to build a mini-ORM in my project but that's where I've found myself. I have entity-classes, or what you could describe as "plain old PHP-objects" (POPOs). These POPOs describe their own properties with types. Theses types may describe a relation to another entity as well as a one-to-many relation, reflected as an array in the other class (many-to-one). I don't have a many-to-many relation in the project currently. So a POPO could look like this:

<?php 

namespace App\Entity;

class Category
{
    /**
     * @var string $uuid - UUID version 4 variant 1
     */
    private string $uuid = '';

    private string $name = '';

    /**
     * @var Category[] $childCategories
     */
    private array $childCategories = [];

    private ?Category $parentCategory = null;

    public function setUuid(string $uuid = ''): self
    {
        $this->uuid = $uuid;
        return $this;
    }

    public function getUuid(): string
    {
        return $this->uuid;
    }

    public function setName(string $name): self
    {
        $this->name = $name;
        return $this;
    }

    public function getName(): string
    {
        return $this->name;
    }

    public function setChildCategories(array $childCategories): self
    {
        $this->childCategories = $childCategories;
        return $this;
    }

    public function getChildCategories(): array
    {
        return $this->childCategories;
    }

    public function addChildCategory(Category $childCategory): self
    {
        if (!$this->hasChildCategory($childCategory)){
            $this->childCategories[] = $childCategory;
        }
        return $this;
    }

    public function removeChildCategory(Category $childCategory): self
    {
        $this->childCategories = array_filter($this->childCategories, function ($category) use ($childCategory) {
            return $category->getUuid() !== $childCategory->getUuid();
        });

        return $this;
    }

    public function hasChildCategory(Category $childCategory): bool
    {
        return in_array($childCategory, $this->childCategories);
    }

    public function setParentCategory(Category $parentCategory): self
    {
        $parentCategory->addChildCategory($this);
        $this->parentCategory = $parentCategory;
        return $this;
    }

    public function getParentCategory(): ?Category
    {
        return $this->parentCategory;
    }
}

Having POPOs like this, I wanted to be able to store these objects in the database, as well as fetch results from the database and turn them back into objects. For this to work for me, I needed to know what types each property is. In the case of arrays, I needed to know the type of the objects inside the array. Normal properties simply has a stated type, for arrays I use the @var annotation:

    /**
     * @var Category[] $childCategories
     */
    private array $childCategories = [];

For each property I also have a getter and a setter. Knowing all this, I can get all relevant properties when I want to store them in the database. With enough insight into the class I can turn each property into a corresponding SQL-type: a string into VARCHAR for MySQL for example, and a reference into a VARCHAR (as I'm using UUID) along with a foreign-key.

But from where do I get the relevant insight?!

That would be the ReflectionClass! Without it I would have to parse each class-file line-by-line, but instead I can use getMethods, getProperties, getType, getDocComment. What I found most fun in this project was fetching the annotation type hint for the arrays. I built a method in a utility class to fetch those:


    // Inside a class called App\Utility\EntityReflection.php

    public static function getPropertyTypeFromComment(string $entityClass, string $property): ?string
    {
        $reflectionClass = new \ReflectionClass($entityClass);
        $reflectionProperty = $reflectionClass->getProperty($property);
        $propertyComment = $reflectionProperty->getDocComment();

        $refType = preg_grep('/([A-Z])\w+/', explode(" ", $propertyComment));

        if (count($refType) > 0) {
            $refType = array_pop($refType);
        } else {
            return null;
        }

        if ($refType === null) {
            return null;
        }

        if (strpos($refType, '[]') !== false) {
            return $refType;
        }

        return null;    
    }

I could call this with $commentType = EntityReflection::getPropertyTypeFromComment(Category::class, 'categories');, which would return Category[].

And that's how I stumbled upon the Reflection API. I now use it successfully storing and fetching objects from a relational database. But that's another story. So have you used reflection in any of your objects, and how did you like it? As I may have hinted at, I find it pretty fun, magical and very useful!

Create reviews for Googled books (in SilverStripe)

Anders Björkland — Thu, 13 Jan 2022 11:25:25 +0000

Today we wrap up our book review platform in SilverStripe. When we last left off we had created a search page that displayed results from querying the Google Books API. Here's what we have planned for today:

Secure the /review URL so that only logged-in users can search for books to review.
Add a review creation/edit page.
Add a review-listing for the homepage.
Add a search form for the homepage.
Tidy up the admin interface.

The code for this article and the project so far is at https://github.com/andersbjorkland/devto-book-reviewers/tree/store-reviews

Note that I reference a theme called simple in the articles but in my repository all the modifications from that theme is located under reviewers template instead.

Securing the review-route

Our plan is to allow logged on users to create reviews for books. The flow is as follows:

A user goes to /review and search for a book they want to review.
From the list of results the user selects a book to review.
The user is redirected to /review/book/{volumeId} where they can create a review for the book.

In the last episode we created a controller to handle the search for books. We will use this controller to also handle the review creation page. This means that this controller should only be accessible to logged on users. While it is possible to secure each single route in SilverStripe, for our use case it is much easier to secure the entire controller. A way to do that is to check the user's privileges when the controller is called. Any SilverStripe controller has the method init() which is run before any of the controller's actions are executed, such as the index() method. Let's secure the entire ReviewController by adding the init() method to it:


class ReviewController extends ContentController
{
    // ...
    public function init() 
    {
        parent::init();

        if(!Permission::check('CMS_ACCESS_' . ReviewAdmin::class)) {
            return $this->redirect('/Security/login?BackURL=' . $this->request->getURL());
        }
    }
    // ....
}

If you have followed along this series you may recall that we created a security group in our second article. This group was called Reviewers and was assigned the permission to view the Review admin page. This admin page (or area) is controlled by the ReviewAdmin class. In the database, this group is related to a permission table entry called CMS_ACCESS_App\Admin\ReviewAdmin. It's this entry that Permission::check() looks up to see if the user has a corresponding privilege. If it does, the init() method does nothing else but allows for continued execution of the controller. If the user does not have the privilege, the init() method redirects the user to the login page. The BackURL parameter is set to the current URL so that the user is returned to the current page after they successfully log in.

Creating the review page

In the last entry in this series we created the template that would list the search results. We didn't highlight the fact that we had a Review "book-name"... link for each result: <a class="reviewLink" href="/review/book/{$volumeId}" title="Review "{$title}"">Review "{$title}"...</a>. Currently the href is pointing straight out into the void. We will correct this by adding an action to the ReviewController to respond to requests for /review/book/{$volumeId}. Here's the anatomy of the request:

/review - the controller's main route.
/book - the request's Action-parameter. A method called book() will be executed if it exists.
/{$volumeId} - the request's ID-parameter. It can be any string.

Add the book action to the controller

ReviewController.php

<?php

namespace App\Controller;

use App\Model\Author;
use App\Model\Book;
use App\Service\GoogleBookParser;
use SilverStripe\CMS\Controllers\ContentController;
use SilverStripe\Control\HTTPRequest;
use Symfony\Component\HttpClient\HttpClient;

class ReviewController extends ContentController
{
    private static $allowed_actions = [
        'index', // this action is normally inferred, let's add it explicitly
        'book'  // We add the "book" action as an allowed action. 
    ];

    public function init() 
    {
        // ...
    }

    public function index(HTTPRequest $request)
    {
        // ...
    }

    // Add the book-method. This method will be called if the user requests `/review/book/{$volumeId}`.
    public function book(HTTPRequest $request)
    {
        $volumeId = $request->param('ID'); // This retrieves the ID-parameter from the url.

        // Next we query the Google Books API to get the book details.
        $client = HttpClient::create();
        $response = $client->request('GET', 'https://www.googleapis.com/books/v1/volumes/' . $volumeId);
        $googleBook = GoogleBookParser::parse($response->toArray());

        // We create new Author objects and store them if they do not currently exist in our database.
        $authors = [];
        foreach ($googleBook["authors"] as $googleAuthor) {
            $author = Author::get()->filter(['Name' => $googleAuthor->AuthorName])->first();

            if (!$author) {
                $author = Author::create();
                $author->Name = $googleAuthor->AuthorName;

                $names = explode(" ", $googleAuthor->AuthorName);
                $author->GivenName = $names[0];
                if (count($names) > 2) {
                    $additionalName = "";

                    for ($i = 1; $i < count($names) - 1; $i++) {
                        $additionalName .= $names[$i] . " ";
                    }

                    $author->AdditionalName = $additionalName;
                }
                $author->FamilyName = $names[count($names) - 1];

                $author->write();

                $authors[] = $author;
            }
        }

        // We create a new Book object and store it in our database (if it doesn't exist already).
        $book = Book::get()->filter(['VolumeID' => $volumeId])->first();
        if (!$book) {
            $book = Book::create();
            $book->VolumeID = $volumeId;
            $book->Title = $googleBook["title"];
            $book->ISBN = $googleBook["isbn"];
            $book->Description = $googleBook["description"];

            foreach ($authors as $author) {
                $book->Authors()->add($author);
            }

            $book->write();
        }

        // We return a response which will be rendered with a new layout called 'Review'.
        return $this->customise([
            'Layout' => $this
                        ->customise([
                            'Book' => $googleBook,
                        ])
                        ->renderWith('Layout/Review'),
        ])->renderWith(['Page']);

    }

    protected function paginator($query, $count, $startIndex, $perPage): array
    {
        // ...
    }
}

themes\simple\templates\Layout\Review.ss

<div class="content-container unit size3of4 lastUnit">
    <section class="container">
        <h1>Review &quot;$Book.title&quot;</h1>
        $ReviewForm
    </section>
</div>

This is the layout that is injected into the Page template. It contains the title of the book and a variable $ReviewForm that will be a form returned from a method called ReviewForm in the controller.

Add the review form

Let's build the ReviewForm method in the controller and a form submission handler:

ReviewController.php

<?php

namespace App\Controller;

// ... use statements

class ReviewController extends ContentController
{
    private static $allowed_actions = [
        'index',
        'book',
        'ReviewForm'    // We add the "ReviewForm" action as an allowed action.
    ];

    public function init() 
    {
        // ...
    }

    public function index(HTTPRequest $request)
    {
        // ...
    }

    public function book(HTTPRequest $request)
    {
        // ...
    }

    public function ReviewForm()
    {
        // The Form is rendered on a page where a ID parameter is present. We will fetch a Book object from the database based on the ID.
        $volumeId = $this->request->param('ID');
        $book = Book::get()->filter(['VolumeID' => $volumeId])->first();

        // If the current user has reviewed the book before, we will fetch the review from the database.
        // This allows us to pre-fill the form with the user's previous review.
        $currentUser = Security::getCurrentUser();
        $review = Review::get()->filter([
                'MemberID' => $currentUser->ID, 
                'BookID' => $book->ID
            ])->first();

        // We create a list of fields for the form.
        $fields = new FieldList(
            [
                // This field holds a ID for the review if it already exists. (This lets us update the review instead of creating a new one.)
                HiddenField::create(
                    'ReviewId',
                    'ReviewId',
                    $review ? $review->ID : null
                ),

                // This field holds the ID of the book. It's required so we can store a relation to the book that the review is for.
                HiddenField::create(
                    'VolumeId',
                    'VolumeId',
                    $volumeId
                ),

                // The next 3 fields are the review itself - a heading (title), a rating and a review (the text body).
                TextField::create(
                    'Title',
                    'Title',
                    $review ? $review->Title : null
                ),
                DropdownField::create(
                    'Rating',
                    'Rating',
                    [
                        '1' => 1,
                        '2' => 2,
                        '3' => 3,
                        '4' => 4,
                        '5' => 5
                    ]
                )->setValue($review ? $review->Rating : null),
                TextareaField::create(
                    'Review',
                    'Review',
                    $review ? $review->Review : null
                )
            ]
        );

        // We create a form connected to this controller and give it a name 'ReviewForm'
        // and pass the fields we created above.
        // We also create a new field list which creates a submit button named `Submit` and "connect" it with the method 'doReview'. It also controls that the required fields are filled. 
        $form = Form::create(
            $this, 
            'ReviewForm', 
            $fields, 
            new FieldList(
                FormAction::create('doReview', 'Submit')), 
                new RequiredFields('Title', 'Rating'));

        // Normally the form what suffice as it is if this was based on a Page-object, but we are not, so we will set the form-action to the URL that will handle the form submission instead.
        $form->setFormAction('/review/ReviewForm/');

        return $form;
    }

    // This is the method tasked with handling the form submission.
    public function doReview($data, Form $form)
    {
        $book = Book::get()->filter(['VolumeID' => $data['VolumeId']])->first();

        // If a review already exists, the form will have a ReviewId field with the ID for it. Otherwise we create a new review.
        $review = $data['ReviewId'] ? Review::get_by_id($data['ReviewId']) : Review::create();

        // We set the review's properties based on the form data and save it.
        $review->Title = $data['Title'];
        $review->Rating = $data['Rating'];
        $review->Review = $data['Review'];
        $review->Member = Security::getCurrentUser();
        $review->Book = $book;
        $review->write();

        $form->sessionMessage('Your review has been saved', 'good');
        return $this->redirectBack();
    }

    protected function paginator($query, $count, $startIndex, $perPage): array
    {
        // ...
    }
}

What have we done so far? Let's have a look 👇

Review listing on homepage

We have made it so much easier to create a review. Now we need to make it so that the review is shown on the homepage. We are going to create a new Page-type that we can associate with a particular controller and in extension with a particular layout. We can then edit the homepage in the CMS to use the new Page-type.

The Homepage Page-type

Let's create a new Page-type called Homepage. This will be a page that uses a specific controller that we build in the next step (the HomepageController).

app\src\Page\Homepage.php

<?php

namespace App\Page;

use App\Controller\HomepageController;
use Page;

class Homepage extends Page
{
    public function getControllerName()
    {
        return HomepageController::class;
    }
}

The Homepage Controller

Let's create a new controller called HomepageController that will handle the homepage (or the Page-type associated with it). It will at first only be associated with the inferred index-action and contain one method to be accessed from the homepage layout (which we will build in the next step). The method is LatestReviews which will fetch the 5 latest reviews created.

app\src\Controller\HomepageController.php

<?php

namespace App\Controller;

use App\Model\Review;
use SilverStripe\CMS\Controllers\ContentController;

class HomepageController extends ContentController
{
    public function LatestReviews()
    {
        $reviews = Review::get()->sort('Created', 'DESC')->limit(5);
        return $reviews;
    }

}

The Homepage Layout

Let's create a new layout called homepage that will be used by the Homepage Page-type. It will display the latest reviews and their ratings.

themes\simple\templates\App\Page\Layout\Homepage.ss

<section class="container">
    <h1>The Home of Book Reviewers</h1>
    <div class="line">
        <br>
        <h2>Latest Reviews</h2>
        <% loop $LatestReviews %>
            <div>
                <h3>$Book.Title</h3>
                <p>
                    <b>$Title</b> <br>
                    $Review.FirstParagraph <br>
                    $RatingStars
                </p>
            </div>
        <% end_loop %>
    </div>
</section>

Modify the Review model

We want to get the rating in a review as the number of stars. We can do this by adding a new method to the Review model that will do this for us:

app\src\Model\Review.php


class Review extends DataObject
{
    // ...

    public function getRatingStars()
    {
        $rating = $this->Rating;
        $stars = '';
        for ($i = 1; $i <= $rating; $i++) {
            $stars .= '⭐';
        }
        return $stars;
    }

}

Editing the homepage in the CMS

The next step is for us to log into the CMS at /admin with administrator privileges and edit the homepage. On the pages-segment (/admin/pages), click on Home and then on the tab Settings in the upper right corner. The first setting is the Page Type. Currently set to Page but we want to change it to Homepage. Alas, Homepage does not appear in the drop down list. The reason is that we have adjusted something that needs to be updated in the database. In the browser, visit /dev/build to update the database with our new Page-type. Then go back to the Home page and click on the Settings tab again. The drop down list now contains Homepage. Select it! Then click Save, and then Publish. Going to the homepage in the browser will now show the new homepage. If you have created a few reviews, it might look something like this:

Make the reviews searchable

We have made it so that the reviews are shown on the homepage. Now we need to make it so that the reviews can be searched. We will expand the HomepageController and the Homepage.ss layout to include a search field. When submitting it will redirect to a book search result page.

Improve the HomepageController

We want to add a form so we can search for books. We will add it with the SearchBookForm method in the HomepageController:

app\src\Controller\HomepageController.php

<?php

namespace App\Controller;

use App\Model\Review;
use SilverStripe\CMS\Controllers\ContentController;
use SilverStripe\Forms\FieldList;
use SilverStripe\Forms\Form;
use SilverStripe\Forms\FormAction;
use SilverStripe\Forms\TextField;

class HomepageController extends ContentController
{
    // Add the form as an allowed action
    private static $allowed_actions = [
        'SearchBookForm'
    ];

    public function LatestReviews()
    {
        // ...
    }

    // Create a form for GET-requests.
    // This form will be pre-filled with the search query if it exists and submissions will go to a 
    // `doSearchBook` method.
    public function SearchBookForm()
    {
        $q = $this->request->getVars()['q'] ?? '';
        $form = Form::create(
            $this,
            'SearchBookForm',
            FieldList::create(
                TextField::create('q', 'Book title', $q)
            ),
            FieldList::create(
                FormAction::create('', 'Search')  // We set no associated action method as we will handle it in the BookController)
            )
        );
        $form->setFormMethod('GET');

        // Redirect to the search results page at `/book` if the form is submitted
        $form->setFormAction('book');
        return $form;
    }

}

We need to update our Book-model to include a getAverageRating() and getAverageRatingStars() methods. We won't have to run another dev/build as this is not a database-related change. (Naming it with the get prefix means that we could loop through books and access the average rating with $AverageRating or $AverageRatingStars.

We also will add a method to cast the Book's description field to HTMLText so that Line Break elements are rendered correctly. We will be able to use $DescriptionHTML in the template and this will in turn render the Book's description with their HTML elements.

app\src\Model\Book.php

<?php

namespace App\Model;

use App\Admin\ReviewAdmin;
use SilverStripe\ORM\DataObject;
use SilverStripe\Security\Permission;

class Book extends DataObject
{
    // ...

    private static $casting = [
        'DescriptionHTML' => 'HTMLText'
    ];

    public function DescriptionHTML()
    {
        return $this->Description;
    }

    public function getAverageRating()
    {
        $reviews = $this->Reviews();
        $total = 0;
        foreach ($reviews as $review) {
            $total += $review->Rating;
        }
        return $total / $reviews->count();
    }

    public function getAverageRatingStars()
    {
        $rating = $this->getAverageRating();
        $stars = '';
        for ($i = 1; $i <= $rating; $i++) {
            $stars .= '⭐';
        }
        return $stars;
    }

}

Improve the Homepage layout

Let's add the search form to the layout. We will see how we access from the controller:

themes\simple\templates\App\Page\Layout\Homepage.ss


<section class="container">
    <h1>The Home of Book Reviewers</h1>
    <div class="Actions line">
        $SearchBookForm
    </div>

    <!-- This div was as the layout originally was -->
    <div class="line">
        <br>
        <h2>Latest Reviews</h2>
        <% loop $LatestReviews %>
            <div>
                <h3>$Book.Title</h3>
                <p>
                    <b>$Title</b> <br>
                    $Review.FirstParagraph <br>
                    $Up.RatingStars($Rating)
                </p>
            </div>
        <% end_loop %>
    </div>
</section>

Create a BookController

The BookController will handle searches for books as well as specific views for each book. On the Book View the reviews associated with the book will be shown. We will see that our index action will use a BookHolder layout, and the view action will use a BookPage layout. We build out these layouts as well.

app\src\Controller\BookController.php

<?php 

namespace App\Controller;

use App\Model\Book;
use SilverStripe\CMS\Controllers\ContentController;
use SilverStripe\ORM\PaginatedList;

class BookController extends ContentController
{
    private static $allowed_actions = [
        'view' // This is the action that will be called when a user clicks on a book title
    ];

    // The index action will be called when the user visits the root of the BookController at '/book'.
    // It displays a paginated book results lists for the current query.
    public function index()
    {
        // Get the current query from the request
        $search = $this->request->getVar("q") ?? '';

        // If you like to specify a sort order, we can do it like this.
        // We won't work more on this feature for this context, but it's something to improve upon.
        $sort = $this->request->getVar("sort");
        $fields = DataObject::getSchema()->fieldSpecs(Book::class);
        if (!key_exists($sort, $fields)) {
            $sort = 'Title';
        } 

        if ($search) {
            $books = Book::get()

                // Only show books that have at least one review
                ->filter([
                    'Reviews.Count():GreaterThan' => 0  
                ])

                // A rudimentary search for title or author.
                ->filterAny([
                    'Title:PartialMatch' => $search,
                    'Authors.Name:PartialMatch' => $search,
                ]);
        } else {
            $books = Book::get()->sort($sort);
        }

        return $this->customise([
            'Layout' => $this
                        ->customise([
                            'Books' => (new PaginatedList($books, $this->getRequest()))
                                        ->setPageLength(5),
                            'Query' => $search,
                        ])
                        ->renderWith('Layout/BookHolder'),

        ])->renderWith(['Page']);
    }

    // This action will be called when a user clicks on a book title and will display the book and its reviews (with a paginated list).
    public function view()
    {
        $book = Book::get()->filter([
            'VolumeID' => $this->request->param('ID')  // We get the ID from the URI (instead of as a query parameter)
        ])->first();


        return $this->customise([
            'Book' => $book,
            'Layout' => $this
                        ->customise([
                            'Book' => $book,
                            'Reviews' => (new PaginatedList($book->Reviews(), $this->getRequest()))
                                        ->setPageLength(5),
                        ])
                        ->renderWith('Layout/BookPage'),
        ])->renderWith(['Page']);
    }
}

themes\simple\templates\Layout\BookHolder.ss

<section class="container">
    <h1 class="text-center">Books</h1>

    <% include SearchBar %>

    <div id="Content" class="searchResults">

        <% if $Books %>
            <% if $Query %>
                <p class="searchQuery">Results for "$Query"</p>
            <% end_if %>
            <ul id="SearchResults">

                <% loop $Books %>
                    <li>
                        <h4>
                            <a class="reviewsLink" href="/book/view/{$VolumeID}" title="&quot;{$title}&quot;">$Title</a>
                        </h4>
                        <div>
                            <p>
                                by 
                                <% loop $Authors %>
                                    $Name<% if not $IsLast %><% if $FromEnd == 2 %> and <% else %>, <% end_if %><% end_if %>
                                <% end_loop %>
                            </p>
                            <p>$AverageRatingStars ($Reviews.Count)</p>
                        </div>                 
                    </li>
                <% end_loop %>

            </ul>
            <% if $Books.MoreThanOnePage %>
                <% if $Books.NotFirstPage %>
                    <a class="prev" href="$Books.PrevLink">Prev</a>
                <% end_if %>
                <% loop $Books.PaginationSummary %>
                    <% if $CurrentBool %>
                        $PageNum
                    <% else %>
                        <% if $Link %>
                            <a href="$Link">$PageNum</a>
                        <% else %>
                            ...
                        <% end_if %>
                    <% end_if %>
                <% end_loop %>
                <% if $Books.NotLastPage %>
                    <a class="next" href="$Books.NextLink">Next</a>
                <% end_if %>
            <% end_if %>
        <% end_if %>
    </div>
</section>

themes\simple\templates\Layout\BookPage.ss

<section class="container">
    <h1 class="text-center">$Book.Title</h1>
    <div id="content">
        <div>$Book.AverageRatingStars</div>
        <br>
        <p>
            by 
            <% loop $Book.Authors %>
                $Name<% if not $IsLast %><% if $FromEnd == 2 %> and <% else %>, <% end_if %><% end_if %>
            <% end_loop %>
        </p>
        <p>
            $Book.DescriptionHTML
        </p>
        <br>
    </div>
    <div id="reviews" class="searchResults">

        <% if $Reviews %>
            <p class="searchQuery">Current Reviews</p>
            <ul id="SearchResults">

                <% loop $Reviews %>
                    <li>
                        <h4>$Title</h4>
                        <div>
                            <p>by $Member.FirstName</p>
                            <p>$RatingStars</p>
                        </div>                 
                        <p>$Review.FirstParagraph</p>
                    </li>
                <% end_loop %>

            </ul>
            <% if $Reviews.MoreThanOnePage %>
                <% if $Reviews.NotFirstPage %>
                    <a class="prev" href="$Reviews.PrevLink">Prev</a>
                <% end_if %>
                <% loop $Reviews.PaginationSummary %>
                    <% if $CurrentBool %>
                        $PageNum
                    <% else %>
                        <% if $Link %>
                            <a href="$Link">$PageNum</a>
                        <% else %>
                            ...
                        <% end_if %>
                    <% end_if %>
                <% end_loop %>
                <% if $Reviews.NotLastPage %>
                    <a class="next" href="$Reviews.NextLink">Next</a>
                <% end_if %>
            <% end_if %>
        <% end_if %>
    </div>
</section>

And before we test this out we want to add a route to our new controller, and add a navigation link for it.

app_config\routes.yml

We add a route for 'book' with an associated controller. Remember to run ?flush in the browser after updating routes.

---
Name: approutes
After: framework/_config/routes#coreroutes
---
SilverStripe\Control\Director:
  rules:
    'review': 'App\Controller\ReviewController'
    'book': 'App\Controller\BookController'

themes\simple\templates\Includes\Navigation.ss

<nav class="primary">
    <span class="nav-open-button">²</span>
    <ul>
        <% loop $Menu(1) %>
            <% if $MenuTitle.XML == "Registration" %>
                <% if not $CurrentMember %>
                    <li class="$LinkingMode"><a href="$Link" title="$Title.XML">$MenuTitle.XML</a></li>
                <% end_if %>
            <% else %>
                <li class="$LinkingMode"><a href="$Link" title="$Title.XML">$MenuTitle.XML</a></li>
            <% end_if %>
        <% end_loop %>

        <%-- Add our new 'Book' link here --%>
        <li class="<% if $URLSegment == "App\Controller\BookController" %>current<% end_if %>"><a href="/book" title="Books">Books</a></li>


        <% if $CurrentMember %>
            <li class="<% if $URLSegment == "App\Controller\ReviewController" %>current<% end_if %>"><a href="/review" title="Review">Review</a></li>
            <li><a href="$AdminURL" title="Admin">Admin</a></li>
            <li><a href="/Security/logout?BackURL=/" title="Logout">Logout</a></li> 
        <% else %>
            <li><a href="/Security/login?BackURL=/" title="Login">Login</a></li>    
        <% end_if %>

    </ul>
</nav>

Tidying it up

Our navigation bar is a bit messy, so we'll tidy it up. We need to log in to the admin area and remove the pages that we are not using. We are not using About Us or Contact Us so we'll remove them.

In the Pages section, click on the Batch Actions checkbox. Then we check the boxes for the two pages About Us and Contact Us. In the dropdown menu, select Unpublish and archive. We can now click on Go. This will remove the pages from the site and from our navigation bar.

While we are tidying up, let's also give our website a title. Click on the Settings section and fill out the form.
Site title: Book Reviewers
Site Tagline: The best place to find books
Then let's click on Save.

All done

HEY! Look how far we have come. This is amazing 🤩

Query me some Google Books API

Anders Björkland — Thu, 06 Jan 2022 08:25:04 +0000

Today we continue to build on our SilverStripe project, the book review platform. We will take a first step away from having to write book details manually when we want to review something. We will use the Google Books API to search for books and list the results in a template. Here's what we will cover today:

Using the Symfony HTTPClient to query the Google Books API
Build a controller and attach a route to it
Build a service class to parse a response from the Google Books API
Pass values from the controller to a template
Using a layout template within a Page-template

While this will take us far we will leave how to actually create new database entries from the API-response for the next entry in this series.

The complete code for this article is available in the GitHub repository devto-book-reviewers:googlebooks

Requests with Symfony HTTPClient

As we are going to consume a RESTful API, we will use a neat library called HTTP Client. It makes it a straight forward task to make HTTP requests and parse the response. We install it by running the following command:



composer require symfony/http-client

Drafting a simple controller

A simple use case for this library is having it run inside a controller and make a GET request to an API:



<?php

namespace App\Controller;

use SilverStripe\CMS\Controllers\ContentController;
use Symfony\Component\HttpClient\HttpClient;

class ReviewController extends ContentController
{
    private static $allowed_actions = [
        'index'
    ];

    public function index()
    {
        $client = HttpClient::create();
        $response = $client->request('GET', 'https://www.googleapis.com/books/v1/volumes?q=isbn:0747532699');
        $data = $response->toArray();

        return $data['items'][0]['volumeInfo']['title'];  // "Harry Potter and the Philosopher's Stone"
    }
}

Connect it with a route

For this controller to be run we need to connect it to a route. We have two options at hand:

We create a ReviewPage that we connect to this controller via getControllerName(), then we instantiate a new page of this type in the admin interface and type the route in the URL field. (This is effectively how we created our Registration page in a previous article.)
We add a route in a config-file and connect it to this controller.

We are going with Option 2 for this project. And here's how we will do it:

Create a new file in ./app/_config/ called routes.yml and add the following:



---
Name: approutes
After: framework/_config/routes#coreroutes
---
SilverStripe\Control\Director:
  rules:
    'review': 'App\Controller\ReviewController'

When we visit /review, we will see Harry Potter and the Philosopher's Stone in the browser. That's great! We know that HTTPClient works and that the controller for this route is working.

Next up, let's see how we can get different outputs when we add a query parameter to the URL (e.g. /review?q=Harry%20Potter). We will need to add some bells and whistles to our controller for this to work. Thanks to SilverStripe, it will not be too hard to do this. We add a new argument to our index() method of a HTTPRequest type and access query parameters on it. If there is a valid query parameter we attach it to our API request.



    //...
    public function index(HTTPRequest $request)
    {
        $q = $request->getVar('q');

        if ($q) {
            $client = HttpClient::create();
            $response = $client->request('GET', 'https://www.googleapis.com/books/v1/volumes?q='.$q);
            $data = $response->toArray();

            return $data['items'][0]['volumeInfo']['title'];
        } 

        return "Sorry, no valid query parameter found.";
    }
    //...

In a similar vein we can add a language parameter to our request so we can limit results to a specific language.



    public function index(HTTPRequest $request)
    {
        $q = $request->getVar('q');
        $langRestriction = $request->getVar('lang') ? "&langRestrict=" . $request->getVar('lang') : "";

        if ($q) {
            $client = HttpClient::create();
            $response = $client->request('GET', 'https://www.googleapis.com/books/v1/volumes?q='. $q . $langRestriction);
            $data = $response->toArray();

            return $data['items'][0]['volumeInfo']['title'];
        } 

        return "Sorry, no valid query parameter found.";
    }

As we continue to interact more and more with the Google Books API it may be worth to create a service class that can parse each item in the response. This will be useful for when we start to list the search results, and when we want to start to add a review for a specific book. Let's build that service now.

A service in your project

A service is a class that is used to perform a specific task. In this case, we are going to create a service that will parse the response from the Google Books API. We will create a new folder in ./app/src called Service. Inside this folder we will create a new file called GoogleBookParser.php:

./app/src/Service/GoogleBookParser.php



<?php 

namespace App\Service;

use SilverStripe\ORM\ArrayList;

class GoogleBookParser
{
    public static function parse(array $item): array
    {
        $authors = $item['volumeInfo']['authors'] ?? [];

        return [
            'title' => $item['volumeInfo']['title'] ?? '',
            'isbn' => $item['volumeInfo']['industryIdentifiers'][0]['identifier'] ?? '',
            'volumeId' => $item['id'] ?? '',
            'publishedDate' => $item['volumeInfo']['publishedDate'] ?? '',
            'authors' => ($authors ? ArrayList::create(
                array_map(function ($author) {
                    return ['AuthorName' => $author ?? ''];
                }, $item['volumeInfo']['authors'])
            ) : ''),
            'language' => $item['volumeInfo']['language'] ?? '',
            'image' => $item['volumeInfo']['imageLinks']['thumbnail'] ?? '',
            'pageCount' => $item['volumeInfo']['pageCount'] ?? '',
            'categories' => ArrayList::create(
                array_map(function ($category) {
                    return ['CategoryName' => $category ?? ''];
                }, $item['volumeInfo']['categories'] ?? [])
            ),
            'description' => $item['volumeInfo']['description'] ?? '',
        ];
    }

    public static function parseAll(array $response): array
    {
        return array_map(function ($item) {
            return self::parse($item);
        }, $response['items'] ?? []);
    }
}

This class can parse a single item or an array of items. We will use this service in our controller to parse the response from the Google Books API. Here's an example of how:




// Include the service
use App\Service\GoogleBookParser;

// ...
// inside the index-method:
$responseContent = $response->toArray();
$books = GoogleBookParser::parseAll($responseContent);
$book = $books[0];
return $book['title'];
// ...

Passing values from a controller to a template

We are soon going to build a template layout. But before we get there we are going to set our selves up for success by making the book results available to it. We do that by passing values from the controller to the template. While we are at it, we can tell the controller which template we would like to use.

The following controller will take the search-parameters and make a request to the Google Books API. It will then parse the response and pass the results to the template. It will also make a request to small API-endpoint to get a list of languages which can be used to restrict the search. Upon successful response the controller will also pass along an array consisting of pagination information.

./app/src/Controller/ReviewController



<?php

namespace App\Controller;

use App\Service\GoogleBookParser;
use SilverStripe\CMS\Controllers\ContentController;
use SilverStripe\Control\HTTPRequest;
use SilverStripe\ORM\ArrayList;
use Symfony\Component\HttpClient\HttpClient;

class ReviewController extends ContentController
{
    private static $allowed_actions = [
        'index',
    ];

    public function index(HTTPRequest $request)
    {
        $search = $request->getVar('q');
        $searchQuery = "q=" . $search;

        $startIndex = $request->getVar("startIndex") ?? 0;

        $langRestriction = $request->getVar("langRestrict") ?? 'any';
        $langRestrictionQuery = $langRestriction ? "&langRestrict=" . $langRestriction : "";

        $maxResults = $request->getVar('maxResults') ?? 10;
        $maxResultsQuery = '&maxResults=' . $maxResults;

        // Get language codes
        $client = HttpClient::create();
        $response = $client->request('GET', 'https://gist.githubusercontent.com/jrnk/8eb57b065ea0b098d571/raw/936a6f652ebddbe19b1d100a60eedea3652ccca6/ISO-639-1-language.json');
        $languageCodes = [["code" => "any", "name" => "Any"]];
        array_push($languageCodes, ...$response->toArray());


        $books = [];
        $pagination = [];

        if ($search) {
            $basicQuery = $searchQuery 
                        . $langRestrictionQuery
                        . $maxResultsQuery;

            $response = $client->request('GET', 'https://www.googleapis.com/books/v1/volumes?'. $basicQuery);
            $responseContent = $response->toArray();
            $books = GoogleBookParser::parseAll($responseContent);

            $pagination = $this->paginator('/review?' . $basicQuery, $responseContent['totalItems'], $startIndex, $maxResults);
            $pagination['pages'] = ArrayList::create($pagination['pages']);
            $pagination = ArrayList::create([$pagination]);
        } 

        return $this->customise([
            'Layout' => $this
                        ->customise([
                            'Books' => ArrayList::create($books),
                            'Pagination' => $pagination,
                            'Query' => $search,
                            'Languages' => ArrayList::create($languageCodes),
                            'LangRestriction' => $langRestriction
                        ])
                        ->renderWith('Layout/Books'),

        ])->renderWith(['Page']);
    }

    /**
     * Returns an array with links to pages with the necessary query parameters 
     */
    protected function paginator($query, $count, $startIndex, $perPage): array
    {
        $pagination = [
            'start' => false,
            'current' => false,
            'previous' => false,
            'next' => false,
            'totalPages' => 0,
            'pages' => false,
        ];

        $totalPages = ceil($count / $perPage);

        $currentPage = ceil($startIndex / $perPage) + 1;

        $previousIndex = $startIndex - $perPage;
        if ($previousIndex < 0) {
            $previousIndex = false;
        }

        $nextIndex = $perPage * ($currentPage);
        if ($nextIndex > $count) {
            $nextIndex = false;
        }

        $pagination['start'] = [
            'page' => $previousIndex > 0 ? 1 : false,
            'link' => $previousIndex > 0 ? $query . '&startIndex=0' : false,
        ];

        $pagination['current'] = [
            'page' => $currentPage,
            'link' => $query . '&startIndex=' . $startIndex
        ];
        $pagination['previous'] = [
            'page' => $previousIndex !== false ? $currentPage - 1 : false,
            'link' => $previousIndex !== false ? $query . '&startIndex=' . $previousIndex : false,
        ];
        $pagination['next'] = [
            'page' => $nextIndex ? $currentPage + 1 : false,
            'link' => $nextIndex ? $query . '&startIndex=' . $nextIndex : false,
        ];

        $totalPages = ceil($count / $perPage);  
        $pagination['totalPages'] = $totalPages;
        $pages = [];

        for ($i = 0; $i < 3; $i++) {
            $page = $currentPage + $i - 1;

            if ($currentPage == 1) {
                $page = $currentPage + $i;
            }

            if ($page > $totalPages) {
                break;
            }
            if ($page < 1) {
                continue;
            }

            $pages[] = [
                'page' => $page,
                'link' => $query . '&startIndex=' . ($page - 1) * $perPage,
                'currentPage' => $page == $currentPage
            ];
            $pagination['pages'] = $pages;
        } 

        return $pagination;
    }
}

As you may have noted, we are using nested templates in the return statement of the index method. This is because we want to use the same general structure as the rest of the site (which the Page template stands for), and we want finer control over the layout for this particular page (which the Layout/Books template stands for). Within the ->customise method we are passing an array of variables to the template. Each key in this array can be accessed in the template by using the $ prefix. Let's go ahead and build that template now.

Building a layout for our book search

We are going to build a layout for our book search page. We will use the Layout/Books template, which will be included within the Page template. If we look at the Page template, as it is presented in the simple theme, we have the following structure:



<!-- ... -->
<div class="main" role="main">
    <div class="inner typography line">
        $Layout
    </div>
</div>
<!-- ... -->

We are going to build a template that will be injected where $Layout is! Here's how we will do that. Create a new template in the templates/Layout directory called Books.ss. Give it the following structure:



<section class="container">
    <h1 class="text-center">Review a book</h1>

    <% include SearchBar %>

    <div id="Content" class="searchResults">

        <% if $Books %>
            <p class="searchQuery">Results for "$Query"</p>
            <ul id="SearchResults">

                <% loop $Books %>
                    <li>
                        <h4>
                            $title
                        </h4>
                        <div>
                            <% loop $authors %>
                                <p>$AuthorName</p>
                            <% end_loop %>
                        </div>

                        <a class="reviewLink" href="/review/book/{$volumeId}" title="Review &quot;{$title}&quot;">Review &quot;{$title}&quot;...</a>
                    </li>
                <% end_loop %>

            </ul>
            <div id="PageNumbers">
                <div class="pagination">

                    <% loop $Pagination %>
                        <span>

                            <% if $start.link %>
                                <a class="go-to-page" href="$start.link">|<</a>
                            <% end_if %>

                            <% if $previous.link %>
                                <a class="go-to-page" href="$previous.link"><</a>
                            <% end_if %>

                            <% loop $pages %>
                                <% if $currentPage %>
                                    <strong><a class="go-to-page" href="$link">$page</a></strong>
                                <% else %>
                                    <a class="go-to-page" href="$link">$page</a>
                                <% end_if %>
                            <% end_loop %>

                            <% if $next.link %>
                                <a class="go-to-page" href="$next.link">></a>
                            <% end_if %>

                        </span>
                    <% end_loop %>

                </div>
            </div>
        <% end_if %>
    </div>
</section>

This template has an include statement that includes the SearchBar template. This is a partial template that we will create in the templates/Includes directory. Call it SearchBar.ss.



<form class="Actions">
    <div class="line">
        <div class="field">
            <label for="search-input">Search</label>
            <input id="search-input" type="text" name="q" class="text" value="$Query">
        </div>
        <% if $Languages %>
        <div class="field">
            <label for="langRestrict">Language</label>
            <select name="langRestrict" id="langRestrict">

                <% loop $Languages %>
                <option value=$code <% if $Up.LangRestriction == $code %>selected<% end_if %>>$name</option>
                <% end_loop %>

            </select>
        </div>
        <% end_if %>
    </div>
    <div class="line">
        <div class="field">
            <input type="submit" class="btn" value="Search" />
        </div>
    </div>
</form>

We can see the use of some of the variables we passed to the template from the controller. Something is new here however! Why would we need a $Up variable? This is part of the SilverStripe template syntax to access variables that are one step above the current scope. Whenever we are within a loop-structure we are also one step removed from the variables that were passed to the template. LangRestriction is one of the variables that we passed to the template, and Up.LangRestriction is the variable that we can access from within the loop.

While we are on about loops we should also address how we have access to $code and $name variables. These are keys on elements of each of the $Languages array. In the controller we had $languageCodes = [["code" => "any", "name" => "Any"]];. So the first element of the $Languages array would have $code set to any and $name set to Any. It's the same logic that goes into explaining the variable uses in the Books.ss template.

If everything is working correctly, we should now have a working layout for our book search page. We can visit the /review URL and see the results.

Adjusting the CMS interface in Silverstripe

Anders Björkland — Thu, 30 Dec 2021 16:22:40 +0000

Today we are learning more about the CMS called Silverstripe. I've written about it before when we built the first components of a review site. In a follow up to this article we will see how we can get around having to type out all the book information manually, but before we can do that we need to learn how we can expand the functionality of the CMS interface. So here are a few simple tricks at hand given a standard Silverstripe installation (v4)

Add a link to the admin page

Silverstripe provides us with a template variable called $AdminURL which we can use to create a link to the admin page. So how we could add a link to the admin page with the simple theme (which is the default theme for Silverstripe) would look like this:

.\themes\simple\templates\Includes\Navigation.ss

<nav class="primary">
    <span class="nav-open-button">²</span>
    <ul>
        <% loop $Menu(1) %>
            <li class="$LinkingMode"><a href="$Link" title="$Title.XML">$MenuTitle.XML</a></li>
        <% end_loop %>
        <li><a href="$AdminURL" title="Admin">Admin</a></li>
    </ul>
</nav>

Add a link in the CMS navigation menu

We can adjust the vertical CMS navigation menu which is located at the left in the admin interface. This menu usually contains navigation to administrate pages, campaigns, files, etc. This menu is called LeftAndMain MenuList. Its appearance and content is rendered with the template vendor\silverstripe\admin\templates\SilverStripe\Admin\Includes\LeftAndMain_MenuList.ss. We can inject our own template by populating same "namespace". By doing this, SilverStripe will prioritize our template - be it just a partial template. Here is how we do that:

Copy the template LeftAndMain_MenuList.ss
Create these nested folders in the .\app folder: templates\SilverStripe\Admin\Includes\
Paste the template into the templates\SilverStripe\Admin\Includes\ folder.

As you may have recognized, we now have same folder structure in the app\templates folder as we found in the vendor\silverstripe\admin\templates\. This is all that it takes to inject our own template into the CMS interface. Currently though, we would not see any difference as we just did a copy and paste. So what we will do next is modify our template.

<ul class="cms-menu__list">
    <% loop $MainMenu %>
        <li class="$LinkingMode $FirstLast <% if $LinkingMode == 'link' %><% else %>opened<% end_if %>" id="Menu-$Code" title="$Title.ATT">
            <a href="$Link" $AttributesHTML>
                <% if $IconClass %>
                    <span class="menu__icon $IconClass"></span>
                <% else_if $HasCSSIcon %>
                    <span class="menu__icon icon icon-16 icon-{$Icon}">&nbsp;</span>
                <% else %>
                    <span class="menu__icon font-icon-circle-star">&nbsp;</span>
                <% end_if %>
                <span class="text">$Title</span>
            </a>
        </li>
    <% end_loop %>

    <%-- Let's add a link to DEV here --%>
    <li>
        <a href="https://dev.to">
            <span class="menu__icon font-icon-circle-star">&nbsp;</span>
            <span class="text">DEV</span>
        </a>
    </li>
</ul>

Having done this, we now have an excellent little link to go straight here to DEV and read an article about web development 😉