DEV Community

Cover image for [Failed] Investigating Machine Learning Techniques to Improve Spec Tests — V
Emirhan Akdeniz for SerpApi

Posted on • Updated on

[Failed] Investigating Machine Learning Techniques to Improve Spec Tests — V


This is a part of the series of blog posts related to Artificial Intelligence Implementation. If you are interested in the background of the story or how it goes:

This week we’ll showcase storing and reusing trained weights, and the talk about the crucial mistake I did in calculating the early results of the model. We will be using SerpApi’s Google Organic Results Scraper API for the data collection. Also, you can check in the playground in more detailed view on the data we will use.

Crucial Mistake I made in Calculating Model Accuracy

Let me first give the way in which I calculated using training examples:

    true_examples = {|el| el = el.first == "1" ? el.second : nil}.compact
    false_examples = {|el| el = el.first == "0" ? el.second : nil}.compact

    predictions = []

    false_examples.each do |example|
      prediction = test example, 2, vector_array, key_array
      predictions << prediction
    end! {|el| el = el == 1 ? 0 : 1}

    true_examples.each_with_index do |example, index|
      puts "--------------"
      prediction = test example, 2, vector_array, key_array
      predictions << prediction
      puts "Progress #{(index.to_f/true_examples.size.to_f).to_f}"
      puts "--------------"

    prediction_train_accuracy = predictions.sum.to_f / predictions.size.to_f

    puts "Prediction Accuracy for Training Set is: #{prediction_train_accuracy}"
Enter fullscreen mode Exit fullscreen mode

We take the examples that are snippet, and ones that are not snippet, and run them in the trained model. If the prediction of not snippet examples turned out to be 0, they would be counted as correct predictions, and if the snippet results were counted as 1 they would also be counted as correct results. In the end, the result of the rate of correct prediction was 0.8187793427230047. I later used a larger dataset and the result was around 89%.

Here’s the logical fallacy I have fallen; There were exhaustively more non-snippet results as opposed to snippet results. Let's suppose the rate of which be 1:9 in a 10 unit example set. If the model is prone to call things non-snippet more than snippet in a random manner, as it was the case, then non-snippet results will be predicted correctly, which will create a bias in results.

The reason why it increased to 89 percent in the larger dataset, was because there were more type of keys in it. It resulted in snippet size / non-snippet size to minimize further, which triggered a bias in false positive predictions of non-snippet results.

In reality, I should’ve tested the model using only snippet examples. I did, and found out that the model is useless. I will try to tweak and find a way to make it useful. But, I think it is better to show why I made a failed calculation before that to inform everybody, and prevent other people from making the same mistake.

Storing Trained Weights in a Custom Way

It took me a while before I realized my mistake. Before that, I have created a way to store and predict using the model.

Here’s the full code for it:

    class Predict 
      def initialize csv_path, trained_weights_path, vocab_path, object = "Snippet", k = 2
        @@csv_path = csv_path
        @@trained_weights_path = trained_weights_path
        @@vocab_path = vocab_path
        @@object = object
        @@k = k
        @@key_arr = []
        @@vector_arr = []
        @@weights = []
        @@maximum_word_size = 0
        @@vocab = {}

      def self.construct
        @@weights = initialize_trained_weights @@trained_weights_path
        @@vocab = read_vocab @@vocab_path
        @@key_arr = read_csv @@csv_path
        @@vector_arr = define_training_set @@key_arr
        @@maximum_word_size = @@weights.size

      def self.read_csv csv_path

      def self.read_vocab vocab_path
        vocab = vocab_path

      def self.initialize_trained_weights trained_weights_path
        weights = trained_weights_path
        weights = JSON.parse(weights)

      def self.define_training_set vectors { |word| word_to_tensor word[1] }

      def self.default_dictionary_hash
          /\"/ => "",
          /\'/ => " \'  ",
          /\./ => " . ",
          /,/ => ", ",
          /\!/ => " ! ",
          /\?/ => " ? ",
          /\;/ => " ",
          /\:/ => " ",
          /\(/ => " ( ",
          /\)/ => " ) ",
          /\// => " / ",
          /\s+/ => " ",
          /<br \/>/ => " , ",
          /http/ => "http",
          /https/ => " https ",

      def self.tokenizer word, dictionary_hash = default_dictionary_hash
        word = word.downcase
        dictionary_hash.keys.each do |key|
          word.sub!(key, dictionary_hash[key])

      def self.word_to_tensor word
        token_list = tokenizer word {|token| @@vocab[token]}

      def self.extend_vector vector
        vector_arr = vector.to_a
        (@@maximum_word_size - vector.size).times { vector_arr << 1 }

      def self.extend_vectors
        @@vector_arr.each_with_index do |vector, index|
          @@vector_arr[index] = extend_vector vector

      def self.product vector
        @@weights.each_with_index do |weight, index|
          vector[index] = weight * vector[index]


      def self.euclidean_distance vector_1, vector_2
        subtractions = (vector_1 - vector_2).to_a! {|sub| sub = sub*sub }

      def self.execute example
        example_vector = word_to_tensor example! {|el| el = el.nil? ? 0: el}
        example_vector = extend_vector example_vector
        weighted_example = product example_vector

        distances = []
        @@vector_arr.each_with_index do |comparison_vector, vector_index|
          distances << euclidean_distance(comparison_vector, weighted_example)

        indexes = []
        @@k.times do 
          index = distances.index(distances.min)
          indexes << index
          distances[index] = 1000000000

        predictions = []
        indexes.each do |index|
          predictions << @@key_arr[index].first.to_i

        puts "Predictions: #{predictions}"

        prediction = (predictions.sum/predictions.size).to_f
        if prediction < 0.5
          puts "False - Item is not #{@@object}"
          return 0
          puts "True - Item is #{@@object}"
          return 1

    csv_path = "organic_results/organic_results__snippet.csv"
    trained_weights_path = "organic_results/snippet_weights.json"
    vocab_path = "organic_results/vocab.json" csv_path, trained_weights_path, vocab_path, object = "Snippet", k = 5

    true_examples =
    true_examples = {|el| el = el.first == "1" ? el.second : nil}.compact

    true_examples.each_with_index do |example, index|
      puts "--------"
      puts "#{index}"
      Predict.execute example
      puts "--------"
Enter fullscreen mode Exit fullscreen mode


I made an honest attempt into an area that could relieve some pressure on writing of the tests, and I have failed. A subtle mistakes on custom codes may occur and result in overall failure. I am grateful I gave a shot to it, and saw that this approach was not as effective as I thought it was. I will further work on this in the future. However, for next blog post, the topic will most likely be a different one. I would like to apologize to the reader for giving misleading results in the previous blog post, and thank them for their attention.

Originally published at on May 4, 2022.

Top comments (0)