DEV Community: Eve S

1D Polynomial Curve fitting in Numpy and Matplotlib

Eve S — Fri, 29 Mar 2024 15:09:45 +0000

This tutorial is meant to provide a quick intro to a couple useful subjects: generating polynomial data, introducing noise to that data, creating a fit with the least squares method, and graphing the fit and data together with an R² value. First, we are going to make some numpy arrays representing x and y coordinates for a line:



import numpy as np
x = np.linspace(0, 50, num=50)
y = np.linspace(0, 30, num=50)

Here x is an array that has 50 evenly spaced points between 0 and 50, inclusive, and y has 50 evenly spaced points between 0 and 30. This all could just as easily be represented by one 2D array. Now we are going to introduce some variation to the values:



import random 
def noise(k):
    return k + (random.random()*2-1) * 1.5

x = np.vectorize(noise)(x)
y = np.vectorize(noise)(y)

np.vectorize evaluates our noise function for each value of x and y, using random.random() to generate a random number between 0 and 1 to offset the values a bit and make the points (x,y) arranged a bit less linearly. Changing 1.5 to a larger or smaller value will introduce more or less noise. Now we want to create a line to fit our data:



lin_fit = np.polyfit(x, y, 1)
lin_model = np.poly1d(lin_fit)
lin_R_squared = np.corrcoef(x, y)[0, 1]**2

np.polyfit uses the least squares method to create a line matching the points (x, y). If we wanted to match a higher order polynomial to the points, we'd just need a different value for the third parameter, the degree of the polynomial created.
Polyfit returns an array containing the line's coefficients in order from highest degree to lowest - this is import to remember when comparing to the function that has replaced polyfit, numpy.polynomial.polynomial.Polynomial.fit, and returns the coefficients in order from lowest degree to highest.
np.poly1d takes an array of polynomial coefficients and creates a handy function for us that turns x values into y values. It wouldn't be hard to use m, b = np.polyfit(x, y, 1) and later use m*x + b instead of lin_model(x), but poly1d gets more useful the more coefficients are needed.
np.corrcoef returns a correlation coefficient matrix, which isn't really worth getting into. The important thing is we are accessing it to get R², a value that will tell us how well our line fits.
Fitting linear data is great, but we might as well fit a parabola while we're at it. We can use the same x values, but we need some new y values:



yq = np.empty(shape=(50)) #random values
with np.nditer(yq, flags=['f_index'], op_flags=['readwrite']) as it:
    for yVal in it:
        yVal[...] = it.index ** 2

Numpy expects us to make an array of the right size and replace values instead of appending, which takes much longer. np.empty is the fastest way to make an array, here a 1D array with 50 values. It is full of random values, but we are going to replace them anyways.
The next lines are a standard way to iterate through an array and replace values. we need f_index in the args to access the index, and we need readwrite to change the values in yq. Each value in yq is now its index squared, which should look exactly like the values for y = x² from 0 to 50. Now to add noise and create a fit:



yq = np.vectorize(noise)(yq)

quad_fit = np.polyfit(x, yq, 2, full=True)
quad_model = np.poly1d(quad_fit[0])
mean_yq = yq.sum()/yq.size
quad_tss = np.sum((yq - mean_yq)**2)
quad_R_squared = 1 - quad_fit[1]/quad_tss
quad_R_squared = quad_R_squared[0]

Adding noise works the same, but we now creating a fit with degree 2 instead of 1, and setting full to True. This means polyfit will return a 2d array where the first row is our coefficients like before, but the next rows contain some information about how good our fit is. This is needed because coercoff only works for lines. quad_fit having additional lines is why we need to index it when invoking np.poly1d. The next lines are a way to get R²: we get the average value of yq by summing it then dividing by its number of elements; we get the TSS (total sum of squares) squaring the difference between each y element and the average y then summing them all up; last we get R² in the second to last line by accessing the RSS (residual sum of squares) from quad_fit. quad_R_squared is an array with one element so we just make it that element instead. Now we can use Matplotlib to look at our data and fits!



import matplotlib as plt

fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.set_title('Linear regression')
ax1.scatter(x, y)
ax1.plot(x, lin_model(x), '--k')
ax1.text(0.05, 0.95, 
         f'y = mx + b\nm = {lin_fit[0]}\nb = {lin_fit[1]}\nR\u00b2 = {lin_R_squared}', 
         verticalalignment='top', 
         transform = ax1.transAxes, 
         fontsize=14, 
         bbox={'facecolor':'cyan'})

ax2.set_title('Quadratic regression')
ax2.scatter(x, yq)
ax2.plot(x, quad_model(x), '--k')
ax2.text(0.05, 0.95, 
        f'y = ax\u00b2 + bx + c\na = {quad_fit[0][0]}\nb = {quad_fit[0][1]}\nc = {quad_fit[0][2]}\nR\u00b2 = {quad_R_squared}', 
        verticalalignment='top', 
        transform = ax2.transAxes, 
        fontsize=14, 
        bbox={'facecolor':'cyan'})

plt.show()

Here we are making one figure which contains our two graphs, called ax1 and ax2. The arguments of plt.subplots makes them arranged horizontally. Calling scatter on ax1 lets us plot the points with coordinates from our x and y arrays. ax1.plot(x, lin_model(x), '--k') has x as x coords and the values generated by our fit as y coords. The last argument makes our fit a black dotted line - we could have avoided using scatter for (x, y) by setting the style to have no connecting line at all. The arguments for the text function just control where our info is and what it looks like - we are using it to show the fits' coefficients and R² values on our graphs. Running the file lets plt.show show us our graphs like so:

Python Property() and Descriptors

Eve S — Fri, 08 Mar 2024 16:00:16 +0000

In Python I have been making frequent use of the property decorator:

class Foo:
    def __init__(self, a):
        self.a = a
    @property 
    def a(self):
        return self._a
    @name.setter 
    def a(self, a):
        self._a = a

A decorator (denoted by '@') is a fast and clean way to wrap a function in another function. The above code written without using a decorators is

class Foo:
    def __init__(self, a):
        self.a = a
    def getter(self):
        return self._a
    a = property(getter)
    def setter(self, a):
        self._a = a
    a = a.setter(setter)

This reveals a bit more about what is happening. We are assigning our new variable to the property function with our getter function as an argument, which returns a special type of object called a descriptor. This descriptor has a method setter() which we use to assign the setter function. We could just as easily define the getter and setter then say a = property(getter, setter) to do both in one line, and property can even take a third argument, a deleter. We can check a.fget is getter # True to see that our a's getter is the same in memory as the getter function we defined.

The __init__ method wasn't needed up to this point, but a
property isn't very useful without it. When we instantiate a new Foo object and set its a attribute, Python will use the getter method stored in the Foo class to assign the new object's 'a'. The same goes for reassigning 'a', since the setter will be called. Most commonly this has been useful to me for checking if a new value is acceptable before changing it, manipulating it (by changing type, for example) to fit if needed, and giving an error otherwise.

But how exactly does property work? The object property returns has these attributes fget and fset that point to getter and setter, but that alone isn't enough for it to work. Like any descriptor, it uses magic methods to hook into its basic functionality as an object. In the Python doc on descriptors, there is an example of a class that uses __get__ and __set__ to reproduce some of the functionality of property. I've cut it down some:

class LoggedAgeAccess:

    def __get__(self, obj, objtype=None):
        return obj._age

    def __set__(self, obj, value):
        obj._age = value

class Person:

    age = LoggedAgeAccess() # Descriptor instance

    def __init__(self, name, age):
        self.name = name # Regular instance attribute
        self.age = age # Calls __set__()

    def birthday(self):
        self.age += 1 # Calls both __get__() and __set__()

__get__ is the run when the descriptor instance is evaluated. Creating Person instantiates a LoggedAgeAccess object. Writing p = Person('name', 4); p.age, runs the LoggedAgeAccess instance's __get__ method, which passes the descriptor instance (self), then p, then Person. As __get__ is defined, it will return p.age. The "" marks a private instance which accesses the attribute value without using the descriptor, which if done would cause a loop.

__set__ works similarly and is run when the attribute is reassigned. Note the line self.age = age does not need to call __get__ because it does not need to evaluate the left hand side as an expression.

Defining magic methods in a class is an example of "operator overloading." Every class, even if you don't define it as a subclass, inherits from the "object" class and comes with these magic methods ready-made. They can handle what happens when you instantiate their objects, add them to each other with the plus sign, compare them with ==, and more. When you define one of these magic methods inside a class you are actually overwriting the previous definition of the method, and your new one will be referred to first as the case is with variable scope.

The code example above only works for a making a managed variable named age, but with one more magic function it can be made much more useful.

class LoggedAccess:

    def __set_name__(self, owner, name):
        self.public_name = name
        self.private_name = '_' + name

    def __get__(self, obj, objtype=None):
        return getattr(obj, self.private_name)

    def __set__(self, obj, value):
        setattr(obj, self.private_name, value)

The above class makes use of __set_name__, which checks the name of the variable the class instance is assigned to. getattr and setattr exist for every object and do what they say in the name - they too can be overwritten for a class with __getattr__ and __setattr__.

It is easy enough to added some data validation before setattr to determine if the reassignment should occur, and Python even has an inbuilt Validator class that makes this easier (custom calidatorscan be made as subclasses by being defined with TheClassName(Validator)).

The property decorator, along with the classmethod decorator and others, can handle all this work of hooking in with descriptors for us, but often we need to start using magic methods ourselves to change how our custom objects behave. __init__ is all over the place! This type of work can get close to the underlying C Python is written in. I'll include here the source code for property, a pure Python sample implementation of the property class from the official descriptor guide and a Stack Overflow post that explained it well. If you're surprised that Property is a class in this implementation, don't be! Functions are first class objects in Python. When you write def foo(), you are making 'foo' an object of the type "function". This Property class is similar to a function in that it takes arguments to create an instance which will will rely on its magic methods to be referenced, and can even be referenced just by being evaluated on it's own because of __get__.

class Property:
    "Emulate PyProperty_Type() in Objects/descrobject.c"

    def __init__(self, fget=None, fset=None, fdel=None, doc=None):
        self.fget = fget
        self.fset = fset
        self.fdel = fdel
        if doc is None and fget is not None:
            doc = fget.__doc__
        self.__doc__ = doc

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        if self.fget is None:
            raise AttributeError("unreadable attribute")
        return self.fget(obj)

    def __set__(self, obj, value):
        if self.fset is None:
            raise AttributeError("can't set attribute")
        self.fset(obj, value)

    def __delete__(self, obj):
        if self.fdel is None:
            raise AttributeError("can't delete attribute")
        self.fdel(obj)

    def getter(self, fget):
        return type(self)(fget, self.fset, self.fdel, self.__doc__)

    def setter(self, fset):
        return type(self)(self.fget, fset, self.fdel, self.__doc__)

    def deleter(self, fdel):
        return type(self)(self.fget, self.fset, fdel, self.__doc__)

Hiding an input inside a button

Eve S — Fri, 16 Feb 2024 15:21:24 +0000

I was working on the JavaScript for this website recently and I had a problem: I wanted to ask the user what file they wanted to use for their profile picture, but I didn't want to show the name of the file like the input tag by default does. The file the user submitted was going to be displayed on the page anyways, so I just wanted them to click a button and not be shown their filename.

An example from MDN Web Docs on the file input element before and after file selection

After looking at a thread on Stack Overflow, I decided to try wrapping the input in a label tag, as you often do with forms. This meant giving the input an id and giving the label a "for" attribute with a value of that id. Then I simply set the input display to "none" using styling - the Stack Overflow thread recommended setting opacity to 0 instead of using "display: none" or "visibility: hidden" since these latter two options apparently can make the element noninteractive, but I had no such problem. Because I was using React Bootstrap, I was able to style the label as a button and enter the text I wanted to display as its content, but I imagine putting a button tag in the label and wrapping that around the desired text should work too. You can see additional styling with bootstrap in the below image, but it isn't relevant.

Inspecting the label element on the page. The whole blue part is clickable and opens up file submission!

Selecting HTML Elements in JavaScript

Eve S — Fri, 26 Jan 2024 15:57:50 +0000

Over the past few weeks I've been getting started with web development, and it's already apparent to me that webpages wouldn't be half of what they are without the functionality JavaScript or another programming language provides. One of the main ways to allow a user to affect the DOM through the use of JS is through event listeners, namely the .addEventListener method, which primes an element (or document) to invoke some function in response to a specified event. With this method the user can create elements, remove them, change their attributes or content, the list goes on - but how does one find the element they want to call .addEventListener on?

.querySelector and .querySelectorAll

There are a lot of options! The first I'd like to talk about is the .querySelector method, which looks for an element using a string representing CSS selectors and returns the first match as an element object. More specifically, the element is returned as the appropriate subclass of the element object, always HTMLelement in my experience, but also potentially something like SVGelement or MathMLElement. This method is often called on the document node object (as in document.querySelector(selector)) to comb the whole page for the element but can also be called on an element to only pick from that element's children.

<div id=first>
    <div class=child></div>
</div>
<div id=second>
    <div class=child></div>
</div>

In this example, document.querySelector('div.child') will return the div inside the div of id "first", while document.querySelector(#second).querySelector(div) and document.querySelector(#second div) will both return the second div that has the class "child." The second could be useful in a case where you already have document.querySelector(#second) stored as a variable.

It's worth noting that the document and element objects are both subclasses of the node interface, and abstract class that is not used itself but provides a framework that many DOM API objects like document and element are based on. The node interface does not have the querySelector method or any of the other methods discussed below, but its subclasses often do.

The .querySelectorAll method is called on the same objects and with the same arguments as .querySelector, but it instead returns all of the matching elements as a NodeList. A NodeList is an array-like iterable that contains any sort of node objects, which for .querySelectorAll are elements, often HTMLelements. The method doesn't return an array because the DOM API tries to provide values that can be handled by any programming language, or be "language independent." Invoking document.querySelectorAll('div') for the above HTML would return an NodeList of length 4 containing all of the divs.

.getElementsBy

.querySelector, added in 2013, is newer than the following methods and allows the most specificity in one's search. The document object already had the .getElementById, .getElementsByClassName, .getElementsByName, and .getElementsByTagName methods. For all of these, the argument is a string representing the desired value for the attribute in the method name. Elements have all of these methods except the ones for ID and name, ID because there is (or should be) only one element with a specific ID per page so narrowing the search window to an element isn't needed, and name because that attribute is mainly used for matching elements inside a form tag and won't be repeated outside the tag. There is another method that document and element have, .getElementsByTagNameNS, but this deals with namespace, an XML feature not in HTML that I don't and won't bother to understand right now.

Which to use?

These methods explain more what they are doing, so if you only need to get elements with a class name instead of, say, getting them by both a class and tag name, they will make your code more readable. .getElementById is useful in this regard. There is another important difference, though: these methods return live objects while querySelectorAll returns a static one. That is to say, if assigns the returned objects of these methods to variables and then adds a new element that fits the search criteria, only one of the variables add the new element:

static = document.querySelectorAll('div.child');
live = document.getElementsByClassName('child');
console.log(static); //NodeList [ div.child, div.child ]
console.log(live); //HTMLCollection { 0: div.child, 1: div.child, length: 2 }
const newDiv = document.createElement('div');
newDiv.className = 'child';
document.body.append(newDiv);
console.log(static); // NodeList [ div.child, div.child ]
console.log(live); // HTMLCollection { 0: div.child, 1: div.child, 2: div.child, length: 3 }

If there were a change in one of the elements, like adding an id of "new" to the first div.child, that would update in both lists because they both contain references to the DOM elements. Notice .getElementsByClassName returns an HTMLCollection instead of a NodeList; an HTMLCollection is always live and only contains elements (instead of being able to contain any node). A NodeList is usually static, but can be live, as when .getElementsByName returns a live NodeList. NodeLists are also array-like, and need to be accessed by index, whereas HTMLCollections can be accessed by index/key, the id attribute with the namedItem property, and the name attribute with the namedItem property.

This is all to say that you should use one of the methods other than querySelector if you want to assign it to a variable that will update as new elements that meet the criteria are added. These methods also have the benefit of having more explicit names. But the way querySelector and querySelectorAll use CSS selectors allows for much more precise element selection without resorting to the .children property or others.

Works Cited

JavaScript and HTML DOM. JavaScript and HTML DOM Reference. https://www.w3schools.com/jsref/

MozDevNet. Web technology for developers: MDN. MDN Web Docs. https://developer.mozilla.org/en-US/docs/Web

Mortensen, P., & Standaert, M. (1957, September 1). HTML input - name vs. ID. Stack Overflow. https://stackoverflow.com/questions/7470268/html-input-name-vs-id/7470407#7470407

Sam, huwiler, & J, T. (1960, February 1). Javascript why use NodeList instead of using array. Stack Overflow. https://stackoverflow.com/questions/21768551/javascript-why-use-nodelist-instead-of-using-array

Semah, B. (2023, December 7). HTMLCOLLECTION vs nodelist – what’s the difference? freeCodeCamp.org. https://www.freecodecamp.org/news/dom-manipulation-htmlcollection-vs-nodelist/#differences-between-htmlcollection-and-nodelist