DEV Community: Ompeluseura LevelUP Koodarit

What more I've learned from the Advent of Code (days 6-8)

Minna N. — Sun, 13 Dec 2020 10:55:16 +0000

After a pretty good start, I got stuck on day 7 for a long time and haven't wanted to spend all my precious little free time with AoC. My day 7 recursion was looking pretty good but produced some false positives which I eventually weeded out manually and subtracted from the total. Part 2 is in shambles and I don't have a clear picture of how it should be solved. I heard people are figuring out the puzzles with pen&paper and I might try that, too.

But! I've managed to solve days 6 and 8 with two stars. 😅 Current total: 15🌟

Nifty datatype: Sets

One way to remove duplicate values from an array is to create a set out of it. Values in sets can only occur once. I used this feature on day 6 part 1: I collected all 'yes' answers from a group in an array first and then created the set:

let groupAsSet = [...new Set(groupArray)];

It was then easy to find the number of unique 'yes' answers from the group by set.length.

Destructuring vol.2

I enjoyed solving day 8 (at least part 1 😆). I started by separating the operation and argument.

let [operation, argument] = command.split(' ');

For my original solution, this time I used a regular expression with match method to also separate the sign and digits.

let [_, sign, number] = argument.match(/(\+|\-)(\d+)/);

Using the underscore is the influence of @caiangums. 😊 I saw this usage in his code: the first element in the array that match returns is the whole matched string for the regex between /.../, which I have no use for, and using an underscore nicely depicts that.

Next I used the sign and number to calculate changes in either accumulator or program position (index). I wanted to use the ternary operator here.

sign === '+' ? accumulator += Number.parseInt(number) : accumulator -= Number.parseInt(number);
(...)
sign === '+' ? index += Number.parseInt(number) : index -= Number.parseInt(number);

But... I ended up cleaning up my code and just converted the argument to number directly:

argument = Number.parseInt(argument);
(...)
accumulator += argument;
index += argument;

Much cleaner! Can't believe I didn't think of it right away. 🤦

Reminder: Arrays are reference types

On day 8, my solution was to for loop through the boot code changing one command at a time. First I didn't realize I never "reset" the array at the beginning so I ended up just changing it one line at a time. Then I understood I needed a temporary array to make the one change:

let modifiedCommandArray = commandArray;

The same thing happened again! Then it hit me: oh, right, arrays are reference types so I'm actually modifying the same array but just using a different name. Fixed the code by using the spread operator ....

let modifiedCommandArray = [...commandArray];

Reminder: Include `break` in your `switch`

I didn't make this mistake – this time – but I'm quite sure I could and then wonder what's going on. For switch-case structure, you usually want to end your case block with break or all the remaining code will be executed as well. Whoops!

I'm diggin' what my day 8 switch structure looks like:

switch(operation) {
  case 'acc':
    accumulator += argument;
    index++;
    break;
  case 'jmp':
    index += argument;
    break;
  case 'nop':
    index++;
    break;
}

That's all this time! Plodding on. 👢👢

Cover photo by Okwaeze Otusi on Unsplash

What I've learned from the Advent of Code so far (days 1-5)

Minna N. — Sun, 06 Dec 2020 12:22:20 +0000

I consider myself an advanced beginner in programming. I lack a lot of knowledge in best practices, gotchas, elegance... let alone algorithms, optimizations... I have no clue.

I'm quite sure I won't make it through Advent of Code but I wanted to give it a go anyway. I first thought to use Java to solve the puzzles because I'm more comfortable processing line-by-line input and doing "tricks" with it (I did a comprehensive course on it just at the beginning of this year), but decided on JavaScript because it's more beneficial for the things I'm learning at the moment.

On the first five days, I had a couple of facepalm moments 🤦 but also some proud moments 🏆.

Here are some of the things that have helped me on puzzle-solving days 1-5.

Neat feature: Destructuring

On day 2 I was quite proud of myself for remembering the destructuring assignment feature. The task is to process a list with the following data:

int-int char: string

For example:

1-3 a: abcde
1-3 b: cdefg
2-9 c: ccccccccc

One line contains a password policy and a password, so first I separated the policy from the password

const [policy, password] = line.split(': ');

Then I separated the numbers from the character in the policy:

const [amount, character] = policy.split(' ');

And finally the first number and the second number (representing min and max values in the first part of the puzzle and two positions in the second part):

const [min, max] = amount.split('-');

Very handy!

Neat method: Array.from()

For the colour code validation on day 4, I use indexOf(). First I had an array with the possible values like so:

let validChars = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'];

I got a tip to make it sooo much more elegant:

let validChars = Array.from('0123456789abcdef');

Coolio! 😎 As it sounds like, here Array.from() creates an array from the given string.

If you are wondering why I'm processing the numbers as strings, it's just so much simpler because the valid characters are either numbers or strings. And actually, the value comes as a string to validation so === works more reliably this way.

I'm really digging this array of valid values, too. First I had

if (value === 'amb' || 
    value === 'blu' || 
    value === 'brn' || 
    value === 'gry' || 
    value === 'grn' || 
    value === 'hzl' || 
    value === 'oth' ) { ... }

for the hair colour validation 😅 but I just changed it to

let validColors = ['amb', 'blu', 'brn', 'gry', 'grn', 'hzl', 'oth'];

if (validColors.indexOf(value) != -1) { ... }

Tip: Break up processing into functions

On day 4 you have to do data validation and I was puzzled by how I'd be able to end processing of an invalid value in a nested loop and get back to the outer loop to validate the next value. I got a tip – one that I should remember by now – that I should make more helper functions. There's no such thing as too many functions (within reason). 😄

My colour code validation was made much simpler with a helper function that returns either true or false.

function hexValidity(hexValue) {
  let validChars = Array.from('0123456789abcdef');
  let colourArray = Array.from(hexValue);

  if (colourArray[0] != '#' || colourArray.length != 7) {
    return false;
  }

  for (let i = 1; i < colourArray.length; i++) {
    let currentChar = colourArray[i];

    if (validChars.indexOf(currentChar) === -1) {
      return false;
    }
  }
  return true;
}

Tip: Create variables more often

The code is easier to read when you first assign results of functions, values from arrays, etc. in variables and use them in another structure. For example in my colour validation code for day 4, I first had:

if (validChars.indexOf(colourArray[i]) === -1) {
  return false;
}

Compare with

let currentChar = colourArray[i];

if (validChars.indexOf(currentChar) === -1) {
  return false;
}

Tip: Use modulo where you can

I keep forgetting how useful (and multi-use) modulo % is.

For my toboggan trajectory on day 3, I skip to the beginning of the line ("horizontal index" 0) when I go over the length of the array (31) so in my code, I subtract the length from the horizontal index if it's over 30 (last possible index). With modulo, I could just use index % 31 and be done with it.

If you have a situation where a value has to loop back to 0 at some point, use modulo.

Best practice: Early exit

It's best to start by validating your data so you can break out of a loop/function as early as possible. For example on day 4, it's wise to check if the passport ID even has the required 9 characters before you start validating if each of the characters is a digit. Same with the hex colour codes: if it doesn't have a hash # at the beginning and exactly 6 characters after it, there's no point validating it in more detail.

Take heed: Scope of variables

This was a moment of a huge facepalm. On day 4 you have to do data validation, which in itself is quite complicated to do for seven different value types.

After I'd extracted the value validation itself into a separate function, as mentioned above, I found myself facing an infinite loop. The code was able to process the first three values ok but then it got stuck looping with second and third value. A lot of debugging later, I was this much wiser: 💡 remember to always declare the initial variable of a for loop 💡 or the code may end up using a completely wrong variable.

I had forgotten the let from a couple of for loops where used i as the index counter. 🤦

This actually brings to mind another tip for myself: keep in mind the existence of for/of loop! I could've made my code a lot simpler with it.

A proud moment on day 3

First I was completely at a loss with the puzzle on day 3: how am I supposed to figure out a trajectory through lines of data? I don't know about vectors or any map algorithms.

I started visualizing the problem as a matrix, but then was unsure how that would be done in JavaScript (would've been easy-peasy in Java) but it got me a step further: I put the lines into an array (array item per line) for vertical movement and used charAt for the horizontal dimension of my "matrix". Once I had my function for the part 1 working and I was looking at part 2, I first thought "oh no, the function is going to be so messy with the for loop times 5". But then I realized, if I refactor my first function a bit I can reuse it by giving the numbers for traversing (x steps right, y steps down) as parameters and just assign the results to variables. 🏆

Cover photo by Markus Spiske on Unsplash

DIY Christmas Radio

Laura Vuorenoja — Sun, 29 Nov 2020 17:32:30 +0000

Many people who are starting their programming journey often wonder where to get the ideas for their side projects. My advice is that work with a topic you are passionate about. For example, if you love cooking, make yourself a cookbook service or perhaps a digital egg timer. Or if you like to wander in nature, code an app that tracks your routes or helps you to identify which bird is singing. You will have an extra motivational boost when solving the problem will benefit you also otherwise.

"The thing" for me has always been music, so most of my side projects have been related to music in a way or another. One of my favorite apps is Spotify and luckily for me, they have published excellent APIs that one can utilize versatilely in their projects.

So it is no surprise that my latest project is also built on Spotify API. As Christmas is getting closer, my daily listening queue is filled with Christmas tunes. However, I tend to listen to the same songs every Christmas. This year I decided to find some new favorites.

We have an excellent internet radio for Christmas songs in Finland called Jouluradio. The only problem when listening to Jouluradio is if you like some new song you hear, saving it for later listening is quite cumbersome. You need to find the track information from the Jouluradio web page and manually search the track from Spotify.

Fortunately, Jouluradio publishes its playlist of the last 20 songs on the service web page. I decided to make "a radio app" that grabs this information. If the tracks are found in Spotify, they are added to my personal Christmas radio playlist and when playing it I can easily save the ones I like for later listening.

My radio app, named Kaneli (cinnamon in Finnish) is a command-line program written in Golang. The trickiest part of the project was to implement the user OAuth 2.0 authentication to Spotify that I wrote about in my last week's post. The user authentication and authorization part is required for the app to have permission to add tracks to the user-owned Spotify playlist.

Otherwise, the program is mainly about fetching and posting JSON to various endpoints.

API providers and app interaction flow

When the songs' information is acquired from Jouluradio, the app searches from Spotify if the track is found there. The Spotify search API endpoint provides handy tools for this.


...

for _, track := range response.PageLastPlayed.Content.RecentlyPlayed.Songs {
  query := url.QueryEscape(fmt.Sprintf("artist:%s track:%s", track.Artist, track.Song))
  // search for song by artist and title
  trackResponse, trackErr := doGetRequest(fmt.Sprintf("https://api.spotify.com/v1/search?type=track&q=%s", query), bearerHeader)
  if trackErr != nil {
    fmt.Printf("Unable to fetch track data %s\n", err.Error())
    continue
  }

  trackData := SpotifyResponse{}
  err = json.Unmarshal(trackResponse, &trackData)
  if err != nil {
    fmt.Printf("Unable to parse track data %s\n", err.Error())
    continue
  }

  // just pick the first found track
  if len(trackData.Tracks.Items) > 0 {
    item := trackData.Tracks.Items[0]
    fmt.Printf("Add track %s: %s\n", track.Artist, track.Song)
    songIds = append(songIds, item.URI)
    removeIds = append(removeIds, &SpotifyRemoveItem{URI: item.URI})
  }
}

...

The search query is done for each track using the artist and song title information. If the search returns results, the first found result id is saved for later use.

When the looping is done, it's time to add the results to the user's playlist. For this Spotify provides a playlist API endpoint.

In this phase, I had one problem though: how to prevent the program from adding duplicate tracks. First I tried to find an API that would allow me to search if the playlist contains the specific track already. But luckily, there didn't exist this kind of API. Instead, I figured out a more efficient way of avoiding the duplicates: before adding the items, the program would make a delete request for the tracks that are about to be added.


...

apiPath := fmt.Sprintf("https://api.spotify.com/v1/playlists/%s/tracks", playlistID)

// first delete all tracks with similar id to avoid duplicates
_, err = doJSONRequest(apiPath, http.MethodDelete, &SpotifyPlaylistDelete{Tracks: removeIds}, bearerHeader)
if err != nil {
  return err
}

// then add all tracks to the start of the list
_, err = doJSONRequest(apiPath, http.MethodPost, &SpotifyPlaylistModify{URIs: songIds, Position: 0}, bearerHeader)

...

Both the delete and the add request are done by utilizing batches, so it is more efficient than removing or adding the tracks one-by-one.

The codes can be found in GitHub if you want to take a closer look. To run the app, you need to register your application in the Spotify developer dashboard and check the further instructions in the repository README.

The current version of the app loops through a couple of different radios dedicated to specific genres and adds the tracks to the user-defined playlist. So far I have been running the program manually at random times but I guess it would be possible to create a cron job for it to run e.g. once per hour. That is if one would like a really extensive radio list 😄

You can find my Christmas radio on Spotify.

Cover photo by Markus Spiske on Unsplash

Datetimes Are Hard: Part 2 - Writing and running code

Anniina Sallinen — Sun, 29 Nov 2020 13:21:05 +0000

In this part of the series, we go through how to parse and manipulate datetimes. This can be done in data pipeline with code or you can specify the format when you load data into database. In this blog post we focus on the first case.

Usually, data pipelines code are written with Python. In this blog post I'm using Clojure, though, mainly because I use it at work and it has nice library for handling datetimes. I'm using clojure.java-time library, version 0.3.2. As one can guess from the name of the library, it's a wrapper library for Java 8 time API. One of the good things about Clojure is that you can use java libraries and classes. Java happens to have a nice time API, so why not use it?

At this point, I want to point out that this is not a coding tutorial, and the purpose of the blog post is not to teach anyone how to program with Clojure. If you're not familiar with Clojure, the syntax might look quite overwhelming at first. I suggest you take a look at the Clojure syntax documentation if the syntax prevents you from understanding the examples given in the blog post.

Datetime data

I'm going to introduce the datetime classes I've used at work to represent datetimes. What they have in common is that their string presentation includes both date and time. Some classes in time API include only date or only time, but I'm not going to discuss them here. The code in examples are run in a REPL.

First, we are going to require the library, and alias it as t.

(require '[java-time :as t])

Local datetime

Local datetime is a datetime without timezone information. When there is no information about the timezone, datetime is assumed to be in local time.
For example:

(def now-time (t/local-date-time))
(str now-time)

would output for example

"2020-11-22T18:49:20.446"

You have to be mindful about when you can use local datetime and when to use some other class, which has a timezone. For example, birthday is something you could use local datetime for, but you wouldn't want to use it for example for scheduling a meeting, would you?

Local datetime is always relative to what local means. I might run some code locally, and check that datetime looks correct, but when I deploy the solution to for example to AWS, local might mean a different thing. Method now returns a different time depending on the system clock, unless you provide a timezone as an argument for it.

Instant

Instant is a single point on the timeline. The point is represented as nanoseconds since the epoch, but in string format, it looks like this:

(def instant-now (t/instant))
(str instant-now)

"2020-11-22T17:09:43.456Z"

As you may remember from the previous part of the series, Z at the end means datetime is in UTC. Instant can be used for example to record an event, for example when the event was received from an external system. Because instant is point on the timeline, it doesn't store date and time fields, only a number representing the point in time relative to epoch.

Offset datetime

Offset datetime is a datetime with an offset from UTC. As you may remember from the previous part of the series, it may have a negative or positive value, depending on whether it's less or more than UTC.

(def offset-now (t/offset-date-time))
(str offset-now)

"2020-11-22T18:58:54.961+02:00"

The offset datetime doesn't include the timezone information. Due to daylight savings, the same timezone might have a different offset depending on the time of the year. The timezone can be deducted from the offset, but the class itself doesn't store information about it.

Zoned datetime

Zoned datetime is a datetime with an offset and a timezone. It contains the most information about date and time.

(def zoned-now (t/zoned-date-time (t/zone-id "Europe/Helsinki")))
(str zoned-now)

"2020-11-22T19:13:54.682+02:00[Europe/Helsinki]"

Zoned datetime is needed for example when you need to convert local datetime to instant, since you would need the offset to calculate the time since epoch.

Parsing and formatting datetimes

If the datetime in data happens to be in a wanted format already, you can just give the datetime as an argument to the constructor

(def parsed-local-date-time (t/local-date-time "2011-12-03T10:15:30.234"))

Instead, if the datetime has space as a delimiter instead of character T, you need to pass the datetime format as an argument, too

(def parsed-local-date-time (t/local-date-time "yyyy-MM-dd HH:mm:ss.SSS" "2011-12-03 10:15:30.234"))

Otherwise local-date-time is not able to parse it.

Sometimes the datetime doesn't contain timezone information, but you know which timezone it should have. It might be that the documentation of the API tells you that, or you have confirmed it from the API developer, so you can set it as you parse the datetime:

(def parsed-zoned-date-time (t/zoned-date-time (t/local-date-time) (t/zone-id "Europe/Helsinki")))

Now let's have a look at how it works the other way around. The query might return for example an instant, and you need it to be in a certain format for sending it to an API.

You can do it as an one-liner

(->> (t/instant) (t/format (t/with-zone (t/formatter :iso-local-date-time) "Europe/Helsinki")))

but I personally think it looks a bit nasty, so I created a function for it:

(defn instant->iso-local-date-time-str [instant-to-format]
    (let [iso-formatter (t/formatter :iso-local-date-time)
          formatter-with-hki-zone (t/with-zone iso-formatter "Europe/Helsinki")]
        (t/format formatter-with-hki-zone instant-to-format)))


; More generic function, for formatting instant with any predefined formatter and timezone:
(defn instant->formatted-str [instant-to-format predefined-formatter time-zone]
    (let [iso-formatter (t/formatter predefined-formatter)
          formatter-with-hki-zone (t/with-zone iso-formatter time-zone)]
        (t/format formatter-with-hki-zone instant-to-format)))

Because in the example we used an instant, we need to provide timezone information before formatting it. That's because an instant has a different string presentation depending on the timezone. If you want to have the string presentation in UTC, you don't need the timezone and you can just go with

(t/format (t/formatter :iso-instant) (t/instant))

which gives you the same string as you would just

(str (t/instant))

You can find all predefined formatters in Java DateTimeFormatter documentation.

Manipulating datetimes

Once the datetime data is parsed into an object, manipulating it is quite straightforward. For example, if you want to add two weeks to the date:

(-> parsed-local-date-time
    (t/plus (t/weeks 2)))

Getting the start of the last month would be like this:

(-> parsed-local-date-time
    (t/adjust :first-day-of-month) 
    (t/adjust (t/local-time 0)))

Possible adjusters (such as :first-day-of-month) are:

:day-of-week-in-month
:first-day-of-month
:first-day-of-next-month
:first-day-of-next-year
:first-day-of-year
:first-in-month
:last-day-of-month
:last-day-of-year
:last-in-month
:next-day-of-week
:next-or-same-day-of-week
:previous-day-of-week
:previous-or-same-day-of-week

It is good to note, that you cannot use the adjusters for instant, because instant doesn't have information such as year or day of month, so you would need to convert instant to another datetime before using adjuster.

For converting datetime from a timezone to another you would need a zoned datetime. Converting the datetime from for example Helsinki timezone to UTC can be done like this:

(-> (t/zoned-date-time (t/zone-id "Europe/Helsinki"))
    (t/with-zone-same-instant "UTC"))

If the provided datetime is not a zoned-date-time but for example offset-date-time, you would first need to convert it into zoned-date-time before changing the timezone.

(-> (t/offset-date-time "2020-06-30T20:30:21.145+05:00")
    (t/zoned-date-time)
    (t/with-zone-same-instant "Europe/Helsinki"))

Pitfalls with handling datetimes

It's easy to mess up the datetimes. It's easy to forget about the timezones and offsets, and just use local datetime only to realize later datetimes are different from what you'd expected. In some cases that's fine, but in most data and analytics platforms, timezone and offset matters. They're also important if there is any kind of scheduling involved.

If you've set up for example Black Friday offers to open at midnight, and you're in Finland, you can find yourself in a situation when the offers are opening at 2 AM instead of at midnight because the system timezone is UTC. Probably not the most horrible situation, but you might get some angry feedback from the customers that have been waiting for the offers.

Another thing that might cause trouble is daylight savings. Once a year, you might have a situation where you're missing data from one hour. And once a year, you might find yourself in the situation that you have one hour at night that has more data than usual. This might not cause any issues, but it's a good thing to acknowledge.

And how about traveling? When we talk about some wearables that produce data, the local might change to another in the middle of a day. I do not envy the developers that need to face that type of issue when they build the system.

I hope this blog post has provoked some toughts and gave you a good overview of how datetimes can be handled with Clojure. In the next part I'm going to discuss about datetimes from the perspective of the database.

Photo by Lucian Alexe on Unplash

Datetimes Are Hard: Part 1 - Incoming data and formats

Anniina Sallinen — Sun, 22 Nov 2020 10:09:06 +0000

I have worked in the industry since 2014, first as a software developer and from 2018 on as a data engineer. Especially nowadays I have to work a lot with datetimes, and have come to realize that they are one of the hardest things in my job. I thought I'd write a series about datetimes from a data engineer's point of view. This is the first part of the series.

As a data engineer, one of my responsibilities is to build data pipelines. Multiple internal and external systems are producing data that we want to import and use. Since we cannot control the source systems and how they have designed their systems, we need to adjust to the decisions they have made. This means we need to, almost always, do some transformations to the data and the datetimes in it, before inserting the data into a database.

Datetime formats

There is a standard, called ISO 8601, which standardizes presenting date and time. It uses the Gregorian calendar and 24-hour clock and the purpose of the standard is that the dates and times are unambiguous. For example, 01/11/19 can mean either November 1st, 2020, or January 11th, 2020, so it is good to have a standard. I'm happy that I haven't met datetimes that are not following the standard, since it would make all of it even harder. This post doesn't aim to discuss the complete standard, but rather the formats I've seen in my everyday work.

Although there is a standard, it doesn't mean that handling datetimes would be simple. Although ISO 8601 defines a standard, there are still multiple formats available to use. Basically in the standard Y stands for a year, M stands for month and D stands for day. This means that YYYY-MM-DD is a datetime with a four-digit year, two-digit month, and two-digit day of the month. An example of such date could be for example 2021-01-20.

In the standard h stands for hour, m stands for minute and s stands for second. Using those, time can be represented for example as hh:mm:ss. This is a simple case. There might be fractions of a second, too. When there are fractions of a second, they are marked with s. They are separated from seconds with comma or dot, so the format becomes hh:mm:ss.sss. This is not how the time is always presented, though. The separator : is optional, so it could be hhmmss as well. Here's a table to summarize what I just explained:

CHARACTER	MEANING
Y	Year
M	Month
D	Day of the month
h	Hour
m	Minute
s	Second
s	Fractions of a second

Timezones

Usually, datetimes also include timezone information. ISO 8601 represents timezone as local time, UTC, or offset from UTC. The simplest case is UTC (Coordinated Universal Time). In this case it is represented as Z at the end of timeformat. An example of this is 23:59:59Z.

Then we have an offset from UTC. UTC is the same time as GMT (Greenwich Mean Time) which means it has a value of 0. The offset can be either positive or negative, meaning hours and minutes less or more than in zero timezone. The most common ways I've seen it is hh:mm or just hh. In this case, an example of time with offset is 23:59:59+02 or 23:59:59+02:30. Without : separator it could be 235959+0230.

The most tricky case is the one that doesn't have any timezone information. The time can then be assumed to be a local time, but it's a big assumption, and sometimes it is not clear what the local time is in the source system.

Finally, we can put the date and time information together. Usually separator T is used to separate date and time. In the older version of ISO 8601, it was permitted to omit T, but the decision was retracted later. I have seen both used, though. So to put everything together, these are few examples of how the source system can represent datetime:

WITH SEPARATOR	WITHOUT SEPARATOR
2021-01-20T19:00:00	20210120T190000
2021-01-20T17:00:00Z	20210120T170000Z
2021-01-20T12:00:00-05	20210120T120000-05
2021-01-20T19:00:00.125	20210120T190000.125
2021-01-20T19:00:00.125Z	20210120T190000.125Z
2021-01-20T19:00:00.125-05	20210120T190000.125-05
2021-01-20T19:00:00.125-05:30	20210120T190000.125-0530

So this was the first part of the series. After you've identified the date format that incoming data uses, you can parse it using a programming language of your choice. In the next part I'm going to show how to parse and manipulate datetimes with Clojure.

Photo by Kevin Ku on Unsplash

Git ohjeet suomeksi: Muokkaa edellistä committia

Lenni Ojala — Sat, 21 Nov 2020 16:43:24 +0000

Joskus git-kommitin tekemisen jälkeen tekisi mieli tehdä muutoksia - korjata commit-viestin kirjoitusvirhe tai tehdä vielä pieni muutos koodiin.

Tässä artikkelissa kerrotaan ohjeet commit-viestin muokkaamiseen sekä edellisen commitin tiedostojen muuttamiseen. Bonuksena myös ohjeet Vim-editorin käyttämiseen.

📝 Git-historiaa ei kannattaa muuttaa, jos olet jo puskenut (push) committeja remote-repositorioon muiden käytettäväksi, sillä muiden ihmisten historian sotkeminen ei ole hyvä tapa.

Alla on ohjeet commit-viestin muuttamiseen (ohjeet 1 ja 2) sekä edellisen commitin tiedostojen muuttamiseen (ohje 3):

1. Edellisen commit-viestin korvaaminen uudella

Kirjoita komentoriville uusi commit-message amend-parametria käyttäen:
git commit --amend -m "uusi commit message"

2. Commit-viestin muokkaaminen ja Vim-editorin käyttö

Jos haluat muokata commit-viestiä, jätetään -m (message) parametri pois, jolloin viesti avautuu gitin tekstieditoriin (yleensä Vi tai Vim). Tässä ohjeet editorin käyttämiseen:

Anna komento git commit --amend
Editori avautuu commit messagen muokkaamista varten
Paina tietokoneen näppäimistöltä i ja editori siirtyy kirjoitustilaan (insert)
Muokkaa commit-messagea. Risuaidalla (#) alkavat rivit eivät kuulu git-committiin.
Paina näppäimistöltä esc, jolloin editori siirtyy komentotilaan
Uusi commit message tallennetaan ja editori suljetaan kirjoittamalla :wq ja painamalla näppäimistöltä enter/rivinvaihto (tässä : aloittaa komennon, w tarkoittaa write (eli tiedoston tallennus) ja q tarkoittaa quit)

Vim-editorin sulkemisesta löytyy läjäpäin meemejä. Tämä naurattaa: "Olen käyttänyt Vimiä nyt noin 2 vuotta, lähinnä siksi, että en tiedä miten se suljetaan."

3. Tiedostojen muuttaminen edelliseen committiin

Edellisen commitin tiedostoja voi muuttaa amend -parametrin avulla seuraavasti:

Tee tiedostoihin halutut muutokset (lisää uusia tiedostoja tai tee muutoksia tiedostoihin):
Lisää tiedostot staging-tilaan komennolla git add ja tiedostonimi
Tässä välissä kannattaa ajaa komento git status ja tarkistaa, että halutut tiedostot ovat stagingissa
Commitoi lisäys edelliseen committiin komennolla git commit --amend tai git commit --amend -m "haluttu commit message"

Loppusanat

Jos tämän jutun git-versiohallintaan liittyvä terminologia kuulostaa vieraalta, ohje suomeksi löytyy esimerkiksi LevelUP-koodarien sivuilta.

The Best Python Resources for Beginners

Laura Vuorenoja — Thu, 05 Nov 2020 19:07:14 +0000

My friend told me the other day that she would like to learn Python coding for data manipulation and visualization. Even though I know that there are tons of Python tutorials out there, I had no clue where to point her to. In particular, because I haven't ever learned Python properly, I might not be the best judge for evaluating Python tutorials 😄.

So I decided to phone-a-friend and ask wiser people for some recommendations. As my friend has no previous background in programming, the material needed to be for absolute beginners. Here's a list of the suggestions I got:

Resources in English

Resources in Finnish

Is one of your favourite tutorials or learning resources missing? Please suggest more!

Great sneak photo 🐍 by Tamara Gore on Unsplash

Deploying containerized NginX to Heroku - how hard can it be?

Anniina Sallinen — Mon, 02 Nov 2020 07:40:58 +0000

I'm currently taking a Docker MOOC course. In part 3 of the course, there is an exercise about deploying a dockerized software to Heroku with CI/CD pipeline. The software can be anything, so I decided to deploy the course materials, and use Github Actions to make the deployment. Github Actions was very straightforward to use in my opinion, and I had no problems with that. Instead, I ran into some issues to get the materials available in Heroku. In this blog post, I thought I'd share the challenges I faced and tell how I solved them.

Course materials and how they're built

So the course material is a simple website built with Jekyll. The website was containerized, so it already had a Dockerfile in the course material repository. I forked the repository to be able to set up a CI/CD pipeline for it, and to deploy it to Heroku.
The Dockerfile that already existed in the course repository looks like this:

FROM jekyll/jekyll:3.8.3 as build-stage

WORKDIR /tmp

COPY Gemfile* ./

RUN bundle install

WORKDIR /usr/src/app

COPY . .

RUN chown -R jekyll .

RUN jekyll build

FROM nginx:alpine

COPY --from=build-stage /usr/src/app/_site/ /usr/share/nginx/html

So from the Dockerfile you can see that it uses a multi-stage build: in the first phase it uses Jekyll image, and in the second phase NginX. The repository doesn't contain any configuration for NginX, so it just uses the default configuration file.

The problem with NginX and Heroku

NginX is a widely used web server that can also act for example as a load balancer or reverse proxy.
As a reverse proxy, it takes a request coming from a client and forwards it to a server. This way the client doesn't communicate with the server itself, because the proxy is internet-facing, not the server. It can also act as a load balancer and send requests from clients evenly to different servers if there's multiple.
In this case, NginX is used as a basic web server, serving the static pages produced with jekyll build command.

Heroku is a PaaS (platform as a service) where you can deploy your software and host it in the cloud. It supports multiple programming languages, and also docker containers.

By default, NginX listens to the port 80. Now when I build the docker image locally on my laptop, and then run it with the command docker run -p 8080:80 docker-course-material I can open localhost:8080 in browser, and it will display the course materials. Now when I first deployed the materials to Heroku, the deployment to Heroku was successful, but opening the website showed a message that something went wrong. I noticed an error in the logs:

State changed from starting to crashed
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)

After googling a while, I found the information that Heroku assigns a random port for the web apps when the application is deployed. It says in the Heroku documentation that

Each web process simply binds to a port, and listens for requests coming in on that port. The port to bind to is assigned by Heroku as the PORT environment variable.

This means that I would need to make NginX listen to the random port that Heroku assigns for the application. How can I accomplish that?

First version of the solution

Let me be clear, although I have worked as a software developer before I went to data engineering, I had no experience in NginX whatsoever. I however realised I had to create a configuration file for it to make it listen to another port. My first goal was to make it listen to some other port, such as 8080 locally so that I could test the configuration.

I read that NginX uses configuration file in path /etc/nginx/conf.d/default.conf. My first solution was to create a configuration file that would replace the default file parent image had added, and that in the file a different port was defined. I struggled with the configuration a bit, but the first working version of the configuration file with hardcoded port looks like this:

server {
  listen 0.0.0.0:8080;

  location / {
    root /usr/share/nginx/html;
    index index.html;
  }   
}

The line with listen 0.0.0.0:8080; tells the NginX to listen localhost and port 8080. After copying the configuration file in Dockerfile, I was able to bind to the port 8080 and access the materials. This didn't solve the problem with Heroku yet, though.

Binding PORT environment variable

So now that I had succeeded in changing to port to 8080, I would still need to figure out how to define the port dynamically when the Heroku assigns it. I couldn't then just hardcode the port number, but instead I needed to use some placeholder and replace it when the container is run.

So I replaced the hardcoded 8080 in the configuration with $PORT and added a line to Dockerfile that would replace it with the environment variable. Because it needed to be done at runtime, the only option was to use CMD. I couldn't use RUN command in the file, because the environment variable PORT set by Heroku is available only when the container is started. RUN command is executed once at build time.

For replacing the placeholder with the value of environment variable I used sed s command. Sed is a simple stream editor and can be used to transform text. sed s is probably to most commonly known sed command, and it can be used to replace text with some other using regular expression.
The example of sed s command
sed -i 's/cat/dog/g input-file.txt'
replaces all occurrences of a string cat with string dog in the file input-file.txt. Option -i means that the substitution is done in-place.

Final version

So in the end, this is how the configuration file looks like:

server {
  listen 0.0.0.0:$PORT;

  location / {
    root /usr/share/nginx/html;
    index index.html;
  }   
}

and this is how the Dockerfile looks like:

FROM jekyll/jekyll:3.8.3 as build-stage

ARG PORT

WORKDIR /tmp

COPY Gemfile* ./

RUN bundle install

RUN echo $PORT

WORKDIR /usr/src/app

COPY . .

RUN chown -R jekyll .

RUN jekyll build

FROM nginx:alpine

COPY --from=build-stage /usr/src/app/_site/ /usr/share/nginx/html

COPY nginx.conf /etc/nginx/conf.d/default.conf

CMD sed -i -e 's/$PORT/'"$PORT"'/g' /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'

Dockerfile has only two new lines: copying nginx configuration file, and replacing the placeholder with value of $PORT. The final command nginx -g 'daemon off' is needed when the Docker is used, so that the NginX stays on the foreground and the container is not stopped immediately.

The final thing that I need to mention is that the problem I encountered is not, in fact, NginX-specific, but rather Heroku-specific. This problem can occur with any web server and Heroku, and needs to be solved by binding on port that Heroku assigns. I just happened to encounter this problem for the first time with NingX.

The solution is pretty simple, but I managed to use a couple of hours with this problem. If you do encounter the issue, I hope this blog post helps!