DEV Community: Keeyan Nejad

How to migrate Google Photos to Proton Drive Photos

Keeyan Nejad — Thu, 07 Dec 2023 18:19:42 +0000

Proton Drive just released their photo syncing feature on Android. I've been patiently waiting for this since Drive came out.

In this post I'm going to explain a an easy way to migrate your data from Google Photos to Proton Drive.

This will get outdated quickly as I'm sure they will integrate a better migration service, but until then this should work.

First step is to go to takeout.google.com and download all of your Photos data.

You will eventually receive an email with a link to download all your photos.

Once downloaded, you can unzip all your pictures with this command:

UNZIP_DISABLE_ZIPBOMB_DETECTION=TRUE unzip <takeout-filename.zip>

Now you have a nested set of directories with many pictures in them. I recommend deleting all directories that don't have a name like Photos from 2010 as the rest are from albums which are likely duplicated in the Photos from XXXX files.

Now you will be left with all your photos as well as some json metadata files, which contain some info about when the photos were taken.

You could now, upload all the photos to Proton Drive via the web interface. Unfortunately, the photos will all be in the wrong order as they are sorted by the file creation date (which will be today) instead of the date the photo was taken. Also some files could have missing timestamp data.

Thankfully, there is a really cool tool called google-photos-takeout-helper. So you can go ahead and install that, and then run this command to sort all the timestamps:

google-photos-takeout-helper -i Takeout -o FixedTakeout

This will copy all the files to FixedTakeout with the correct metadata.

Now that this is done, we can finally upload the folder to the Proton Drive Photos service, and they should hopefully be in chronological order.

At the time of writing this post however, there is an annoying bug with the photo upload feature on the web interface which causes many failures.

So a better option, would be to copy all these files to your Android device, and select the new folder to be backed up to Photos, in the Drive settings page.

A Conservative Case for Concerns

Keeyan Nejad — Tue, 06 Oct 2020 19:55:18 +0000

There are many, many articles which condemn the use of concerns in a Rails project, and they all have valid points.
I mostly agree with the arguments, such as them causing Bi-directional dependencies and arbitrarily splitting up code into multiple files.
That said, I think a lot of these articles go too far by claiming you should never use concerns.

To be clear, if I saw a model that started with 20 concerns being included, I would be... Well, concerned.
That's why I've titles this a conservative case for concerns.
I think they should be used very sparingly, but not avoided like the Coronavirus.

Namely, I think when a concern does not depend on any specific methods or attributes of a model existing, doesn't contain code that is likely to evolve with business requirements, and could be used in a completely different project, then it's likely a valid use for a concern.

Recently, I was working on a project where we needed to import related CSV files.
In these files the associations were done based on the name of the record.
For example if we have a CSV file for articles, and each article belongs to an author, the CSV would have an author column which could have the name "Keeyan Nejad".
Then in the authors CSV we would have a name column which would include the names (assume for this example that all authors have a unique name).

When importing the CSV, the Author model would get a name from the name column.
Then I would import the articles, but when it would come time to associate an Article with an Author all I would have is the authors name, not the ID.
To solve this I would have to write something like this:

Article.create(
  title: csv_row['title'],
  content: csv_row['content'],
  author: Author.find_by(name: csv_row['author'])
)

This isn't too bad, but then I it turns out that the Authors belong to a country, and instead of having a country ID we have the name of the country.

So to import the Authors I would have to do this:

Author.create(
  name: csv_row['name'],
  country: Country.find_by(name: csv_row['country'])
)

I wanted to clean this up a bit and this is where I found a valid use for a Rails concern.
I wanted a way to associate a record by the name rather than by the ID column.

After a bit of playing around I came up with this code:

module AssociableByName
  extend ActiveSupport::Concern

  included do
    def self.associate_by_name(model)
      define_setter_for_model(model)
    end

    def self.define_setter_for_model(model)
      define_method("#{model}_name=") do |reference|
        association_class = self.class.reflect_on_association(model).klass
        association = association_class.find_by(name: reference)
        raise "Could not find #{model} by name #{reference}" if association.nil?

        send("#{model}=", association)
      end
    end
  end
end

Then in the Article model I just add these lines:

include AssociableByName
associate_by_name :author

What this code will do is create a new setter in the model called author_name= which will simply get the author ID from their name and create the association.

With that change, I can then update the importers to work more consistently:

Article.create(
  title: csv_row['title'],
  content: csv_row['content'],
  author_name: csv_row['author']
)

And that's it!
Now the article will automatically associate with the Author by the name, rather than having to do the lookup, I also get the added benefit that it will throw an error if the association doesn't exist (which is consistent with author_id if the ID didn't exist)

What do you think? I'm happy to be proven wrong and learn why this code isn't a good Concern, so let me know if you have any thoughts!

How to Change PostgreSQL Server with Minimal Downtime

Keeyan Nejad — Thu, 24 Sep 2020 21:15:50 +0000

Introduction

In this article you will move your database over to a new server, with only a few seconds of downtime.

Prerequisites

Before you begin this guide you'll need the following:

One Ubuntu 20.04 database server running PostgreSQL by following How To Install and Use PostgreSQL on Ubuntu 20.04.

Step 1 - Creating a server

The first step to migrating to a new server, is of course to create a new server instance. You can go through How To Install and Use PostgreSQL on Ubuntu 20.04 again in order to set up the new database server you just created.

Step 2 - Preparing the old database for replication

In order to transfer all the database data to the new server without any data loss or significant down time, you can configure the new server as a Hot Standby node and then shut down the old server and promote the new server to the primary node when ready.

To do this you must first SSH into the original server with the following command:

ssh root@old_database_address

Replace old_database_address with the address of the old database server.

Then edit the file /etc/postgresql/postgresql_version/main/postgresql.conf, replacing postgresql_version with the minor version number running on your server.

This can be found by running psql --version. For example, for psql (PostgreSQL) 9.6.17 postgresql_version will be replaced with 9.6

Append the following lines to the file:

wal_level = hot_standby
hot_standby = on
max_wal_senders = 3
max_replication_slots = 3

The first line will cause the Write Ahead Log to log information required to run read-only queries on a Hot Standby server. The second line will enable the server to act as a Hot Standby node. This isn't needed yet, but you will be making the same changes to the new server later and it is easier to copy the same configuration rather than making minor changes to each.

Next, you need to allow access to the new database server for backups, and for the web server by editing /etc/postgresql/postgresql_version/main/pg_hba.conf, replacing postgresql_version as before.

Add the following lines:

host replication replica old_database_address/32 md5
host replication replica new_database_address/32 md5

Replace the values as follows:

old_database_address should be the IP address of the old database.
new_database_address should be the IP address of your new database server

Finally, a new user needs to be created for the new server to be able to access:

su - postgres -c "psql -c \"CREATE ROLE replica REPLICATION LOGIN ENCRYPTED PASSWORD 'topsecretpassword';\""

Replace topsecretpassword with a secure password of your choice.

This will run an SQL command as the postgres user, which will create a new role.

Once these changes have been added, the PostgreSQL server should be restarted:

systemctl restart postgresql.service

You should then make the exact same changes to the new database server by repeating step 3 but instead of connecting to the old database you should connect to the new database server.

This can be done by running the following instead of ssh root@old_database_address:

ssh root@new_database_address

Replace new_database_address with the IP address of your new database server.

Now that both databases have been prepared to handle replication, it is time to actually replicate the data on the new database.

Step 3 - Creating the Hot Standby node and transferring the database data

In this step you will set up the new database instance to become a replica of the old server, using the built in Hot Standby node in PostgreSQL. To begin you must log into the new server with the following command:

ssh root@new_database_address

Replace new_database_address with the IP address of the database server.

In order for a Hot Standby node to receive the data from the original database, first, all its data must be deleted. This can be done by stopping the postgres instance and deleting the /var/lib/postgres/postgresql_version/main/ directory:

systemctl stop postgresql.service
rm -rf /var/lib/postgresql/postgresql_version/main

As always, replace postgresql_version with your version of the postgres database.

The first command will stop the PostgreSQL service from running, and the second will delete all the database data so that it can load the data from the original system.

Now you can copy the original databases to the new server:

su - postgres
pg_basebackup --pgdata=/var/lib/postgresql/postgresql_version/main/ --write-recovery-conf --username=replica --host=old_database_address --xlog-method=stream

Don't forget to change the postgresql_version to your version of PostgreSQL and old_database_address to the address of the original database.

The first command will make you the postgres user, and the second command will copy the data from the original server.

This command has a few switches which need to be set:

--pgdata switch specifies the directory where you want the data to be saved, the value has been set to the default directory where PostgreSQL databases are stored.
--write-recovery-conf creates a default recovery.conf file when finished that makes setting up the server as a Hot Standby node easier.
--username lets you set the user name you created when creating the new role.
--host allows you to specify the hostname of the original server.
--xlog-method ensures that all the Write Ahead Log files are copied along with the backup.

Once you execute this command you should be prompted for the password you created earlier when creating the replica role.

Note: If this command hangs it could be because there is an internal firewall rule set on the original server preventing access. This can be fixed by whitelisting the IP address added to the pg_hba.conf file earlier, depending on the original servers firewall settings.

Now that the new server has been set up as a replica you need to point the web app to the new database server and make the new database server the primary node.

Step 4 - Switching the primary database node to the new database server

Note: At this point you have to change the database address on any websites to the new database address and temporarily shut it down

Now, you must stop the original database server and promote the old one:

ssh root@old_database_address
systemctl stop postgresql

As always replace old_database_address with the address of your old database.

This will stop your original database server.

Next promote the new database server to master. This will make it the primary server so that it can start writing to the database:

ssh root@new_database_address
su - postgres
pg_ctlcluster postgresql_version main promote

Again, replace new_database_address with the IP address of the new database, and also replace postgresql_version with the version of postgres running on the server.

Now that the old database has been stopped and the new one has been promoted to master, all that needs to be done is to restart any sites which depend on the database.

Now you should be able to visit your websites and everything should be running as normal.

Step 5 - Cleaning up

Once you are satisfied that everything is working, you can shut down your servers on your old cloud provider and update the white listed addresses in PostgreSQL so that only your new server has access.

To begin, SSH into your database server again:

ssh root@new_database_address

Replace new_database_address with the IP address of your new database server.

Next you can edit the pg_hba.conf file and remove the old database and address. Once done editing the file should look something like this:

local all postgres peer
local all all peer

Now that everything is done, you can safely delete the original databases. Once that is done you have completely finished migrating to a new database server.

Conclusion

In this article you learned how to migrate your database from server to another with minimal down time. This process consisted of copying a database over to the new server using a Hot Standby node, then switching the primary database server, and shutting down the old database server.