DEV Community: Bajic Dusko

Setting up Nextcloud as alternative to Google services

Bajic Dusko — Thu, 18 Jun 2020 13:48:32 +0000

This is a second post in a row in Moving away from Google services series. With the intention of replacing Google Drive, I have decided to try out a Nextcloud self-hosted solution.

Why Nextcloud

It can replace storage feature Google Drive offers.
It can upload mobile photos automatically, which replaces Google Photos.
You can install it on your home machine.
You can install many different applications inside Nextcloud, such as
- Carnet to replace Keep Notes from Google.
- Calendar
- Contacts
- Tasks
- Mail
You can link external storage via SFTP or SMB
Via WebDAV it syncs calendar events and tasks.
Office documents management.
Support for adding users, family management and its photos synchronization as well.

Basically, it supports wide range of features I consume from Google every day.

Private or "almost private"

If we're running a private cloud, then it should be hosted on a machine we physically have control of. I don't have a spare machine to run it on, so I've decded to purchase a smallest and cheapest VPS I could find. Time4VPS is great service with very acceptable prices. I've purchased
Linux 2 package for 4 Eur per month. It offers:

Single core CPU on 2.6GHz
2GB of RAM
20GB SSD storage
Linux distribution of choice. I've decided to install Ubuntu Server 18.04.

It's worth mentioning that I've tried installing Nextcloud on Raspberry 3 Model B at home. While the installation was successful and Nextcloud was running, unfortunatelly, Raspberry 3 have 1GB of RAM only, which isn't enough to host Raspbian, Nextcloud and MariaDB server, as MariaDB alone eats ~600MB.

It's important though, that if you have a Raspberry 4 at hand with at least 2GB RAM, you can install Nextcloud on Raspbian without issues. It'll work.

I have decided to go with VPS option due to my Raspberry performance issues. It's almost private solution, as mu virtual machine is somewhere in Lithuania.

Installing Nextcloud

Whether you choose to purchase a VPS or go with home setup, this guide assumes Debian based operating system in the background.

1. Install dependencies

Nextcloud is a PHP application running on MySQL database by default. Therefore we have to install its dependencies first:

sudo apt install apache2 mariadb-server libapache2-mod-php

sudo apt install php-gd php-json php-mysql php-curl php-mbstring php-intl php-imagick php-xml php-zip

2. Download and unpack Nextcloud

Navigate to /var/www/ directory. You can find latest Nextcloud version here.

sudo wget https://download.nextcloud.com/server/releases/nextcloud-19.0.0.zip

Extract downloaded package

sudo unzip nextcloud-19.9.9.zip

Make sure that directory structure looks like this: /var/www/nextcloud/ with the nextcloud contents inside.

3. Prepare database

Execute sudo mysql to enter MySQL CLI.
Create nextcloud user: CREATE USER 'nextcloud' IDENTIFIED BY 'nextcloud';
Create nextcloud database: CREATE DATABASE nextcloud;
Wire these two by adding permissions: GRANT ALL PRIVILEGES ON nextcloud.* TO 'nextcloud'@localhost IDENTIFIED BY ‘nextcloud’; and then flush'em FLUSH PRIVILEGES;
Quit MySQL CLI: quit

4. Configure Apache

User permissions

Our web server is Apache and user it creates is named www-data. Therefore, we have to allow this user to access to nextcloud directory so that Apache can serve its contents. While you're in /var/www directory, execute these commands:

sudo chmod 750 nextcloud -R

sudo chown www-data:www-data nextcloud -R

Configure Nextcloud

Navigate further to /var/www/nextcloud/config and open config.php file. This file contains basic configuration of Nextcloud setup and should be customized to fit our needs. I had a couple of push-n-pulls with this configuration until I make it correct.

sudo nano config.php

This is the example of my configuration:

<?php
    $CONFIG = array (
    'passwordsalt' => 'some_salt',
    'secret' => 'some_secret',
    'trusted_domains' => 
    array (
        0 => 'nextcloud.dusko.dev',
    ),
    'datadirectory' => '/home/dusko/nextcloud_data',
    'check_data_directory_permissions' => false,
    'dbtype' => 'mysql',
    'version' => '18.0.3.0',
    'dbname' => 'nextcloud',
    'dbhost' => 'localhost',
    'dbport' => '',
    'dbtableprefix' => 'oc_',
    'dbuser' => 'nextcloud',
    'dbpassword' => 'some_password',
    'installed' => true,
    'instanceid' => 'ocgn1qo2r675',
    'overwrite.cli.url' => 'https://nextcloud.dusko.dev',
    'htaccess.RewriteBase' => '/',
    'preview_max_y' => '1024',
    'preview_max_x' => '1024',
    'jpeg_quality' => '60',
    'maintenance' => false,
    'twofactor_enforced' => 'true',
    'twofactor_enforced_groups' => 
    array (
    ),
    'twofactor_enforced_excluded_groups' => 
    array (
    ),
    'has_rebuilt_cache' => true,
    );

Note on some of these properties:

- trusted_domain has to be set properly, if you're serving Nextcloud via your domain. In my case it's a CNAME pointing to VPS.

- I was forced to set check_data_directory_permissions to false as the data directory I set to be used is shared between multiple users, + the path I've set is a soft-link and I had a hard time setting this up properly. Easiest way of going around was to set it to false and never look back.

- preview_max_x and preview_max_y is related to thumbnails. There is a detailed thumbnails explanation in Nextcloud: Honest review post. I have literally lost whole week on setting this up and I'm still suspicious if it works properly.

Enable Nextcloud

To make Nextcloud available on webserver, we have to play with Apache a bit. First, let's configure nextcloud virtual host.

Open nextcloud.conf from sites_available directory:

sudo nano /etc/apache2/sites-available/nextcloud.conf

Here is an example of my configuration which includes Alias and subdomain with SSL setup. Alias is just a showcase as you might want to serve nextcloud on example.com/nextcloud. In my case that's not feasable and I went with subdomain setup.

Alias /nextcloud "/var/www/nextcloud/"

<Directory /var/www/nextcloud/>
        Require all granted
        AllowOverride All
        Options FollowSymLinks MultiViews

        <IfModule mod_dav.c>
                Dav off
        </IfModule>

</Directory>
<IfModule mod_ssl.c>

    <VirtualHost nextcloud.dusko.dev:443>

        ServerAdmin my_email.com
        ServerName nextcloud.dusko.dev

        DocumentRoot /var/www/nextcloud

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        SSLEngine on
        SSLCertificateFile      /etc/ssl/certs/dusko.dev.pem
        SSLCertificateKeyFile   /etc/ssl/private/dusko.dev.key.pem

        <FilesMatch "\.(cgi|shtml|phtml|php)$">
            SSLOptions +StdEnvVars
        </FilesMatch>
        <Directory /usr/lib/cgi-bin>
            SSLOptions +StdEnvVars
        </Directory>
    </VirtualHost>
</IfModule>

Now we just have to enable the site and open it. Execute these two commands:

sudo a2ensite nextcloud

sudo a2enmod rewrite headers env dir mime

sudo systemctl restart apache2

Open the domain you've set and you should be greeted with Nextcloud config page.

Upon logging in, you can install necessary applications and go through Settings section, to restrict access and tighten up the security a bit.

5. Mobile app

By installing Nextcloud on private server and setting up everything correctly, next step is to connect the mobile application to our Nextcloud instance. As I have the Android device only, I've tested and used Android version only.

Go to PlayStore, and download the app.

On the first run, you'll be prompted to enter server url and credentials. Once you pass that, you should be able to see the files and setup Auto-Upload feature.

Mobile application is highly troubling product at the moment. It's review is part of Nextcloud: Honest review as well. Make sure to go through that post before you decide whether to install the Nextcloud or not.

This post is originally posted on: dusko.dev

Nextcloud: Honest review

Bajic Dusko — Thu, 18 Jun 2020 13:17:33 +0000

You have probably went through short tutorial on how to setup Nextcloud in your private environment. If you haven't, make sure to check Setting up Nextcloud as alternative to Google Drive.

I am using Nextcloud for 4 weeks already, and this is my review.

Initial Nextcloud setup

In post install process, we have to go through the process of setting up some Nextcloud features of interest. First I've moved to security and turned on 2FA, by installing TOTP Provider application. You can go one step further and play with IP geo-blocking and brute-force IP blocking. So far I haven't made any of these settings.

Installing the apps

Under the Apps menu, there is a categorized AppStore with plenty of apps to choose from. I have installed Calendar, Contacts, Tasks, Mail and Carnet. All of these applications can be synchronized via Cal/Card/WebDAV to mobile phone via the application I'll install later.

Out of my current experience, those applications with 4.5 stars and applications with Featured flag are worth checking.

Reviewing each application is pointless. It's enough to say that the applications are working reliably. There are usual UI glitches here and there on Mail and Calendar, and UI/UX isn't perfect but it's acceptable. I guess I get used to Google apps UI for years and expecting all other apps to work the same way.

Installing the applications is straight-forward process. In the beginning, I had a trouble of finding installed app, as it isn't under the Apps menu, instead, it's under the Settings menu. Some apps are configured in Personal section, while some others are configured Administration section. This tends to confuse the user.

Navigation is confusing a bit in the beginning, but once you get to know what to expect, it's much better.

External storage

Without going into details how (this will be explained in the future post), I have attached two directories from my home NAS as external storage.

Now, I'm able to browse the photos from my NAS located in my home, via Nextcloud instance running on some remote VPS. As visible on the screenshot above, I've added GDrive Pictures directory, which is the directory synchronized with Google Drive at the same time. If you upload something to Google Drive, that file will be downloaded to my NAS and it'll show up in Nextcloud as well.

As shown on screenshot above, setting up the external storage is simple as choosing the protocol and filling the fields. I had it working from 6th or 7th try, as you have to guess where to put / in the directory path and to guess the domain value, WORKGROUP on the screenshot.

Once you pass that, it'll work stable and without issues.

Thumbnail generator

Having in mind that I've connected home NAS with photos originating from Google Drive and photos originating from mobile phones as well, generating thumbnails is massive work initially. You might think, why not just opening the directory and thumbnails will be generated in flight? Yes, you're right, but that'd be quite slow. All those images have to be uploaded from home NAS to Nextcloud running on VPS in order to generate thumbnails. You'd have to wait for the whole process to complete in order to see actual photos in the gallery.

For big galleries, this is a deal-breaker, as being unable to browse your photos "instantly", makes the product unusable.

Therefore, we have to pre-generate small thumbnails (called preview files from now one) which will be loaded in gallery when certain directory is opened.

By default, Nextcloud will generate files from 32x32 to 4096x4096 with step of power of 4 in between. For some photos, this means using more storage than initial file takes. For my >200GB media collection, this means generating preview files with almost the same size as original file. This is also a deal-breaker as I don't have storage that large on my VPS. Instead, I have 20GB only :).

I come up with two strategies for preview files:

I'll force the nextcloud to store the preview files on NAS as well
I'll have to limit the size of thumbnails somehow.

Preview files on NAS

Having preview files on NAS at home is a solution that works. Those files are small and are transfered at decent speeds to Nextcloud server. However, as VPS comes up with limited monthly bandwidth, that means that preview files would be transfered from home to VPS and from VPS to client browser. This would happen each time and for each user. Eventually, traffic builds up and bandwidth might be breached.

Additionally, it's quite hard to store only preview files on NAS. Due to coupled directory structure, storing preview files assumes storing some other metadata on NAS as well and this slows down the Nextcloud too much.

Preview generator

In the AppStore, there is an app called Preview generator. This application is used to generate previews, of course, but additionally we can do it from the console, instead of opening the directory containing the photos.

First issue you'll discover is that there is no GUI setup for it. This has to be done through the configuration files on the filesystem.
Next you'll realize that simply running the console commands won't do much as you'll get preview files with the same size. Therefore we have to find a way to configure the preview generator to generate smaller files.
Preview generator configuration isn't simple. Additionally, you won't find unique solution in the forums. Instead, you'll find a couple of solutions, claiming to be correct, but mutually quite different.

After whole week of try'n'errors with preview generator configurations I've manage to set it up and to have only necessary preview sizes for web gallery and mobile gallery, including bigger image compression. I've ended up with 3-4 GB of preview files, which is acceptable.

By utilizing the crontab, I'm running preview generator 4 times an hour, to generate previews for freshly uploaded content.

I'm not sure whether this become a huge issue for me as I have a tiny VPS, but I cannot escape the impression of this feature not being scalable enough. If you're offering the product called private cloud, I know you're assuming that users will install it on massive storage, but there are many users like me installing it on small VPS or on Raspberry with external storage attached and getting into preview generator problem instantly.

Another issue with these preview files is the computing power required to generate a couple of versions of same image and compress it. Running this on Raspberry takes days and days. I haven't tried it thought, that's something I read on forums.

If you're connecting your home storage to remote nextcloud, make sure to run the preview generator once everything is set up, as you'll find yourself re-uploading whole collection multiple times (as I did).

Mobile app

As a mobile developer myself, I might be too harsh on reviewing the mobile app. I've installed android version and connected it to my nextcloud instance. That worked instantly!

Out of main features for mobile application, I'd single-out browsing the nextcloud files and auto-upload feature.

Auto-Upload

Auto-Upload feature scans for changes on local directories on mobile phones (read: detects new photos) and uploads them to Nextcloud. It offers some predefined directory upload configuration, however I haven't found those much useful as it doesn't fit my environment. There is a super-cool Custom directory option where we can choose local directory to watch and remote directory to upload the files to. That's what I used.

I see this feature as a main use-case for the app, and you expect it to work flawlesly, however, it's pretty buggy.

There are some related features to auto-upload like "conflict resolve" (this reminds me on git) and choosing the default resolve strategy and I thought these features are advanced, but the more I use the app, the more I realise that these features are introduced to patch the unstable upload and transfer responsability to user. This is the result:

I can only guess what happened in the background. This is the result of file conflict resolve where (2) is added to the file name. And it happened 8 times in this case. This is almost always the case with large video files.

Having this feature as unstable as now makes the whole product almost unusable. I want to have photos from my and my wife's mobile device uploaded to Nextcloud. At this point I'm not sure if some photos are lost, skipped or duplicated.

Truth to be said, as unstable as is, eventually it uploads the content to server, but it's definitelly something I can't rely on.

Browsing photos

You can browse all files from mobile app. It's nice to have this feature and somehow its assumed to have it, however, it's quite slow and UX/UI should be improved.

By opening the photos directory you'll be presented with empty thumbnails for at least 10-15 seconds. And yes, each photo has a share button on it...

Disclaimer

I am very grateful for all the hard work developers are putting into this application, it is built in free time and I am aware that it's community driven product. That doesn't justify the lower quality of the product though. It has to be tightened up a bit and it'll work like a charm.

I get the feeling that there is a lack of proper QA. It must not happen that main use-case gets shipped and failing for everyone.

After I have installed the application, Auto-Upload wasn't working at all. By reading the reviews on the PlayStore I figured that developers wrote "if you want this feature to work, install beta version". That's not how its done people.

There is a huge potential in this product and, as other users in the reviews said, Not working Auto-Upload is a deal breaker for me, I'm gonna search for better product and I think the same. You see, this one feature makes people ditch whole Nextcloud product and that's the problem.

Synchronizing Calendar and Contacts

As mentioned, Calendar and Contacts can be synchronized through CardDAV/CalDAV. To have it working on Android, we have to install DAVx5. It costs 1$ and it's worth much more.

In order to login to nextcloud with this application, we have to generate an application password on Nextcloud. This is a must as DAVx5 app has to authenticate without 2FA involvement. With application password, we'll proceed to create an account on DAVx5 for our Nextcloud instance and it'll work by itself. To avoid confusion, this is just a local in-app account.

Mindblown moment for me was when I figured out that you can use it to connect native Calendar app on your mobile device (in my case Samsung Calendar) and create events in native Calendar app, to have it synchronized to Nextcloud eventually.

Its UI could be improved a bit, however as this application is made to work in the background, without much interraction, it's not a big issue.

Final word

Nextcloud is definitelly a solution to use in private environment. Its web app is awesome. Mobile app needs more work to stabilize. Nextcloud configuration and optimization could be improved overall.

From personal perspective, Auto-Upload feature works but it is not reliable and I'm looking to other ways of syncing mobile photos to Nextcloud. Using script and cron it to pull Google Photos might be one of the solutions (as Google Photos backup works reliably anyway), but then I don't need the Nextcloud at all and I'm stick to Google service which I'm trying to find alternative for in the first place.

If I ditch Nextcloud, I'm losing Calendar and Tasks features which I find as nice replacement for Google Calendar and Keep Notes.

Do you have similar experience with Nextcloud or you'd like to share a solution for the issues I've mentioned above. Please, tell me about it.

This post is originally posted on: dusko.dev