DEV Community: offen.software

Keeping track

offen.software — Mon, 31 Oct 2022 10:36:24 +0000

We are building Offen, a fair and self hosted web analytics software that treats operators and users as equal parties. Here's what's been happening over the past five months.

We are very pleased that @appwrite is supporting us as part of their initiative to promote the Open Source community. Big thanks guys for being our first GitHub sponsor! We really appreciate the recognition and generous funding. It helps us to continue to improve Offen Fair Web Ananlytics and make the web a better place.

Users of Firefox, Chrome and Edge can now install our companion browser extension Offen Istant Access that keeps track of all Offen Fair Web Ananlytics instances they have visited. The extension detects if there is an official installation running on the currently open website and displays a link to the dedicated Auditorium. Read our Offen blog post to learn more.

Also, our consent banner and the Auditorium for operators as well as users can be displayed in one more locale. Offen is now available in English, French, German, Portuguese, Spanish and, most recently, Vietnamese. Thanks to the wonderful contribution by @hiensarahly.

As always, be sure to stay tuned and follow us here or on Twitter and Mastodon for the next release updates.

Introducing the Offen Consent Tool

offen.software — Mon, 20 Jun 2022 13:10:16 +0000

We are building Offen, a fair web analytics software that treats operators and users as equal parties. Operators can gain insights into how users interact with their services, while ensuring that they retain full control over their data.

Along the way, we created the Offen Consent Tool. A lightweight solution for managing user consent on websites. Read the full version of this article on the Offen blog.

The Tool is using 1st Party Cookies to store user's consent decisions. To enable this mechanism, you need to deploy the respective server to a sibling domain, i.e. if you plan to use the tool on www.example.com, it should be served on a domain like consent.example.com. The tool can serve any number of domains at once, so it's possible to use the same deployment for multiple domains at once.

Next deploy the application to a domain like consent.example.com. On the host site www.example.com embed the client script:

<script src="https://consent.example.com/client.js">

which exposed window.ConsentClient. In your client side code, construct a new client instance pointing at your deployment and request user consent for the desired scope(s):

const client = new window.ConsentClient({ url: 'https://consent.example.com' })
client
  .acquire('analytics', 'marketing')
  .then((decisions) => {
    if (decisions.analytics) {
    // load analytics data
    }
    if (decisions.marketing) {
    // trigger marketing tools
    }
  })

The Offen Consent Tool further allows you to create the binary yourself and provides a development setup. It can also be used as a library and be integrated into any web server written in Golang. Learn more

Read the Docs for further assistance with installation and use. Open Docs

If you have any feedback, comments or bug reports about this or other project, we would love to hear from you. Open an issue on GitHub.

Celebrating v1.0.0

offen.software — Mon, 04 Apr 2022 12:50:01 +0000

We are building Offen, a fair and self hosted web analytics software that treats operators and users as equal parties. Here's what's been happening over the past four months. Read the full version of this article on the Offen blog.

Yes, you have read correctly. Offen v1.0.0 is here! Altough this does not bring major changes, we have decided to keep Offen and its configuration APIs stable for now. That means you will always be able to upgrade a v1 instance to a later v1 version without any further steps required.

Obviously, the development of Offen is not finished with ther current v1.0.0 version. We will continue to maintain the project and adapt it to new requirements. For this we also need your support. Continue to test and install Offen latest releases and share your experience with us.

Furthermore, our consent banner and the Auditorium for operators as well as users can now be displayed in two more locales. Offen is available in English, German, French and recently also in Spanish and Portuguese.

Maybe now would be a good time to look back on the past 2 years. However, since we are already working on projects that take the idea behind Offen even further, writing a project summary is postponed for the time being. In the meantime, we suggest taking a look at this interim retrospective and our analytics.txt project. Also be sure to stay tuned and follow us here or on Twitter and Mastodon for the next release updates.
_

Introducing the Offen Protocol

offen.software — Tue, 18 Jan 2022 09:10:24 +0000

We are building Offen, a fair and lightweight web analytics software that treats operators and users as equal parties. Along the way, we discovered many subtleties and details to consider and created the Offen Protocol for all software out there that aims to handle usage data in a transparent way. Read the full version of this article on the Offen blog.

The underlying concept is the definition of five actions that clients can take when they interact with a server that processes their data. These actions correspond to the rights of the data subject as defined by GDPR.

Probe is used to request additional information about the service.
Register is used when a client wants to make itself known to the server.
Submit is the action taken when a client transfers data to the client.
Query will be used when clients want to query the server for data.
Purge can be used by clients that want to initiate removal of data.

The full specification document can be found on the website. The protocol is not overly complicated and is perhaps even more of a convention than a specification. However, we have extracted what we use in Offen and added these implementations to the GitHub repository, which also contains the specification itself.

Please let us know what you think. We’re happy to open the discussion: tweet at us, send us an email, or open an issue on the GitHub repository.

Meeting our standards

offen.software — Thu, 04 Nov 2021 21:10:01 +0000

We are building Offen, a fair and lightweight web analytics software that treats operators and users as equal parties. Here is what we have achieved in the past weeks.

Statistics about the location of visitors had been on our to-do list for a fairly long time. Yet implementing it in a way that met our privacy standards proved to be a veritable challenge. After careful consideration and intense research, we finally decided on an approach based on time zones.

To derive the geographical location, this method does not rely on an IP database, but asks the browser for the selected time zone and tries to assign it to a country. This fully protects the privacy of users and provides sufficiently accurate results for analysis as well.

Furthermore, we have addressed the issue of user awareness in Offen. Since the only direct link to the User Auditorium is in the consent banner, it was important for us to provide additional features to increase the attention of users. Widgets now give operators the opportunity to easily integrate a reference to the User Auditorium. Preferably with a link on every page.

While there are still a few improvements on the agenda, with the implementation of location statistics we are a major step closer to v1.0. Be sure to stay tuned and follow us here or on Twitter and Mastodon for the next release updates.

Cohorts and customisation

offen.software — Tue, 29 Jun 2021 10:54:58 +0000

We are building Offen, a fair and lightweight web analytics software that treats operators and users as equal parties. Here's a brief glimpse of our achievements over the last ten weeks.

Customisable consent banner

To make it more convenient for operators to implement Offen in their websites, banners can now be freely adapted via CSS. This is done within some limits to ensure readability and functionality of the banner. Discover your inner artist and adapt the consent banner to the design of your website. Learn more in our Docs.

Explore cohorts

It is now possible for operators to filter the collected usage data based on URL, referrer, UTM parameters and landings as well as exits. In addition, we have added the tag "None" as a fallback for referrer values that are not supplied. Filters can be set and removed by clicking on the corresponding link element.

Engage in dialogue

We aim to bring Open and the idea of fair data transfer closer to an interested audience. To do this, we want to engage in a dialogue with operators, users and activists and are looking for suitable events around the world. We are happy to receive recommendations for relevant lectures and conferences. Got something in mind that we should apply for? Please let us know.

How to make your open source project contributor friendly 🤗

offen.software — Wed, 09 Jun 2021 10:19:35 +0000

There are many reasons that may lead you to the decision to run a project in an open source fashion. You might want to make your application code available to public to increase trust, or make it especially easy to distribute your library to increase usage. Another common motivation is to establish a healthy community of contributors around your project, allowing you to solicit feedback, make your project more diverse and maybe also distribute the burden of maintenance.

Contributions are not only about writing code

Before we talk about how to be friendly to contributors, let’s take a quick detour to define what a contribution to an open source project actually is in the first place. Oftentimes, contributing to an open source project seems to be synonym with opening a Pull Request that fixes a bug or adds a feature. To us, it means a lot more than that. Opening an issue about how something does not work for you is a meaningful contribution as well. Proofreading documentation is just as important as writing code. Offering help to localize the project in a different language is a major contribution. Leaving product feedback, both praise and criticism contribute to moving a project forward. So when we talk about a contributor friendly project, we talk about all of these means of contributing, not only about writing code.

Things we try to do in `offen/offen`

Offen is a fair and lightweight alternative to common web analytics tools. It’s self hosted software and all code is available at the offen/offen repository on GitHub. In this constellation, feedback and participation is especially important for us. In this article we’d like to share what we do to make our repo friendly and welcoming towards any kind of contribution from the community.

Head first in the README, in-depth docs for later

Striking the right balance between a brief introduction and an in-depth introduction is tough. If you have an idea and just want to get started, you don’t want to be bothered to read through detailed setup instructions. But if you have everything up and running, you probably would like to understand the how and the why of the project’s setup at some point.

In offen/offen we are catering both of those needs in different places. For people who just want to get things running as quickly as possible, we have a short intro in our README. More in-depth explanations on the development setup and the application architecture have been collected in a dedicated documentation site for developers.

This setup is also interesting because it will force you, the project maintainer, to answer two important questions: Can you get external contributors to get up and running with your setup in a few paragraphs? Can you also explain the setup and the architecture in-depth to outsiders in a way that it makes sense?

Contribution guidelines that are helpful and friendly

An established and recommended practice is to have a CONTRIBUTING.md file at the root level of your repository that has details about how to contribute to a project. This might seem like a trivial thing to write, but we found in reality it’s a tricky task to come up with something that is both inviting and helpful for potential contributors.

Often times, documents like this focus on what potential contributors should not do: Don’t open a PR without an issue, don’t open an issue without having searched the existing ones. Those are valid points from a project management perspective and also help the contributors themselves manage the repository. However, phrasing all of this as a list of "donts" and "only ifs" exclusively will subconsciously deter people from contributing. What would you prefer: a duplicate issue from time to time, or no issues at all being raised because users are too scared of doing the wrong thing? Also: do not forget to invite people to making non-code contributions as well.

Check our contribution guidelines for an idea of how we approach this. We also have a Code of Conduct that defines the way we'd like to interact with people.

“Boring tech” is accessible

If you want people to contribute code to your project, the choice of the tech you use is a major factor. Trying to make tech choices boring will help contributors to get started quickly, just because it’s already known to work, and also known how to work with.

In our example, Offen is built as a server side application with a relatively complex frontend running in the browser. On the server we chose Go as a language and the Gin framework. Those might not be most exciting choices in tech, but they are optimized for easy setup, robustness and plenty of good documentation. As a contributor, this means that when I run into issues setting up the project, I will likely find answers online soon instead of having to wrangle some custom framework the maintainers came up with themselves.

In our client side application we started out using the great, but slightly esoteric Choo framework, but soon migrated to Preact, just because the React ecosystem (Preact is a lightweight "clone" of React) is so ubiquitous and well-understood by a lot of people. We also do not use any fancy language features, but try to keep things simple so that you don't need to learn about brand new syntax constructs before you can contribute code. Think twice before you add something like TypeScript to a project you want to be accessible. Every additional layer - no matter how great or helpful they are for you who already knows how to use them - will make it harder for others to get started.

This topic also extends to your project's local development setup. The more extra tools you need the less likely it is someone will try to get started setting up your project on their machine. The setup should also work on all major OSes. A lot of people use Windows, some use Linux, many use a Mac, none of them should be left out. For Offen, we decided to use a docker-compose based setup, which gives us a reproducible cross-OS setup without people having to install anything other than Docker. You can check the compose file in our repository to see how that looks like.

Labels make issues discoverable

In case you host your project on a platform like GitHub or GitLab, the repository’s issue tracker will likely be one of the most important ways of interacting with contributors. Writing clear and concise issues that do not require much insider knowledge is one thing, but how do you make these issues discoverable for anyone who wants to contribute to your project? A common approach is to use labels for your issues.

One aspect of labeling issues is conveying information that is specific to your project. What language would you need to be proficient with to work on the issue? Is the issue not related to code, but an ongoing discussion people could participate in?

The other aspect of labeling is the external ecosystem that has evolved around GitHub issues. For example, sites like Up For Grabs allow anyone to search for issues labeled help-wanted across all public repositories on GitHub. Good first issue lets you find issues labeled good first issue that are well suited for beginners. Applying such labels will connect you with others that otherwise wouldn’t know about your project in the first place.

Check the issue tracker of offen/offen to see how we label our issues here.

Alternative means of getting in touch

We use GitHub for hosting our repository, and while this makes both code and issues publicly visible for anyone, having a GitHub account should not be a requirement for contributing to the project. Whenever we solicit feedback or contributions, we try to make it clear that people can also email us (bonus points for providing a PGP key for those who prefer to use one), tweet at us or use Mastodon to get in touch. Just because a service like GitHub or GitLab is free, it does not mean anyone can or wants to create an account with it. Having multiple feedback channels lowers the barrier of entry for anyone who wants to contribute.

What do you do to make your project contributor friendly?

Do you run an open source project that is geared towards creating a community of contributors? What do you do to make it easy for anyone to make meaningful contributions?

Let us know what you do, and also do not hesitate to start contributing your ideas and feedback about Offen. We’d be happy to have you on board.

Written by Frederik Ring

Disclose what you collect

offen.software — Tue, 25 May 2021 10:21:46 +0000

We have just drafted a standards proposition that will allow websites and services to disclose information about their use of analytics software and tracking software.

As this relates to our work on Offen, we want to give some insight into this matter.

The variety of privacy regulations worldwide and the consequently varying techniques for collecting usage data make web analytics a cluttered field. Terms such as "data protection" or "privacy-friendly" are widely used and show up regularly in consent banners and privacy statements of websites and services.

Nevertheless, it is often difficult to find out what they really stand for in a specific case. Thats why we believe that a comprehensive description of the use of analytics and tracking is missing. It should be understandable for a non-technical audience, but also suitable for processing by software.

Earlier this year, we started working on such a standard and named it analytics.txt. For now, it is available as a draft and waits to be discussed. We are actively looking for reviewers and welcome any feedback. Feel welcomed to take this idea forward together and get in touch with us. For more information on the standard and its implementation, have a look on the dedicated website.

View analyticstxt.org

Automate backing up your Docker volumes

offen.software — Mon, 03 May 2021 09:44:58 +0000

Docker is an ubiquitous tool for us while building Offen, a fair and open source web analytics software. It is foundational for our development setup, but we also use it for deploying our own Offen instance to production.

One thing that we found missing was a simple and lightweight tool for taking and managing remote backups of Docker volumes. This is why we wrote our own tool called offen/docker-volume-backup which. In this post I'd like to introduce you to the tool and how to use it for automatically taking backups of the Docker volumes in your own setup.

Introduction to Docker volumes

Volumes are Docker's way of managing persistent data. As Docker containers themselves are ephemeral, volumes can be mounted into the container's filesystem, enabling you to persist data beyond the lifecycle of a container. Volumes are commonly used for storing database data or similar.

For example, this is how you would use a Docker volume to persist data for an Offen container:

docker volume create offen_data
docker run -d \
  -v offen_data:/var/opt/offen \
  offen/offen:latest

In the running container, data stored in /var/opt/offen will be persisted in the offen_data volume and can be reused in other containers.

Using `offen/docker-volume-backup`

offen/docker-volume-backup is designed to run sidecared next to an application container and periodically take backups of volumes to any S3 compatible storage (i.e. AWS S3 itself or storages like MinIO or Ceph). It can run on any schedule you wish and it can also take care of rotating away old backups after a configured retention period.

If needed, it can temporarily stop and restart your running containers to ensure backup integrity.

Using alpine as the base image and using the MinIO client instead of AWS CLI for uploading files to the remote storage keeps the image small and lightweight.

Defining the sidecar container

The easiest way of managing such a setup is using docker-compose. A compose file that backs up its volumes would look something like this:

version: '3'

services:
  offen:
    image: offen/offen:latest
    volumes:
      - db:/var/opt/offen
    labels:
      - docker-volume-backup.stop-during-backup=true

    backup:
    image: offen/docker-volume-backup:v1.0.2
        # Ideally, those values should go into an `env` file or Docker secrets
        # as they contain credentials. It's easier to spell them out here
        # in the context of this tutorial though.
    environment:
            # A backup is taken each day at 2AM
            BACKUP_CRON_EXPRESSION: "0 2 * * *"
            # Backups are stored with a timestamp appended
            BACKUP_FILENAME: "offen-db-%Y-%m-%dT%H-%M-%S.tar.gz"
            # Backups older than 7 days will be pruned.
            # If this value is not given, backup will be kept forever.
            BACKUP_RETENTION_DAYS: "7"
            # Credentials for your storage backend
            AWS_ACCESS_KEY_ID: "<YOUR_ACCESS_KEY>"
            AWS_SECRET_ACCESS_KEY: "<YOUR_SECRET_KEY>"
            AWS_S3_BUCKET_NAME: "my-backups"
            # If given, backups are encrypted using GPG
            GPG_PASSPHRASE: "<SOME_KEY>"
    volumes:
            # This allows the tool to stop and restart all
            # containers labeled as docker-volume-backup.stop-during-backup
      - /var/run/docker.sock:/var/run/docker.sock:ro
            # All volumes mounted to /backup/<some-name> will be backed up
      - db:/backup/offen-db:ro

volumes:
  db:

Of course, you can also use the image using plain Docker commands:

docker volume create offen_data
docker run -d \
  -v offen_data:/var/opt/offen \
  -l docker-volume-backup.stop-during-backup=true
  offen/offen:latest
docker run -d \
  -v offen_data:/backup/offen-db:ro \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  --env-file backup.env \
  offen/docker-volume-backup:v1.0.2

Manually triggering a backup

Instead of running the backups on a regular schedule, you can also execute the command in a running container yourself:

docker exec <container_ref> backup

Restoring a volume from a backup

To recover from a backup, download and untar the backup file and copy its contents back into the docker volume using a one-off container created for just that purpose:

docker run -d \
  --name backup_restore \
  -v offen_data:/backup_restore
  alpine
docker cp <location_of_your_unpacked_backup> backup_restore:/backup_restore
docker stop backup_restore && docker rm backup_restore

The volume is now ready to use in other containers. Alternatively, you can use a one-off volume created beforehand.

More information

Detailed documentation and the source code is available at the GitHub repository and at Docker Hub. Source code is licensed under the Mozilla Public License 2.0.

Wrapping up

Knowing you have remote backups around in case of unexpected infrastructure glitches helps moving forward with confidence and not too much worry. I hope this article demonstrated that adding them to your Docker setup is only a matter of configuring an additional container, and helps you get going with your backups so you can move forward with your product.

Written by Frederik Ring

Broaden the view

offen.software — Wed, 21 Apr 2021 12:14:54 +0000

We develop Offen, a fair and self hosted web analytics software that treats operators and users as equal parties. Here's what's been happening over the past eight weeks.

Widescreen
We have further optimized our display features. The Auditorium for operators now makes better use of screen real estate on desktop devices. Also, we have optimized the display of bar charts in mobile view.

Offen is now available in German, English and recently also in French.
Our consent banner and the Auditorium for operators and users are available in the respective language. If you would like to support our fair approach to web analytics by contributing Spanish, Portuguese or other language versions, please do not hesitate to request an invite.

More secure
Our Docker image of the application are so far running as root. This would theoretically give the possibility to inject malicious code into Offen.
This did not happen in any Offen version. But to fix this possible vulnerability, all images published from now on will run the application as a non-privileged offen user.

Coming up
We want to present Offen and the idea of fair data transfer behind it to a professional audienc. Recommendations for relevant talks and conferences worldwide are very welcome. Got something in mind that we should apply for? Let us know.

How it all began

offen.software — Wed, 10 Mar 2021 10:41:48 +0000

Here's how we came to develop Offen, a fair and open source web analytics software.

Privacy issues on the web have been a talking point for some time now. Since we have both been running websites for quite some time, this has always been a topic for us as well. Thats why the choice of which web analytics tool to use has always been a very important consideration for us.

So in 2018, while once again searching for a new analytics tool, we found that there was no offering that allowed for a really fair data transfer between website operator and website user. That got us thinking. Of course, as website operators, we had an interest in seeing what was happening on our pages. Obviously, we wanted to know how we could improve what we offer. But couldn't this be done in a fair way, treating operators and users as equal parties?

Another few months had to pass and by the end of 2018, after further in-depth research, we were convinced. Yes, of course it was possible. And if it didn't exist yet, we should build it ourselves! So we started looking for support to build our own tool. And as we dug deeper, we were amazed at how many funding sources for developing open source software exist around the world. Nevertheless, there are of course a lot of applicants. That is why we believe that one has to think carefully about one's application.

That's why we focused on one German and a few international calls for proposals. And by summer 2019, we had submitted a few applications. To our great delight, our application to the NGI Zero Privacy & Trust Fund of the Dutch NLnet Foundation was finally successful. What happened next is covered in detail on our blog. In summary, it has been a great experience that is far from over.

Being able to secure funding for your non-commercial open source project gives you the distance you need to think more deeply about how you can create a real alternative to existing solutions. A constellation that we can only recommend to every developer out there.

Improving interfaces

offen.software — Mon, 08 Feb 2021 11:30:28 +0000

We develop a fair and lightweight web analytics software called Offen that treats operators and users as equal parties. Here is a glance at what has been happening over the last six weeks.

Offen is now also available in English and German.

Our consent banner and the Auditorium for operators and users are available in the respective language. A further locale, Indonesian, is in the works at the moment. If you would like to support our fair approach to web analytics by contributing French, Spanish or other language versions, please do not hesitate to request an invite.

Custom time intervals

To further develop Offen into a fully competitive analytics tool, we still have some essential UX improvements on our to-do list. In the last few weeks, we have been implementing just one of them. Using the new feature, you can now easily display customised intervals of existing data. In addition, we have revised the choice of default time periods. As always, we welcome feedback on usability.

Experimental JS API

For those who would like a little more precision in exercising user consent, here's some good news. Offen captures a consent decision from the user automatically and gathers pageviews if consent is granted. Now, for finer control, you can instead use the JavaScript API provided by the Offen script and trigger a pageview event all by yourself. Read more about this in our docs.