DEV Community: YngveNPettersen

Client hints or client lies?

YngveNPettersen — Fri, 13 Jan 2023 12:38:09 +0000

The User Agent header

The User Agent header has been used for decades to identify the browser connecting to the server, both for statistics purposes, and to work around bugs in specific versions of the browser.

Unfortunately, it has also been used to block browsers, because they are not one of the “supported browsers”. This has been implemented both as pure blocks, or more insidious ones like not sending the same content as for other browser, “because that browser does not support it, anyway”.

To work around such problems, browsers started to include parts of the identification strings of some of the other browsers, mostly the major ones. This may have started with Internet Explorer, if not earlier.

While at Opera, we tried to identify as just plain “Opera”, without any of the other pretensions, but we eventually had to start sending fake User Agents to many sites, due to them blocking us or sending us bad data.

One of the more notable cases was what I call the “Catch-22 cookie”, which was a bad cookie checker that usually works like this:

A new user loads the site.
Since the user does not have any cookies, check if the client supports cookies by sending a cookie.
Redirect to a new page that checks that the cookie was returned, and, if not, tell the user to enable cookies.

The problem in our case was that the code in step 2 sending the cookie first checked the User Agent string against a list in a Microsoft IIS server Client Capabilities module. It then only sent a cookie if the list said that the User Agent supported cookies. And since the module did not have Opera listed as a client that supported cookies, it did not send any cookies.

So, when reaching step 3, Opera, of course, did not have a cookie to send, because it never received one, and the user got told to enable cookies. Catch-22!

The site in question was rather large and important, so we needed it fixed. However, in the end, we actually had to submit patches to Microsoft to fix their server product.

To work, such a list needs to be based on perfect knowledge about all existing clients, which is difficult and costly, if not impossible.

The proper way of conducting this test would have been to always send a dummy cookie and remove it after the test completed. And, the capabilities module’s list should have been a block list for known user agents that the server shouldn’t send cookies to.

At Vivaldi, we’ve also encountered issues when identifying as “Vivaldi” and tried to use customized User Agents with sites that broke. Eventually, we gave up and started just identifying as Chrome to all websites, except those few that we knew would handle our Vivaldi ID correctly (e.g., our own site or our partners’).

But, it is not just the name of the browser that can cause trouble, it can also be the version number.

When Opera reached version 10, we actually encountered websites that believed the version was 1.0, not 10.0. The reason was that the scripts that processes the header were hardcoded to assume there was just one digit in from of the version dot, so two-digit numbers broke the script. We thus had to stop the version at 9.9 and add a second Opera-specific User Agent name to identify the 10.x+ versions.

We encountered similar issues when we reached Vivaldi 1.10. Again, websites assumed there was just one digit in the minor version number, so 1.10 was considered 1.1. Instead, we had to use 1.91 as the version number in the User Agent string.

When we later reached 2.10 and were still encountering extensive browser sniffing at levels that made adding overrides unsustainable, we decided “ it!” and stopped sending “Vivaldi” in the general User Agent header.

It is not just Client version numbers that are causing issues. A couple of months ago, I discovered that the Chromium code had frozen the MacOS version in the User Agent string at 10.15.7, the last released Mac OSX 10.x release, because websites would refuse to accept requests from machines running MacOS 11 (or later) and using that in the OS part of the User Agent String. Oops!

This is actually a problem for our bug reports, since we record the OS version from the User Agent string when bugs are reported. So, unless the reporter adds extra information, we may not realize the report is for Mac OS 11, 12, or 13.

In a related development, a year ago, the Chromium team actually had to run a long series of tests while preparing for their release of Chromium 100, due to the possibility of them encountering the same kind of problems. I assume Mozilla did something similar before they reached version 100.

More recently, there has been work to retire the User Agent header, starting with a reduction in the version information in the User Agent header. Chromium 106+ is no longer sending detailed information abut the version, just “106.0.0.0”. This is part of the transition to a new system of providing information about the browser: “Client Hints”.

Client Hints

Over the past few years, work has been going on to create a new system to provide more accurate information about the browser to the website, called Client Hints.

This new system is based around a set of HTTP headers prefixed with “Sec-CH-“, e.g. “Sec-CH-UA”, the new User Agent header. There are also headers for other information, browser engine, device, OS, resolution of the document on the display, etc.

Standards-wise, the system is based on the server indicating which of these headers it wants to receive. The browser then sends them if it supports them. However, Chromium, at the very least, is currently always sending three of these headers, including the User Agent (Sec-CH-UA), mobile, and platform information, and may send others.

Chrome’s Sec-CH-UA header contains information about browser brands and their version (Chrome and Chromium), as well as a “brand” value called “Not A Brand”.

The brand values are regularly (based on the version numbers) shuffled around in the header, and the “Not A Brand” value is also regularly modified by inserting various non-letter characters like “;”, “:”, and “.”, a process called “GREASE”, which has the effect of varying the header, so websites cannot rely on a particular sequence of values, or the text in the values. It thus attempts to force them into writing parsers that are standards compliant and don’t take shortcuts.

Vivaldi is currently not including a brand in this header, only sending “Chromium” and the “Not A Brand” values.

Client Lies

The big question about Client Hints is whether they (in particular Sec-CH-UA) will work better for browsers than the User Agent string?

Will websites properly parse the headers (Really?? ) and only use them responsibly (🤣 🙃). Or, will they start abusing this information to block “unsupported” browsers, too (😭)?

Unfortunately, the early indications are that some won’t parse properly, and others will use them to block “unsupported” browsers.

We have, so far, encountered two cases, one of each, where sites would not work for Vivaldi due to how the websites processed the Sec-CH-UA header information.

The first case was a website that worked in one version of Vivaldi but failed to load in the next, due to a server-side problem. This turned out to be due to the value shuffling and GREASE modification of the “Not A Brand” value. Chromium varies the sequence of values differently in each Chromium version, e.g.:

“Chromium”;v=”108″, “Not?A_Brand”;v=”8″

In the failing version, the sequence was “Not a Brand” and “Chromium”, and the “Not a Brand” value included a semicolon (“;”) character, which is used to separate values in unquoted text, but is just a normal character when it is in a quoted value. The website’s header parser ignored the quotes, and the result was that, when the “Not A Brand” value is first, the parser (and the server script) crashed.

Further investigation revealed that browsers with a branded header (e.g., Chrome and Microsoft Edge) would never have the “Not A Brand” value at the start of the header. Unbranded ones would have it that way every even-numbered Chromium version, that is, the Extended Stable versions, like 106 and 108. And every third of those would have a semi-colon in the string in a way that would break the parsing.

We “solved” that problem by freezing the sequence, so the “Chromium” brand was always first in the header.

In my opinion, part of the problem here is that Chromium does not vary the header enough, and not frequently enough. As mentioned, the branded browsers would never send the variation that caused our problem, and fact that Chromium only varies the content for each Chromium release, and not more often, means that it can take a long time (years) to discover parser problems like the one we encountered.

Part of the reason why the header isn’t varied more frequently is that the header value can be used as part of the key by websites to return cached webpages requested with the exact same set of URL and certain headers. Frequently changing the header value would increase the load on the website cache system and servers.

I still think that should not be a blocker for varying the sequence more frequently.

The second problem was a Japanese site that used the JavaScript APIs to access the Sec-CH-UA values and for whatever reason refused to serve the requested content if the brand wasn’t “Chrome” or “Microsoft Edge” (Firefox was accepted using a different test).

We have not yet worked around this issue, but, as far as we can tell, the only workaround is to use one of the “approved” brand names in our header. Using “Vivaldi” as the brand would not work.

If we encounter further issues of this kind, my guess is that the only way we can avoid the problem is to do like we did with the old User Agent string: pretend to be Google Chrome and make sure we do not send any Vivaldi-specific information.

That may not have been the desired goal for Client Hints, but if the last several decades of using User Agent string information have proven anything, it is that only the major vendors (OS or browser) are able to tell the truth. Everybody else will have to tell lies one way or the other – even Microsoft had to tell lies when they started distributing Internet Explorer.

Pulling the plug on expired Operating Systems

YngveNPettersen — Thu, 24 Nov 2022 14:41:19 +0000

All browsers run in an Operating System (OS) environment, whether it is named Android, iOS, Linux, macOS, ChromeOS, or Windows. These OSes provide a large amount of functionality to the applications running on them, like file system, keyboard, mouse, and graphics support. They manage the hardware-specific drivers for this functionality so that the applications don’t have to implement everything from fundamentals.

Life of an Operating System (OS)

Each OS is regularly updated with bug fixes, new functionality, and adaptions to new hardware. Usually, new functionality is added every few years, while bug fixes, especially security patches, are released more frequently to update existing OS versions.

Maintaining older versions of OSes become problematic after a while, as one needs to test all these versions on hardware and configurations that may no longer be available, or difficult to find. There will also be a question of having the resources, including employees who know the older systems and hardware.

How long an OS version is maintained, and what will be maintained, is usually decided well in advance of the last update. Usually, there will be several years of only releasing security updates, and even these can be difficult, if not impossible, to develop as the OS gets older.

Once the End-Of-Life(EOL) date is reached, the updates will stop, and users on the now-unsupported OS version will have to move to a newer version (if possible, or to new hardware), or live with the risk of having their computer attacked.

Just in the last few years, various OS versions have reached their EOL dates, e.g Android 6, Windows 7 (although one could pay for a few more years of support), MacOS 10.12, and early next year Windows 7 (extended support) and Windows 8.1 reach EOL, too.

Browser challenges due to different OS versions

All the various OS versions cause challenges for browsers, too. All that nice functionality the OS is managing for the applications is accessed through various function calls, called SDKsand APIs. These tend to change, new ones added, and old ones removed in newer versions of the OS, which means that to support older OSes special code needs to be written to adapt to the changes, and all of these variations need to be tested, usually on relevant hardware, which as mentioned can be hard to find. These adaptations also tend to complicate the logic of the code, making maintenance and testing more difficult, and can introduce bugs that are difficult to find.

So, when an OS version reaches EOL it becomes a question of whether or not the browser should support that older version, and how long it should be supported. One factor can be the number of users on the older platform, and another is how difficult continued support will be to maintain.

Goodbye Windows 7 and Windows 8.1

Regarding Windows 7, at present support has been maintained for three years past the consumer EOL, probably because there were still a lot of users, and because corporate users could still purchase maintenance updates. There might have been some overlap with Windows 8.1 variants. The extended support by Microsoft for Windows 7 ends in early 2023, as does the support for Windows 8.1.

At this point, for each EOL OS version, it becomes a question for browser engine vendors whether they should continue to support these versions, or remove the adaptions for the old version(s).

This evaluation must consider how difficult will it be to maintain, especially if supporting tools like the compilers and SDK packages, both of which need to be updated regularly to be able to support the newest OS versions, stop supporting the older versions, too. Once that happens, the question becomes even more pointed: Support old, obsolete OS versions, or the newer versions?

The correct answer is, of course, to support the newer versions, and retire the older ones.

Thus, at present, the Chromium team has decided to end support for Win7 and Win8.1 in early 2023. Following the point at which they no longer support these OS versions, the teams will start to remove the obsolete code, making it effectively impossible for a Windows executable to run on those OS versions.

Will Vivaldi support Win7 and Win8.1?

Sorry, no. Since Vivaldi is based on Chromium, we will get these updates removing Win7 and Win8.1 support when we next update the Chromium source, to Chromium 110, which will be the foundation for the next Vivaldi version after Vivaldi 5.6.

As a related example, on Linux, while Chromium did end support for 32-bit (386) Linux, Vivaldi continued to release builds as long as they continued to build and run until the builds stopped running. As it turns out, the most recent 386 release of Vivaldi (3.8) does not run, because Chromium updated to a version of a central library that was not supported by 386 versions of Linux (some Linux distros do distribute patched versions that avoid the issue in their 386 versions).

Recommendation: Update to a more modern operating system (OS)

When such support ends it is time to, at least, update the OS to a newer version, although in some cases that might mean buying new hardware (at least more disk space and memory, although Windows 10 may require a new machine).

Another option is to change to an alternative OS that supports your hardware, e.g. Linux supports quite old hardware and should be able to run 64-bit versions on CPUs from the past 10-15 years.

When you are using a computer with an OS installation that is more than 5-6 years old, it is usually time to consider updating to a more modern OS, anyway, even if the browsers and other applications you are using still work. It is usually just a question of time before the OS is no longer supported. By considering your options early, you won’t have to do emergency migrations (or risk security problems). For your system, having an up-to-date environment will provide the best features and security.

The trouble with Chromium translations

YngveNPettersen — Fri, 11 Nov 2022 12:33:17 +0000

Most applications that are intended for a broad international audience have their UI translated to various languages. But the number of languages varies widely, depending on the resources of the vendor, especially their ability to recruit translators.

Vivaldi is currently being translated to 91 languages, a few more than in Chrome.

Under the hood, Vivaldi’s UI string translation system actually consists of two independent systems: the Chromium one and rhe system used by Vivaldi’s native UI.

This article will only cover the Chromium system and the challenges of using it.

First things first

The Chromium string/text translation and resource system consists of two kinds of files:

The GRD files, which can declare the US English version of the strings, and the location of various file resources like icons and documents (HTML, JS, CSS) used by the Chromium UI, and

The XTB files, one for each language, contain the various translations of the original strings in the associated GRD file.

When building the product (Vivaldi, in our case), these files are processed by various scripts in the build system and converted into files that can be handled by the Chromium code handling strings and the resources.

One of the challenges for a product like Vivaldi is that the strings and resources defined by the Chromium project are very specific to Chromium and Google Chrome, such as logos and the company and product names used in the string.

But, of course, we in Vivaldi want to use the Vivaldi logo, and to use “Vivaldi” and name of our product and company.

This means that we have to change the resource definitions and the strings (and translations) to use Vivaldi’s preferred resources and names.

How do you change a file without changing it?

If you’ve read my article about maintaining a Chromium fork, you will have noticed that I said that you should never modify the Chromium translation files. Yet, I just said above that, to use our preferred resources and strings, we must change the files. Why shouldn’t we change the files, and how do we work around the problem so we can use our chosen resources?

There are two reasons why we should not change the files:

First, the Chromium resource files are frequently updated by the Chromium team. New resources and new strings are added, and old ones are changed to improve their meaning, and occasionally some are removed. All of these changes mean that when upgrading the Chromium source code, there is a significant risk that these changes will occur to the specific lines of the file we modified, or close to them. That means that we would have to resolve the conflicts between the new text and our changes, which will significantly increase the time needed to complete the update.
Second, for the strings we would have to not just modify the GRD file entry, we would have to modify the corresponding entry in each of the 80+ translation XTB files associated with each file, and to top it off, each of those entries has a numeric identifier calculated from the original string in the GRD file, so if you change the original string, you have to recalculate the value and update each XTB file for that entry. Ouch! Lots of work. Additionally, each of those updated entries in each file is another possible update merge conflict that has to be resolved manually. Double ouch!

“String” is a term used in computer programming for “a section of text“.

So, how do we resolve this problem? How do we update the resources, strings, and translations without modifying the Chromium resource files?

The answer is that we both do and don’t change them.

What we have done for Vivaldi is to create our own resource GRD and XTB files for each set of Chromium resource files that we want to update. We add our file resources, strings, and translations in these files. The translation files are usually used to add the translations for the extra languages we support, but in some cases we do an extensive rewrite of the original string, which requires more translations to be added in our version.

Then, while building the application, we have updated the project and the scripts it used to automatically insert our updated changes into the data, before they are used to generate the binary files used by the application.

The result is that we don’t have to update the original files, but we can update the resources, strings, and translations.

This process is also used to automatically replace mentions of Chromium and Google Chrome company and product names with Vivaldi’s name, both in the original US English strings and the translations. This process does have its challenges, especially since “Google” is frequently used in combination with other words to name products we don’t support, like “Google Pay”, so we have to exclude such replacements.

Occasionally, there are strings that mention the Google, Chrome, or Chromium names when replacing them with Vivaldi is not desirable (and an example just showed up in the forums, where information about a system Google is working on said “Vivaldi” instead, which has now been “fixed”). In these cases, we exclude that particular string from being replaced.

Another recent example was the string “Chrome is made possible by the Chromium open-source project”, which was auto replaced with “Vivaldi is made possible by the Vivaldi open-source project”, not “Vivaldi is made possible by the Chromium open-source project”. Oooops! This was fixed by adding a full override of the text with correct wording.

Is there any other way?

Could we avoid using this kind of system? Well, there are other ways to implement such a system.

We could add an independent set of resource files (and we have those for our own), and add our replacements in those files using different identifiers for them and replace the originals everywhere they are used. However, we would still have the problem with later updates, both of the strings and their meaning, and starting to use them elsewhere (which would have to be discovered and updated). Then, there is the issue of more potential merge conflicts during updates.

Quite simply, using different identifiers would not work very well, since their use would have to be maintained continuously. Just replacing the original entries will generally work better.

And that ignores the use of product names in many strings. There are a lot of those names used around the code, and copying and modifying them into a different set of files would be a major undertaking. They would also still have to be updated with new strings every Chromium upgrade.

The best way to avoid the search and replace of product names (and thus avoid the funny cases) would be for the Chromium team to stop using “Google”, “Google Chrome”, “Chromium”, etc., hardcoded into the strings. Instead they could use variables that can insert the downstream project’s own preferred name in those strings. But this kind of project would be a major undertaking by the Chromium team, and I have my doubts that they would be willing to take it on.

What do the other Chromium-based browser teams do? I have absolutely no idea. Maybe they use a similar system, or they have found their own way to manage the issue.

This article was originally published on my blog, Yngve’s corner. If you like these technical deep dives, be sure to stop by.

Soooo … you say you want to maintain a Chromium fork?

YngveNPettersen — Wed, 27 Jul 2022 09:39:26 +0000

Originally published on Vivaldi.net.

(Note: this article assumes you have some familiarity with Git terminology, building Chromium, and related topics)

Building your own Chromium-based browser is a lot of work, unless you want to just ship the basic Chromium version without any changes.

If you are going to work on and release a Chromium-derived browser, on the technical side you will need a few things when you start with the serious work:

A Git source code repository for your changes
One or more developer machines, configured for each OS you want to release on
Test machines and devices to test your builds
Build machines for each platform. These should be connected to a system that will automatically build new test builds for each source update, and your work branches, as well as build production (official) builds. These should be much more powerful than your developer machines. Official builds will take several hours even on a powerful machine, and requires a lot of memory and disk space. There are various cloud solutions available, but you should weigh time and (especially) cost carefully. Frankly, having your own on-premises build server rack may cost “a bit” up front, but it lets you have better control of the system.
A web site where you can post your Official builds so that your users can download and install them Now you are good to go, and you can start developing and releasing your browser.

Then … the Chromium team releases a new major version (which they do every 4 or 8 weeks, depending on the track) with lots of security fixes. Now your browser is buggy and unsecure. How do you get your fixes to the new version?

This process can get very involved and messy, especially if you have a lot of patches on the Chromium code. These will frequently introduce merge conflicts when updating the source code to a newer Chromium version because the upstream project have updated the code you patched, or just nearby, but there are a few things you can do about that to reduce the problems.

There are at least two major ways to maintain updates for a code base: A git branch, and diff patches to be applied on a clean checkout. Both have benefits and challenges, but both will have to be updated regularly to match the upstream code. The process described below is for a git branch.

The major rule is to put all (or as much as practical) of your additional independent code that is whole classes and functions (even extra functions in Chromium classes) in a separate repository module that have the Chromium code as a submodule. Vivaldi uses special extensions to the GN project language to update the relevant targets with the new files and dependencies.

Other rules for patches are:

Put all added include/imports after the upstream includes/import declaration.
Similarly, group all new functions and members in classes at the end of the section. Do the same for other declarations.
Any functions you have to add in a source file should always be put at the end of the file, or at the end of an internal namespace.
Generally, try to put an empty line above and below your patch.
Identify all of your patches’ start and end.
Don’t change indentation of unmodified original code lines, unless you have to (e.g. in Python files).
Repetitive patching of the same lines should be fixuped or squashed. Such repetitions have the potential to trigger multiple merge conflicts during the update, which could easily cause errors and bugs to be introduced.
NEVER (repeat: NEVER!!!) modify the Chromium string and translation files (GRD and XTB). You will be in for a world of hurt when strings change (and some tools can mess up these files under certain conditions). If you need to override strings add the overrides via scripts, e.g. in the grit system merging your own changes with with the upstream ones (Vivaldi is using such a modified system; if there is enough interest from embedders we may upstream it; you can find the scripts in the Vivaldi source bundle if you want to investigate). Vivaldi uses (mostly) Git submodules to manage submodules, rather than the DEPS file system used by Chromium (some parts of Vivaldi’s upstream source code and tools are downloaded using this system, though). Our process for updating Chromium will work whichever system is used, with some modifications.

The first step of the process is identifying which upstream commit (U) you are going to move the code to, and what is the first (F) and last (L, which you create a work branch W for) commit you are going to move on top of that commit. If you have updated submodules you do this for those as well.

(There are different ways to organize the work branch. We use a branch that is rebased for each update. A different way is to merge the upstream updates into the branch you are using, however this quickly gets even messier than rebasing branches, especially when doing major updates, and after two years of that we started rebasing branches instead.)

The second step is to check out the upstream U commit, including submodules. If you are using Git submodules you configure these at this stage. This commit should be handled as a separate commit, and not included in the F to L commits.

Then you update the submodules with any patches, and update the commit references.

The resulting Chromium checkout can be called W_0

Now we can start moving patches on top of W_0. The git command for this is deceptively simple:

git rebase --onto W_0 F~1 W

This applies each commit F through to L (inclusive) in sequence onto the W_0 commit and names the resulting branch W.

A number of these commits (about 10% of patched files in Vivaldi’s source base) will encounter merge conflicts when they are applied, and the process will pause while you repair the conflicts.

It is important to carefully consider the conflicts and whether they may cause functionality to break, and register such possibilities in your bug tracking system.

Once the rebase has completed (a process that can take several workdays) it is time for the next step: Get the code to build again.

This is done the same way as you normally build your browser, fixing compile errors as they are encountered, and yet again registering any that could potentially break the product. This is also a step that can take several work days. A frequent source of build problems are API changes and retired/renamed header files.

Once you have it built and running on your machine, it is time to (finally) commit all your changes and update the work branch in the top module and push everything into your repository. My suggestion is that patches in Chromium are mostly committed as “fixups” of the original patch; this will reduce the merge conflict potential, and keeps your patch in one piece.

Then you should try compiling it on your other delivery platforms, and fix any compile errors there.

Once you have it built and preferably have it running of the other platforms, you can have your autobuilders build the product for each platform, and start more detailed testing, fixing the outstanding issues and regressions that might have been introduced by the update. Depending on your project’s complexity, this can take several weeks to complete.

This entire sequence can be partially automated; you still have to manually fix merge conflicts and compile errors, as well as testing and fixing the resulting executable.

At the time of writing, Vivaldi has just integrated Chromium 104 into our code base, a process that took just over two weeks (the process may take longer at times). Vivaldi is only using the 8-week-cycle Extended Stable releases of Chromium due to the time needed to update the code base and stabilize the product afterwards. In our opinion, if you have a significant number of patches, the only way you can follow the 4 week cycle is to have at least two full teams for upgrades and development, and very likely the upgrade process will have to update weekly to the most recent dev or canary release.

Once you get your browser into production every couple of weeks you are going to encounter a slightly different problem: keeping the browser up to date with the (security) patches applied to the upstream version you are basing your fork on. This means, again, that you have to update the code base, but these changes are usually not as major as they are for a major version upgrade. A slightly modified, less complicated variant of the above process can be used to perform such minor version updates, and in our case this smaller process usually takes just a few hours.

Good luck with your brand new browser fork!

Cover photo by Ari Greve