DEV Community: Ricardo Iván Vieitez Parra

Privacy Pass: The Revolution in CAPTCHA Mitigation and User Privacy

Ricardo Iván Vieitez Parra — Fri, 09 Jun 2023 00:00:00 +0000

Introduction

As we traverse the digital frontier, the importance of robust security measures, devised to shield against harmful elements, has become paramount. CAPTCHAs – an acronym for Completely Automated Public Turing test to tell Computers and Humans Apart – serve this purpose, playing a pivotal role in authenticating online interactions. Since their introduction in 2003 by L. von Ahn and colleagues ¹, CAPTCHAs have become a commonplace presence, attempting to discern humans from automated bots, thereby safeguarding digital spaces from malicious activities.

Since their inception, CAPTCHAs have long been the go-to solution for differentiating between human users and automated bots on online platforms. These tests typically involve tasks like deciphering distorted characters, selecting specific images, or solving puzzles. While CAPTCHAs may have been initially effective in combating bots, they have become increasingly inadequate and burdensome for users in recent years.

Picture of an early CAPTCHA, as shown in the paper by von Ahn et al.

The rapid growth of the internet, combined with the increasing sophistication of malicious actors, has led to the widespread adoption of CAPTCHAs by countless websites and online platforms. These tests have become the gatekeepers of digital spaces, serving as the first line of defence against automated attacks, fraudulent activities and data breaches.

By presenting users with challenges that are easy for humans to solve but difficult for machines, CAPTCHAs aim to accomplish two crucial objectives. First, they seek to authenticate human presence, ensuring that real users can access and utilise online resources while maintaining a secure environment. Second, they aim to hinder or altogether prevent malicious bots from infiltrating websites, safeguarding sensitive information, and maintaining the integrity of online interactions.

In recent times, the prevalence of CAPTCHAs seems to have risen considerably, and it’s not hard to understand why. The growing reliance on digital platforms for various tasks, such as online shopping, social media engagement, and financial transactions, has made them attractive targets for actors seeking to exploit vulnerabilities and gain unauthorised access.

To keep pace with these evolving threats, CAPTCHAs have been continuously evolving as well. From deciphering distorted text or numbers to identifying specific objects in images or solving logical puzzles, the variety of CAPTCHA types has expanded to offer different levels of difficulty and security. However, these conventional CAPTCHAs are not without their limitations and drawbacks, often leading to frustrating user experiences and accessibility challenges for certain individuals.

Picture reCAPTCHA version 2, a typical modern CAPTCHA

Issues with CAPTCHAs

The rise of sophisticated bots armed with advanced algorithms and machine learning techniques has rendered traditional CAPTCHAs significantly less effective. Bots can now solve CAPTCHAs with surprising accuracy, compromising the security of online platforms and leaving them vulnerable to various malicious activities. As a result, the need for a more robust and reliable solution has become paramount.

Furthermore, traditional CAPTCHAs often lead to a subpar user experience. Users frequently encounter distorted images, complex puzzles, or illegible text, causing frustration and hindering their ability to access the desired online content. Not only does this affect user satisfaction, but it also leads to potential abandonment of platforms, resulting in lost opportunities and revenue.

There is yet-another concern with CAPTCHAs. Building any system at Internet scale that can with answer with certain certainity: ‘is this a real person?’ is inherently a very hard problem. Following up with the continuous arms race that occurs between CAPTCHA designers and CAPTCHA solvers is simply beyond the reach of most individuals and organisations.

In turn, this has lead to the consolidation of CAPTCHA-as-a-Service providers, i.e., third party services that offer integrations for embedding CAPTCHAs in our sites and applications. The largest such provider is Google’s reCAPTCHA, advertised the following way: ‘reCAPTCHA is a free service that protects your site from spam and abuse. It uses advanced risk analysis techniques to tell humans and bots apart’.

The centralized nature of mainstream CAPTCHA systems concentrates vast amounts of user data in the hands of a few entities. This concentration of data power undermines the principles of decentralization and user autonomy, reinforcing the existing power dynamics in the digital landscape. The potential for data misuse or abuse looms large, especially when considering the far-reaching influence these providers possess over individuals’ online experiences.

These privacy concerns are inherently difficult to address, because the reason that these services are able to work is likely in large part because of the large amounts of data they collect, which they can analyise and use to make inferences into what looks like typical human interaction and what looks like typical bot behaviour. This point is further reinforced by services like reCAPTCHA v3, that have largely done away with puzzle-solving and instead assign each visitor a score, with minimal or no interaction. Hence, it is probably safe to say that a privacy-friendly CAPTCHA is unlikely to be effective against sophisticated attackers, and that, even if it were, it may very well be difficult for humans to solve.

Introducing Privacy Pass

Cloudflare are a prominent Internet infrastructure company that offer a range of services, including protection against automated attacks. Among their services is a protection feature called ‘Challenge’, that enables website owners to verify human visitors, protecting their sites against automated attacks. The specific factors taken into account when issuing a challenge are unknown and dependent on configuration, but in many cases, it used a CAPTCHA as one of its signals.

Because a vast swath of the Internet is behind Cloudflare with such challenges enabled, many sites started in practice requiring solving a CAPTCHA on first visit. This was particularly inconvenient for private users, such as those utilising the Tor Browser, who would often need to complete one or several CAPTCHAS for every site visited. ‘Regular’ private users are in many ways indistinguishable from malicious users, which led to the unfortunate result of forcing them to solve a vast number of CAPTCHAS or worse, blocking access entirely.

Circa 2017, CloudFlare in collaboration with other actors in the industry and academia, developed what came to be known as ‘Privacy Pass’ ². In its original form, it would be a browser extension that would allow users to prove their humanity by solving a CAPTCHA, which would result in a special signed token that can later be sent to servers proving that a challenge had been successfully solved.

The main innovation of Privacy Pass was its ‘blind’ nature. This means that servers cannot link the tokens they sign with the ones redeemed in the future. As a result, users can maintain their privacy without compromising their communication with the server.

Moving on and into the browser

As innovative as Privacy Pass may have been, its lack of native browser support in practice limited its ability to take off. Installing a browser extension requires users to take additional steps to download it and configuring it, and that’s not taking into account that they must be aware of it in the first place. Adding an extension is also not always possible due to policy or platform limitations, and, as with all extensions, comes with an increased attack surface.

The IETF PrivacyPass working group are now in the process of standardising the protocols used in PrivacyPass so that it can be natively supported in all devices as a CAPTCHA alternative.

At a high level, it leverages the existing mechanisms in the HTTP protocol for authentication. When a server requires proof that the user is not automated, is transmits a challenge in the WWW-Authenticate header, just like it would for any other authentication scheme. It looks something like this: WWW-Authenticate: PrivateToken challenge="<base64url encoded data>", token-key="<base64url encoded data>".

The challenge part is a serialised and base64url encoded representation of the following structure:

 struct {
 uint16_t token_type;
 uint16_t issuer_name_length;
 char issuer_name[1..2^16-1];
 uint8_t redemption_context_length;
 char redemption_context[0..32];
 uint16_t origin_info_length;
 char origin_info[0..2^16-1];
 } TokenChallenge;

The TokenChallenge structure as shown in the latest draft of the Privacy Pass HTTP authentication scheme, with some changes to the types used to more closely resemble C syntax.

With the issuer_name field meant to include the hostname of the token issuer, the and the optional origin_info meant to include the hostname of the party issuing the request, note that there is little space left for including personally identifiable information about the user. The place where a unique identifier could fit is in the optional 32-bit redemption_context field, which is not visible to the token issuer.

Once a supporting client receives a challenge, it initiates a protocol with the issuer (as specified in the issuer_name field), first by requesting the resource at /.well-known/token-issuer-directory and then requesting a token be issued. The issuer is not provided with the raw value of the TokenChallenge issued to the client, but rather with a digest of it, meaning that it does not have access to read what little information can be encoded in the request.

After receiving a token from the issuer, the client relays this value to the origin that requested it using the standard HTTP Authorization header, for example, like this: Authorization: PrivateToken token="<base64url encoded data>"

The structure of the token is the following, where Nid and Nk are fixed integers depending on the token_type value:

 struct {
 uint16_t token_type;
 char nonce[32];
 char challenge_digest[32];
 char token_key_id[Nid];
 char authenticator[Nk];
 } Token;

The Token structure as shown in the latest draft of the Privacy Pass HTTP authentication scheme, with some changes to the types used to more closely resemble C syntax.

The origin requesting the token must verify that the authenticator field includes a valid signature over the entire value of the token, not including the authenticator itself (i.e., the first 66 + Nid bytes) and that the key ID identified by the token_key_id is valid.

This approach is deceivingly simple and yet tremendously powerful, because it enables providing an often required facility (closing off certain resources to automated requests) in a privacy-friendly manner that does not leak information about the user, and in a way that is also resistant to correlation attacks by the issuer. What is more, by implementing it as a standard using existing authentication mechanisms as building blocks, it is possible to support this feature while providing graceful degradation for clients that do not support the standard.

When and where can I use this?

Although the standard has not been finalised, it is already used in the wild and it can be used right now. Apple announced support for PATs in June last year in iOS 16 and macOS Ventura. It requires, however, being logged in at the device level to iCloud. As the specification is finalised, it is likely that other vendors will include support for the standard.

PATs are already being used by CloudFlare, for example for clients using ‘managed challenges’ or using the Turnstile service. Likewise, the feature is available to Fastly customers.

However, you do not need to entirely rely on a third-party service to use PATs, and we have developed an open source JavaScript library, PrivacyPass, also available on NPM to support issuing PAT challenges and redeeming tokens. The only, but crucial, part missing is the issuer. For this, you’ll need to use an existing open one, like the ones operated by Cloudflare and Fastly, or you can develop your own after gaining access to Apple’s attester service.

How will this not lead more centralisation?

The problem at heart, thwarting automation and automated attacks, is a difficult one to solve anonymously and at scale, and it is not surprising that existing solutions prior to PATs, like CAPTCHA services, require data collection to tell ‘good’ and ‘bad’ users apart. What is perplexing is that a solution that purportedly addresses the issue privately requires a specific brand of devices and being signed in to the vendor’s account.

This is unfortunate but not unsalvageable. For one, the PrivacyPass protocols do not mandate any particular vendor and is intended and easy for several and parties to perform the attestation part.

Then, there is the issue of why there needs to be an attestation at all, which introduces a bottleneck and closes off the protocol. The reason for requiring attestations is that without a trusted third party vouching for a particular user, it is impossible to enforce any system against automation. In particular, with self-attestation, an attacker can simply mint as many tokens as they desire, and then a few more.

We have explored some alternatives, such as Proof of Work, that would obviate the need for ’trusted’ parties. The main downside of these alternative approaches, even after ignoring the potential environmental concerns of ‘wasting power’, is simply that compute power is too cheap. Let’s do some simple calculations to drive this point home. Let’s say that we want to build a system to prevent spam, and that a cost of just 1 cent would be enough to deter spammers. Hence, we need to implement a proof of work system where producing a proof costs at least 1 cent. Let’s say that a spammer can get a server for $100 per month (likely an inflated number, and the server could in fact cost far less), which comes down to shy of 0.004 cents per second of server time. Therefore, to have a proof of work system that expends the equivalent of 1 cent, we’d need some algorithm that uses up around 250 seconds to construct a proof, or over 4 minutes. Such long-running proof of work would likely be unacceptable in any real-world scenario.

Going back to attestations, we believe that they can be done in an environment that fosters competition, remains privacy friendly and avoids vendor lock-in. For better or for worse, most consumer hardware these days comes equipped with some form of trusted hardware, such as a TPM, that could be leveaged to produce the rate limited tokens required for this protocol, without any changes that compromise privacy from the way those devices work today. As an alternative, external modules (such as a cheap USB dongle) could be made specifically for this purpose.

Closing remarks

We have explored the innovative solution known as PrivacyPass and its potential to revolutionise the way we approach CAPTCHA-related challenges. We began by examining the limitations and drawbacks of traditional CAPTCHAs, acknowledging their impact on user experience and their vulnerability to automated attacks. This led us to the introduction of PrivacyPass, a cutting-edge approach that combines cryptographic techniques with user-centric design principles.

PrivacyPass offers several advantages over traditional CAPTCHAs, most notably in terms of user experience. By generating tokens that vouch for a user’s human identity, PrivacyPass eliminates the need for repetitive CAPTCHA-solving tasks, allowing users to access online content swiftly and effortlessly. Moreover, the cryptographic proof systems employed by PrivacyPass provide enhanced security against automated attacks, ensuring a higher level of protection.

While PrivacyPass presents an innovative and promising solution, it is important to address potential concerns or criticisms that may arise. Most notably, PrivacyPass is reliant on remote attestation, which shuts off users that do not have or do not desire hardware with such capabilities.

Looking ahead, the field of CAPTCHA mitigation is ripe with possibilities for future developments and improvements. As technology advances, we can expect further refinements in the design and implementation of PrivacyPass and other similar solutions. Continued collaboration between researchers, industry leaders, and privacy advocates will undoubtedly lead to advancements that strike a balance between security, user experience and privacy.

In conclusion, PrivacyPass represents a significant step forward in the world of mitigation of automated threats. By prioritising user experience, security, and privacy, PrivacyPass offers an elegant solution to the challenges faced by traditional CAPTCHAs. With real-world applications already implemented by leading organizations, it is clear that PrivacyPass is shaping the landscape of online security. As we move forward, we eagerly anticipate the continued progress and evolution of CAPTCHA mitigation techniques, paving the way for a safer, more user-friendly online experience.

Conclusion

In an era where online security and user privacy are paramount, it is crucial to find innovative solutions that strike a balance between protecting sensitive information and ensuring a seamless user experience. Throughout this blog post, we have delved into the world of CAPTCHA-related challenges and explored the remarkable solution known as PrivacyPass.

PrivacyPass offers a unique approach to combating the limitations and drawbacks of traditional CAPTCHAs. By leveraging cryptographic techniques and user-centric design principles, it revolutionises the way we authenticate human users online. By generating Private Access Tokens that vouch for a user’s human identity, PrivacyPass not only improves user experience by eliminating repetitive CAPTCHA-solving tasks but also enhances security by mitigating automated attacks.

As we conclude, it is essential to recognize the significance of PrivacyPass in the broader context of online security. By adopting PrivacyPass, organisations can create a safer and more user-friendly online environment. Users can enjoy a streamlined browsing experience while knowing that their privacy is protected. PrivacyPass has already garnered attention and adoption from prominent organisations, demonstrating its effectiveness and potential impact.

We encourage readers to explore PrivacyPass further and consider its adoption as a viable solution to CAPTCHA-related challenges. By implementing PrivacyPass, organizations can enhance their security measures while improving the user experience. Stay updated on the latest developments in this field, as ongoing research and advancements will continue to shape the landscape of CAPTCHA mitigation.

In conclusion, PrivacyPass represents a significant advancement in the pursuit of robust online security and user privacy. By recapitulating the importance of online security, emphasizing the role of PrivacyPass in improving user experience, and encouraging its adoption, we can collectively strive for a safer, more user-centric online world.

L. von Ahn, M. Blum, N. J. Hopper, and J. Langford, “CAPTCHA: Using hard ai problems for security”, Lecture Notes in Computer Science, pp. 294–311, 2003. ↩︎
A. Davidson, I. Goldberg, N. Sullivan, G. Tankersley, and F. Valsorda, “Privacy pass: Bypassing internet challenges anonymously” Proceedings on Privacy Enhancing Technologies, vol. 2018, no. 3, pp. 164–180, 2018. ↩︎

Understanding Identity and Access Management (IAM)

Ricardo Iván Vieitez Parra — Wed, 05 Apr 2023 00:00:00 +0000

Introduction

Identity and Access Management (often shortened as IAM or IdM) consists of the set of frameworks, policies and practices that ensure the correct users have access to the right resources in a given system or collection of systems. Almost any system with multiple users will benefit from or require IAM, even more so systems that need limited access to certain parts, actions or resources.

IAM typically comprises two parts, the identity (or authentication, sometimes shortened authn) and access control (or authorisation, sometimes shortened authz). Authentication responds to the question of ‘who?’ and refers to the process used to identify a specific user (for example, using a username and password combination), and authorisation answers the question of ‘is this allowed?’ and represents the policies that determine whether someone may perform a specific action or access a particular resource.

Consider a simple message board system used for internal announcements within an organisation. The identity part of IAM may be fulfilled by prompting users (likely employees) for a username and password. Once users log in, they may have different roles, which define how to interact with the system and determine the access control part. For example, users may be divided into viewers (who can only read posted announcements), editors (who can read announcements as well as publish new announcements and edit and delete existing ones) and administrators (who can do all that editors can do as well as change the role assigned to other users).

Why IAM is important

Imagine a scenario where a company has a database of its clients’ details, including personal and financial information. If that database falls into the wrong hands, it could spell disaster for the company and its clients. For instance, the attackers could use the information to commit fraud, tarnish the company’s reputation, or cause financial loss. This is where IAM comes in; it helps prevent such scenarios by ensuring that only the right individuals can access the system and its resources.

IAM achieves this by identifying users and their roles, authenticating their identity, and authorising them to access resources and services that are relevant to their roles. It is important to note that IAM is not only about security but also helps maintain compliance, reduce risk, and increase productivity.

Because IAM plays such a central role in systems and organisations, implementing them correctly and efficaciously can be incredibly challenging, commanding specialist skillsets, a deep understanding of the business domain, best security practices and intimate knowledge of the systems.

Authentication

Authentication is the process of verifying the identity of a user before granting access to a system. The identity could be validated using various methods.

Authentication methods

Username and password

Passwords are by far the most common method of authentication, where the user provides a unique username and a password, and these credentials are then validated against the system’s records.

Username and password is a common authentication method for many applications. For example, it is the default method for establishing a session in many operating systems, as shown in this Windows Server 2003 log-on prompt.

This method is the most widely used authentication method due to its low entry barrier, ease of implementation, and convenience for users, as passwords are straightforward to use, do not require any additional hardware or software and can be easily changed if needed.

However, the use of password authentication has several disadvantages. Passwords can be easily compromised through brute force attacks, where attackers use automated software to try numerous password combinations until they find the correct one. Users also tend to choose weak passwords, such as common words or easily-guessable patterns, which attackers can easily crack. Furthermore, users often reuse the same password across multiple accounts, which increases the risk of a security breach.

Moreover, several steps must be taken to prevent passwords from being compromised when implementing password authentication. These steps include using encrypted connections and other general good network and operation practices to prevent interception during transmission, rate-limiting to prevent guessing by attackers, and proper password storage methods like hashing and salting.

Magic links

This authentication method involves sending a one-time link to the user’s email address or phone number, which they then open to gain access. Magic links can be a more convenient authentication method, especially for users who may forget their passwords, and avoids the issue of password management. However, the link could be intercepted or forwarded to an unauthorised user. Thus an attacker could gain access to the system, making it a relatively weak authentication method.

Magic links are a relatively low-friction authentication method. A link is sent, typically by email or SMS, that when opened automatically signs the user on.

Cryptographic authentication

Cryptographic authentication is a passwordless authentication method that uses cryptography to establish trust between the user and the server. Traditionally, this is done with digital certificates, also known as client certificates, issued to users and stored on their devices. When the user attempts to access a resource, a cryptographic protocol is run between the server and the user, ensuring that the user is whom they claim to be and possesses the private key associated with the certificate.

Client certificates are a high-security authentication method that has seen relatively low adoption.

While cryptographic authentication has been used for a long time, it has been challenging to implement on a large scale because of the complexity of managing the necessary infrastructure. The administration of certificates, certificate revocation and key management can be difficult and time-consuming, leading to higher development and operation costs than other methods, such as passwords, and potential security risks.

Newer approaches, such as FIDO and WebAuthn, have addressed these challenges of cryptographic authentication by using methods that do not require that the verifying party operate or maintain their own key infrastructure. The private key is typically stored on a hardware device, such as a USB key or a TPM on the user’s device, and a cryptographic protocol is defined to allow servers to validate identities depending on these keys.

WebAuthn is a newer approach to cryptographic authentication that commonly uses operating system features and hardware keys to verify identity.

Multi-Factor Authentication (MFA)

Multi-Factor Authentication (MFA) requires users to provide more than one form of identification to access a system. Each form of identification is referred to as a factor and can fall under three categories: knowledge or something you know (e.g., a password), possession or something you have (e.g., a physical token or a mobile device) and inherence or something you are (e.g., biometric data like fingerprints).

MFA combines two or more of these factors to increase security. For example, a user may be required to enter a password (something they know) and use a physical token (something they have) to gain access to a system. Alternatively, users may be asked to use their fingerprint (something they are) and enter a one-time password (something they have) sent to their mobile phone.

Some common examples of MFA include biometric authentication (such as fingerprints or facial recognition), one-time passwords (OTP) sent via SMS or a mobile app, smart cards, physical tokens, and push notifications to a mobile device. Overall, MFA provides an extra layer of security that can significantly reduce the risk of unauthorised access to a system.

Using multiple factors makes it more difficult for attackers to gain unauthorised access to a system, even if they obtain one of the factors (such as a password). Some factors (e.g., cryptographic keys) additionally support phishing-resistant methods like scoped credentials. These are designed to work only on the intended system and cannot be used elsewhere, preventing phishing attacks where attackers attempt to trick users into providing their log-in credentials.

Implementation options

The implementation of authentication is application-dependent and varies based on business considerations. Three main methods to implement authentication are per-system authentication, dedicated authentication services, and Single Sign-On (SSO) and federated identity.

Per-system authentication

Per-system authentication is where applications manage their own authentication, authentication methods, and user database. This approach can be simpler for developers because it does not require knowledge of the application’s environment and is more straightforward for small organisations because no additional infrastructure is needed. However, it can be challenging to scale as it requires managing user databases across many systems and enforcing policies. Furthermore, users needing access to multiple systems may need help navigating this authentication method.

Dedicated authentication services

Dedicated authentication services are a popular method in which a centralised service is responsible for authenticating users, and various applications can delegate authentication to this service. This approach avoids the challenges of managing users centrally and ensuring consistent security practices across multiple applications, leading to a better experience both for system administrators and users. Additionally, using dedicated authentication services reduces the attack surface area that comes with individual applications needing to implement authentication.

Various dedicated authentication services are available, ranging from commercial options like Okta and Azure Active Directory to open-source solutions like Keycloak and FreeIPA. These services typically provide features such as multi-factor authentication, user management, and access controls and can integrate with various applications using standards like OpenID Connect and SAML 2.0.

Single Sign-On (SSO) and federated identity

Single Sign-On (SSO) and federated identity are similar and sometimes synonymous with dedicated authentication services. The difference is that SSO typically refers to the widespread use of a central authentication service within an organisation. On the other hand, a federated identity allows users to access systems across different organisations using a single set of credentials.

SSO and federated identity may be combined. For example, user management may be entirely outsourced (using an external provider with their own user database), and likewise, organisations may make their authentication service available to other external systems. An example is social media log-in, which allows independent services to delegate validation of users’ identities to a social media network.

These methods are more convenient for users since they only need to remember one set of credentials but they might require more infrastructure. However, the benefits of SSO and federated identity will typically greatly outweigh the additional infrastructure required, as they provide a better user experience and improve security.

Authorisation

Authorisation is the process of granting or denying access to a resource. It determines what a user can do or what resources they can access once they have been authenticated and plays a critical role in securing resources and ensuring that only authorised parties have access to sensitive information or functionality.

Authorisation strategies

As for authentication, there are several ways to implement authorisation, and the best strategy is highly contextual, depending on the environment, business needs and nature of the systems in question. Some of the more common strategies are discussed below.

Role-Based Access Control (RBAC)

Role-Based Access Control (RBAC) is a widely used authorisation strategy that involves assigning roles to users based on their responsibilities or job functions. Each role is associated with a set of permissions that define what resources a user can access or what actions they can perform. This approach is particularly useful in organisations with many users because it simplifies management and scalability.

For example, consider a curling club management application that has different roles, such as players, coaches, and administrators. Players can view their team’s schedules and update their availability for upcoming games, while coaches can also view player statistics and performance metrics. Administrators can manage user accounts but cannot access player-specific data or modify game results. Using RBAC, the application can ensure that each user can access only the information and functions relevant to their role within the club.

Policy-Based Access Control (PBAC)

Policy-Based Access Control (PBAC) is an authorisation strategy that uses policies to govern access. Policies are rules that define what actions a user can perform based on their attributes, such as job title, department, or location. This approach provides greater granularity than RBAC and offers more flexibility. However, it can be more complex to manage.

A travel agency could use PBAC to control access to their booking system. For instance, they could have policies such as ‘Only travel agents with a certain level of certification can book international flights’ or ‘Only users in the sales department can offer discounts to customers’.

Attribute-Based Access Control (ABAC)

Attribute-Based Access Control (ABAC) is an authorisation strategy that uses attributes to determine access. Attributes can be any characteristic that can be used to identify users, such as user role, job title, department, location, time of day, and more. ABAC is highly flexible and can accommodate complex environments and scenarios, making it a popular choice for many organisations.

For instance, consider a tattoo parlour that uses ABAC to grant access to its online booking system. The system may allow customers with a minimum age of 18 years and those with valid identification to book an appointment.

Context-Aware Access Control (CAAC)

Context-Aware Access Control is an authorisation strategy that uses various contextual factors to grant or deny access to resources. The factors may include the user’s location, device, network, time of day, and activity history. By considering these contextual factors, the system can detect anomalous user behaviour and prevent potential cyber-attacks or data breaches. This strategy is often used with machine learning to help identify unlikely behaviours or represent high-risk scenarios.

For example, a grocery store might use context-aware access control to prevent unauthorised access to sensitive information and protect the store’s business operations by ensuring that only authorised personnel can access the inventory management system. The system may allow employees to access the system only during business hours, from the store’s Wi-Fi network, using a company-issued device. If an employee tries to access the system from outside the store or at an unusual time, the system can deny access or trigger an additional authentication step, such as a one-time password or biometric verification.

Access Control Lists (ACLs)

Access Control Lists (ACLs) are a commonly used authorisation strategy in which a list of users and the resources they are allowed to access is created. ACLs are typically applied to individual files or directories in a system, making them effective in managing access to specific resources rather than entire systems. This approach is well-suited for small-scale organisations where access to resources is limited to a few users or groups of users.

ACLs are often used for individual files or directories in a system.

For instance, ACLs can be applied to patient records to control access in a doctor’s practice to ensure that only authorised personnel have access to patient information, which is critical in maintaining patient confidentiality and complying with legal regulations. The doctor, nurses and receptionist may have different access levels, with the attending doctor having full access, other doctors having read-only access, the nurses having access to certain information, and the receptionist having no access.

Implementing authorisation

After choosing an authorisation strategy, the next step is implementing it in the relevant systems.

Similar to authentication, authorisation can be handled internally by each application through internal rules, or it can be delegated to an external service, which may be internally managed or outsourced.

Implementing authorisation can be complex, particularly in large and distributed systems. It requires careful consideration of the system’s security requirements, as well as the needs of the users and applications that will be accessing the system.

Internal rules

One approach to implementing authorisation in an application is by using internal rules. This method involves creating rules directly within the application and determining which users are granted access to specific resources. This approach suits small organisations with limited resources or simple systems requiring less infrastructure. However, as the size and complexity of an organisation’s IT environment grow, the management of internal rules becomes increasingly complex, time-consuming, and error-prone.

While the benefits of implementing internal rules include simplified management and implementation without needing external services, there are drawbacks. The risk of inconsistent or poorly enforced internal rules is a significant concern that can lead to security vulnerabilities. As such, organisations need to periodically review and update their internal practices to ensure that they align with the organisation’s security needs.

Examples of internal rules include row-based security, which restricts access to specific rows within a database, and ‘Gates’ in frameworks like Laravel, which enable developers to define custom authorisation logic.

Delegated authorisation

Delegated authorisation is the practice of delegating authorisation decisions to a purpose-built authorisation server rather than having each application manage its own rules. This server administers and enforces authorisation policies across multiple applications, services, and resources.

When applications require access to protected resources or data, they must request authorisation from the server. The server evaluates the request based on the authorisation policies in place, and either approves or denies it. This approach simplifies the management of authorisation policies across multiple applications and provides a centralised point of control and visibility over access to resources.

Delegated authorisation also helps enforce a zero-trust security model, where all access requests must be authenticated and authorised, regardless of the user, application or request origin. Zero-trust models can, in turn, reduce the risk of data breaches and other security incidents.

Examples of delegated authorisation protocols include OAuth 2.0 and SAML 2.0. These widely adopted protocols provide a framework for delegating authorisation decisions to an external server.

Delegation authorisation patterns

API gateways are an example of a pattern that can manage authorisation. Acting as a layer between the user and the application, an API gateway handles both authentication and authorisation. API gateways can control access to specific resources within an application or multiple applications. Examples of such services include Kong Gateway and Apache APISIX.

Query-based services are another example of a pattern for managing authorisation. These offer a service that applications can query to determine whether to allow or block access to a resource or operation. An example of this approach is using ORY Keto, an implementation of Google Zanzibar.

Conclusion

In conclusion, Identity and Access Management (IAM) is a crucial aspect of modern business operations. IAM ensures that the correct users have access to the right resources, which is essential for security, compliance, and productivity. There are many different authentication and authorisation strategies, each with its benefits and drawbacks. Organisations must carefully choose the strategy that best fits their needs and implement them correctly to maximise effectiveness. Implementing IAM can improve an organisation’s security posture, reduce the risk of data breaches, and increase operational efficiency by supporting business needs.

Opaque IDs: the ultimate protection against enumeration attacks

Ricardo Iván Vieitez Parra — Thu, 30 Mar 2023 00:00:00 +0000

In this post, we’ll discuss two types of attacks, timing attacks and enumeration attacks, which can result in disclosing confidential information to attackers when accessing resources. We’ll then introduce a method to neutralise these attacks using AEAD encryption.

Why IDs are used

A great many applications involve some sort of information retrieval for interactivity. The information is stored in some type of database and each item or resource that is typically assigned a unique identifier, or ID. This identifier can then be used to refer to that item later on.

For example, if you have a web application where users can upload photos, you might store each photo’s metadata in a database and assign each row a unique ID. The ID will then be sent in some form to the user, so that when she wants to see the photo the application can retrieve that file by looking up the ID in the database.

This applies not just to web applications, but rather is a feature of APIs in general. For example, a REST API might provide a user a path like http://api.example/foo/123 to identify a resource of type foo named 123.

How IDs might be exploited to gain access

Disclosing internal information (such as an ID) to users can result in providing them with access to other information that they are not intended to have access to.

While the ID in itself might seem like a harmless piece of information (and it usually is, in isolation), it could allow users to make inferences about the system and potentially gain access to private data.

German tank problem

The German tank problem gets it name from a statistical problem the Allies faced during World War Ⅱ. The Germans built a lot of tanks, and the Allies wanted to know how many as well as the monthly rate of production. Since the Germans used a sequential numbering system to label the components for each tank they built, when Allies captured one, they could see its serial numbers. By looking at the highest serial number they saw, and applying some statistical analysis, they could effectively estimate how many tanks the Germans had built.

This problem can be very relevant to information security. Suppose you are using sequential IDs for the photo upload application mentioned earlier and that each photo is given a sequential ID (e.g., 1, 2, 3, etc.). Then, by just observing the IDs being used, an attacker could gain information about the number of photos stored in the system as well as the rate at which photos are uploaded. Whether this is a concern or not depends, of course, on the specifics of the situation. Most likely, the information in this particular example is of little value, but it might be helpful to, say, a potential competitor considering building a similar application.

Enumeration attacks

While the German tank problem shows how our choice of identifiers might non-public reveal information to an adversary, many or even most applications are interactively accessible to adversaries. This interactivity allows attackers to carry out another attack that can give them the same information directly, without the need to guess: an enumeration attack.

An enumeration attack is a type of attack where an attacker attempts to gather information about a system by systematically trying different values for a particular parameter. In the context of IDs, an enumeration attack might involve an attacker trying a large number of possible IDs to determine which IDs exist and which do not. Once the attacker has determined which IDs exist, they can potentially use that information to carry out further attacks.

Enumeration attacks can be particularly effective when the IDs being used follow some kind of sequence or are represented in a relatively small set and are part of a publicly available URL, as it is trivial to automate the process of trying a large number of possible IDs.

For example, consider the previous API with an endpoint like http://api.example/foo/:id, where :id is a numeric and sequential ID. It is trivial for an actor to try many requests, such as http://api.example/foo/1, http://api.example/foo/2, http://api.example/foo/99, etc., and evaluate the response.

The impact of enumeration attacks depends on the application in question. In many situations, successfully exploiting an enumeration vulnerability will only provide information about which resources are valid, making it similar to the German tank problem. However, in combination with additional vulnerabilities in the application, especially broken access control, it could have farther reaching implications, such as leaking confidential information, including customer data.

Prevention

Enumeration attacks can be avoided or mitigated by addressing the factors that make them possible: small sets and predictable values. A common solution is to use UUIDs version 4, which consist of 122 random bits and are therefore impossibly impractical to guess or enumerate.

Timing attacks

A timing attack is a type of side-channel attack where an attacker attempts to gain information about a system by measuring the amount of time it takes to perform certain operations. In the context of IDs, a timing attack might involve an attacker trying to guess a resource’s ID by trying a large number of possible IDs and measuring the time it takes for the system to respond. By measuring the time it takes to receive a response, the attacker can potentially learn information about the system, such as which IDs exist and which do not.

Timing attacks can be a way to extract information even from systems that implement proper access control. Consider the endpoint from earlier: http://api.example/foo/:id. Let’s say that IDs follow a simple numeric sequence (i.e., 1, 2, 3, etc.) and that the no path IDs is publicly accessible. Our adversary is a user to the system with legitimate access to the resources with an ID of 11 and 17, and no access to any other resources. How could this adversary carry out an enumeration attack?

Because of difficult or impossible to avoid circumstances when developing such an application, the endpoint will likely take different time to respond depending on the result. This information, if consistent, can be revealing of the internal state. Our attacker could proceed to make queries against the endpoint to all resources, for example, from 1 to 100 000. The measured response times could look like something like what is shown in the table below.

ID range	Mean response time (ms)
1–10	50
11	57
12–16	51
17	56
18–7 363	49
7 364–100 000	45

Table with mean measured response times for different IDs. Note the difference between various IDs. The IDs the attacker has legitimate access to have a mean response time of about 56ms, whereas the remaining resources have response times of about 50ms for IDs under 7 364 and of about 50ms for higher values.

Note that from ID 7364 onwards the response time is lower. This information can reveal to the attacker that the system likely contains 7 363 entries.

Depending on how lookups are carried out, timing information can also be used to identify valid IDs even when they don’t follow a sequence and there is a much larger set of valid IDs (such as UUIDs). This means that the specifics of how data are looked up are important, not just the format of IDs.

Mitigation

There are several ways to mitigate timing and enumeration attacks. As mentioned earlier, one common solution against enumeration attacks is using random UUIDs instead of sequential IDs. These have a very low probability of being guessed by an attacker. However, UUIDs can be relatively long, which can be a disadvantage in certain contexts, such as when they need to be used in URLs.

Timing attacks, being a side-channel attack, are impossible to eliminate entirely because so doing would require eliminating all possible side-channels. However, there are ways to practically mitigate them. Ideally, we would write our application so that response times are independent from user-provided data. Since this is impractical, we can practice defence-in-depth and make timing information less useful and more difficult to obtain.

For example, if we are concerned about enumeration attacks, we might add a MAC or digital signature to IDs, which (1) users cannot readily forge and (2) we can verify before proceeding further with the request.

The new IDs could look like the original IDs, but with the MAC or signature prepended to it, like this <MAC>.<ID>. So, the internal resource 1 could result in a user-visible ID like 53CUR3C0D3.1, with 53CUR3C0D3 being a value that we can verify as corresponding to resource 1, but which is difficult to forge or guess. If we verify this value before handling the request further, an attempt at the timing attack described earlier could instead have resulted in measurements like the following.

ID range	Mean response time (ms)
1–10	32
11	59
12–16	33
17	60
18–7 363	31
7 364–100 000	32

Table with mean measured response times for different IDs after only processing requests with a valid MAC. Note that mean response times are divided into two categories this time. For IDs the attacker has legitimate access to, we see similar, slightly higher, response times of around 60ms. For all other IDs (where an invalid MAC was provided), the response time is of around 32ms.

Note that in this case, an attacker does not gain much information besides what they already know (i.e., the representation of resources 11 and 17).

Encrypted IDs

While a MAC or signature can go to great lengths towards preventing enumeration and timing attacks to learn about valid IDs, the issue arising from the German tank problem, namely, making inferences from observable IDs, remains. In order to mitigate this, we can use encryption to make internal representation of the ID itself opaque to external parties.

Indeed, we can use various authenticated encryption schemes (e.g., AES-GCM) to hand out users IDs which are both opaque (they do not allow to infer information about their internal value, structure or representation) and unforgeable (IDs not generated by us can be detected and rejected).

However, there are some potential downsides to using this method. One issue is that the encrypted IDs are not human-readable, so it may be difficult for developers to work with these IDs directly. This can make debugging and troubleshooting more challenging.

Another potential issue is that the overhead involved in encrypting and decrypting IDs can be significant both in terms of computing time and space. This can be a concern particularly in applications where a large number of IDs need to be generated and transmitted or in certain resource-constrained applications.

Example solution in TypeScript

We have developed a small library, @exact-realty/safeid, that implements the techiques discussed here applied to UUIDs, with the possibility of extending it to other arbitrary ID formats.

Approach taken

@exact-realty/safeid uses AES-GCM to produce IDs that are opaque, unforgeable and stable (i.e., the same internal ID will always result in the same encrypted ID).

To do this, first we provide the library with a secret key that will be used to derive other cryptographic IDs.

Then, before providing an ID to a user, we encrypt the internal representation. This is a multiple-step process:

We derive an HMAC key from the supplied secret key (this step is carried out before any encryption or decryption operation).
We use the HMAC key to produce the an IV to be used for the following encryption step, taking the plaintext internal ID as input to the HMAC function. This is the step that ensures that the resulting output is stable.
We derive an AES256-GCM encryption key from the supplied secret key. In this implementation, we additionally use the IV derived in the previous step an an input to this process, which results in a different encryption key for each ID.
We proceed to encrypt the supplied internal ID, using the derived IVs and encryption keys.
We prepend the IV to the resulting encrypted data and encode it using the base64url encoding and return this as a result (which for UUIDs results in a 48-byte string).

For decryption, the steps are as follows:

We derive an HMAC key from the supplied secret key (this step is carried out before any encryption or decryption operation).
We decode the encrypted ID and split it into two parts: the IV and the encrypted data.
We derive an AES256-GCM decryption key from the supplied secret key. In this implementation, we additionally use the IV, which results in a different decryption key for each ID.
We decrypt the encrypted ID to obtain the plaintext internal ID.
With this information, we derive the IV from the plaintext and verify that the IV in the input that we obtained earlier matches what we expect it to be (i.e., the IV that the encryption function would have derived). This step is a sanity check to ensure that the ID has not been tampered with.
We return the plaintext internal ID, which our application can now use for accessing resources.

Progressively loading CSR pages

Ricardo Iván Vieitez Parra — Tue, 17 Jan 2023 01:00:00 +0000

Progressive enhancement

Progressive enhancement is a design philosophy that consists in delivering as much functionality to users as possible, regardless of the manner they may access our system. This ensures that a baseline of content and functionality is available to all users, while a richer experience may be offered under the right conditions. For instance, the website for a newspaper may make the text content of its articles available to all users through HTML markup, while providing interactive media content for users with a capable browser.

Nowadays, client-side scripting is used in most websites to provide various levels of functionality. Frameworks like React, Angular and Vue.js allow developers to deliver highly interactive experiences, which in turn has made modern rich web applications feasible, such as spreadsheet applications that run completely in the browser. Because of many of the conveniences that they provide, these frameworks are also used for all sorts of websites and not just for those with complex interactivity.

The principles of progressive enhancement can be applied to all websites, no matter how they are built or what they do. For websites that are open to the public, progressive enhancement is essential to ensure the best possible experience for our users.

Progressively loading web applications

Server-side rendering vs. client-side rendering

A website can be server-side rendered, client-side rendered or use a combination of both approaches.

When an application is server-side rendered, the server delivers HTML markup with content that is ready for a user to consume. In contrast, an application that is client-side rendered constructs the document presented to users with the help of client-side scripts.

By default, applications built with modern frameworks will be rendered on the client side, whether the content is static or generated dynamically from external parameters (such as data returned from an API).

For example, a typical React application may have some HTML like this:

<!DOCTYPE html>
<html>

<head>
    <title>Example Page</title>
    <script
        crossorigin
        src="https://unpkg.com/react%4018/umd/react.production.min.js">
    </script>
    <script
        crossorigin
        src="https://unpkg.com/react-dom%4018/umd/react-dom.production.min.js">
    </script>
</head>

<body>
    <div id="root"></div>
    <script src="basic.js"></script>
</body>

</html>

With the corresponding rendering logic in basic.js, which could look like this:

const root = ReactDOM.createRoot(document.getElementById('root'))
root.render(React.createrrorElementement('div', null, 'Hello, World!'))

This simple example will show a page with the text Hello, World!.

Since in this example all of the rendering code depends on client-side scripting, users with an older browser, one that doesn't support scripts or one with scripts disabled will see a blank page.

The situation of having a blank page in certain browsers can be avoided by using server-side rendering. Depending on the framework in question, the approach can be different. For React, this can be accomplished with ReactDOMServer. Following the principles of progressive enhancement, server-side rendering should be used whenever possible or feasible to give users the possibility to use and interact with our page. The advantage of this approach is that client-side scripting can still be used to enhance interactivity and functionaliy if the client supports it. However, server-side rendering might not always be a viable option, depending on what our page is meant to do. For instance, a highly-interactive rich application, like an online image editor or game, may require scripting to provide an adequate experience beyond the capabilities of a statically-generated document.

Regardless of whether server-side rendering is a viable option for our page, we can follow some principles of progressive enhancement to improve the experience of all users.

`<noscript>`

Firstly, there is the <noscript> element, which presents content for browsers that do not support or process client-side scripts. We could use this tag to tell our users either that scripts are required and that they should use a different browser or to let them know that some functionality might not be available without scripting.

We could use the <noscript> to display a message like this:

<noscript>
    Scripting must be enabled to use this application.
</noscript>

Handling errors

A second consideration is that a client might support scripts but they could result in an error. For example, an older browser might not understand the syntax used in our scripts, like using const or let for variables, or it might be that we use some functionality that isn't implemented, such as promises.

While we could avoid some of these potential errors by other means (like feature detection), for robustness we should implement an error handler that lets users know that an unexpected error occurred so that they can take appropriate action, such as using a different browser or contacting support.

We can implement this with a separate script that implements an error event handler. The reason for using a separate script is that we can implement this handler with syntax and features that all browsers are likely to support.

For example, our error handler could look something like this:

onerror = function (e) {
  var errorElement = document.getElementById('error')
  var loadingElement = document.getElementById('loading')
  if (errorElement) {
    // Show some information about the error
    if (typeof e === 'string' && e.length) {
      errorElement.appendChild(document.createTextNode(': ' + e))
    }
    // Make the error element visible
    errorElement.style['display'] = 'block'
  }
  // Hide 'loading' message if an error occurred
  if (loadingElement) {
    loadingElement.style['display'] = 'none'
  }
  return false
}

For a corresponding HTML body with error and loading elements:

<div id="root">
    <noscript>
        Scripting must be enabled to use this application.
    </noscript>
    <div id="loading">
        Loading...
    </div>
    <div id="error" style="display:none">An error occurred</div>
</div>

This way, browsers without client-side scripting will display the message in the <noscript> element, while browsers with scripting will show a 'loading' message indicating that the application is not yet ready (alternatively, this can be the page contents when using server-side rendering) or an error message should some error occur.

`<script>` attributes

Generally, <script> elements block rendering. This means that the page won't be interactive or display content until scripts have loaded. Depending on factors like network speed, this can result in poor user experience because the page will appear slow without indication of what is happening.

HTML5 introduced the async and defer attributes to the <script> tag to address this issue. The async attribute specifies that the script should be fetched in parallel and executed as soon as its contents are available. The defer attribute is similar, except that it specifies that the script should be executed right after the document has been parsed, but before the DOMContentLoaded event is fired.

For scripts that should render the initial contents of the page, as we are discussing, the defer attribute is the most appropriate, since it'll allow us to progressively enhance the page when possible while still being able to provide some fallback content while scripts are being downloaded and executed.

We could use a helper function like this in a <script defer> element:

// Exceptions to throw
var InvalidOrUnsupportedStateError = function () {}

// Entry point
var browserOnLoad = function (handler) {
  if (['interactive', 'complete'].includes(document.readyState)) {
    // The page has already loaded and the 'DOMContentLoaded'
    // event has already fired
    // Call handler directly
    setTimeout(handler, 0)
  } else if (typeof document.addEventListener === 'function') {
    // 'DOMContentLoaded' has not yet fired
    // This is what we expect with <script defer>
    var listener = function () {
      if (typeof document.removeEventListener === 'function') {
        // Remove the event listener to avoid double firing
        document.removeEventListener('DOMContentLoaded', listener)
      }

      // Call handler on 'DOMContentLoaded'
      handler()
    }
    // Set an event listener on 'DOMContentLoaded'
    document.addEventListener('DOMContentLoaded', listener)
  } else {
    // The page has not fully loaded but addEventListener isn't
    // available. This shouldn't happen.
    throw new InvalidOrUnsupportedStateError()
  }
}

browserOnLoad(function () {
  // code that does client side rendering
})

Putting it all together

We can combine all of the techniques presented to load scripts in a way that progressively enhances our page load on each step: first, by presenting users with a message that scripting is necessary, then by displaying a loading message while the application gets ready (and an error message if something goes wrong), followed by a fully loaded application once all scripts have been executed.

First, we set up the HTML document:

<!DOCTYPE html>
<html>

<head>
    <title>Example Page</title>
    <script>
    onerror = function (e) {
        var errorElement = document.getElementById('error')
        var loadingElement = document.getElementById('loading')
        if (errorElement) {
            // Show some information about the error
            if (typeof e === 'string' && e.length) {
                errorElement.appendChild(
                    document.createTextNode(': ' + e)
                )
            }
            // Make the error element visible
            errorElement.style['display'] = 'block'
        }
        // Hide 'loading' message if an error occurred
        if (loadingElement) {
            loadingElement.style['display'] = 'none'
        }
        return false
    }
    </script>
    <script defer src="app.js"></script>
</head>

<body style="font-size:24pt;background-color:white;color:black">
    <div id="root">
        <noscript>
            Scripting must be enabled to use this
            application.
        </noscript>
        </div>
        <div id="loading" style="color:blue">
            Loading...
        </div>
        <div id="error" style="display:none;color:teal">
            An error occurred<!--
        --></div>
    </div>
</body>

</html>

Followed by the app.js script:

!function () {
  // Exceptions to throw
  var InvalidOrUnsupportedStateError = function () {}

  // Entry point
  var browserOnLoad = function (handler) {
    if (['interactive', 'complete'].includes(document.readyState)) {
      // The page has already loaded and the 'DOMContentLoaded'
      // event has already fired
      // Call handler directly
      setTimeout(handler, 0)
    } else if (typeof document.addEventListener === 'function') {
      // 'DOMContentLoaded' has not yet fired
      var listener = function () {
        if (typeof document.removeEventListener === 'function') {
          // Remove the event listener to avoid double firing
          document.removeEventListener('DOMContentLoaded', listener)
        }

        // Call handler on 'DOMContentLoaded'
        handler()
      }
      // Set an event listener on 'DOMContentLoaded'
      document.addEventListener('DOMContentLoaded', listener)
    } else {
      // The page has not fully loaded but addEventListener isn't
      // available. This shouldn't happen.
      throw new InvalidOrUnsupportedStateError()
    }
  }

  // Function to load dependency scripts
  // For simplicity, this assumes all scripts are independent
  // from each other
  var loadScripts = function (scripts) {
    return Promise.all(scripts.map(function (src) {
      return new Promise(
        function (resolve, reject) {
          var el = document.createElement('script')
          el.addEventListener('load', resolve)
          el.addEventListener('error', reject)
          el.src = src
          el.crossOrigin = 'anonymous'
          document.head.appendChild(el)
      })
    }))
  }

  browserOnLoad(function () {
    loadScripts([
      'https://unpkg.com/react%4018/umd/react.production.min.js',
      'https://unpkg.com/react-dom%4018/umd/react-dom.production.min.js'
    ]).then(
      function () {
        var root = ReactDOM.createRoot(document.getElementById('root'))
        root.render(React.createElement('div', null, 'Hello, World!'))
        onerror = null
      }).catch(function (e) { onerror(e.message || e) })
  })
}()

Effectively mitigating CSRF

Ricardo Iván Vieitez Parra — Mon, 02 Jan 2023 10:40:58 +0000

What CSRF is

Cross-Site Request Forgery (often shortened to CSRF or XSRF) is a type of attack in which an external site makes a request to another site on behalf of a user without consent. This attack often relies on there being an existing session on the target site, which the attacker hijacks for their own purposes. Traditionally, this attack also relies on cookies (classic CSRF), although a new variant of the attack, called Client-Side CSRF (CSCSRF), is possible on sites that use client-side scripting without appropriate input validation.

Although any site could potentially become a target for CSRF attacks and should therefore implement appropriate mitigations, some factors that can increase the likelihood of becoming a target are:

Having a large audience, where it is likely that users are logged in (for example, major social media sites)
Being a high-value sites where a successful attack is lucrative (for example, sites of financial institutions)
Offering subdomains to third parties (for example, some blogging platforms and shared hosting services). This is a high-risk scenario because some of the protections implemented in browsers to isolate origins may not work.

Examples

Classic CSRF: a concrete example

Cookies are used to store user sessions (or session identifiers) in a vast number of sites, and historically cookies are sent with all requests to a site, even those triggered from external sites.

Imagine the site bank.example, which works with user sessions. When a client logs in, the site sets a cookie like c with some session identifier. Although the session identifier alone is enough to make requests on behalf of a (logged in) client, it is sufficiently difficult to guess for an attacker, so that directly exploiting this mechanism is impractical. The site at bank.example offers transfer functionality, which works by submitting a form to the address bank.example/secure/transfer.do with the transfer amount and a destination account.

How could an attacker exploit this system to transfer funds to themselves? Imagine further that an attacker has compromised an unrelated site, vulnerablenewssite.example, which ideally (for the attacker) is a site that the bank's customers are likely to visit. Then the attacker can make a request from vulnerablenewssite.example to bank.example/secure/transfer.do with their account details by submitting a form. This request will include the users' SESSIONID cookie (which the attacker doesn't know) and result in the attacker stealing funds from the unsuspecting user.

Bonus classic CSRF example: logging out

Another example of a CSRF attack is terminating users' sessions. Many sites implement logging out by directing the user to a location like example.com/logout, which implements the logic necessary to invalidate a user session. An attacker could exploit this fact to terminate a session from an external site by loading this path (if the path also works with a simple GET request, this can be done trivially by embedding the logout path as an external resource in a multitude of ways, such as by using the link, img, object and iframe tags, to name a few).

Although in many cases this may not have many consequences besides annoyance (still, depending on the target site even a short-lived denial of service could be advantageous to an attacker, especially in combination with other attacks) at having to log in again, this illustrates two points:

That the request doesn't need to fully 'succeed' (i.e., probably loading an image from example.com/logout will not actually display an image). As long as the request is made, whether or not a valid resource is presented, an attack has potential to succeed.
Although this attack also works with POST requests by submitting a form, alloing GET requests for things that change the server state in some manner leaves the target site in a precarious position because it opens up more avenues for attack while reducing the number of possible mitigations discussed later.

Client-Side CSRF: a concrete example

Client-Side CSRF has similar consequences to classic CSRF but is carried out in a different way (through insecure input validation) that renders most defences against classic CSRF ineffective.

Consider the bank site from earlier. To mitigate classic CSRF attacks, they've stopped using cookies in favour of local storage and user actions are implemented with requests using client-side scripting. Because local storage can only be read by bank.example and because, unlike cookies, values in local storage are not sent along HTTP requests, a classic CSRF attack will fail.

However, imagine that now, in order to make a transfer, the URL is something like https://bank.example/secure/action.do#?destination=transfer.do&data=amount_100-payee_1234, for making a transfer of $100 to the account number 1234. Then, the page at https://bank.example/secure/action.do has a script that extracts the relevant data from the URI fragment (i.e., the part after #) and makes a request to https://bank.example/secure/transfer.do, which is deduced from these data.

Since all of these parameters are controlled by an attacker, several ways of exploiting this system exist. For example, an attacker could open a new browser window (e.g., through window.open) to the target URL or could trick the user into clicking an attacker-crafted link. Because the exploitation happens through client-side code on the target site, server-side or browser-side CSRF mitigations are ineffective.

CSRF mitigations

`SameSite` cookies

Historically, cookies set on a site are sent along all requests to that site, even for requests that originate from a different site. This is the mechanism that classic CSRF exploits, and, to mitigate this, RFC6265bis introduced the SameSite attribute, which is implemented in all major modern browsers.

The SameSite attribute may take three possible values:

None, which results in the historical behaviour of the cookie being sent with all requests;
Lax, which results in the cookie being sent only for requests either originating from the same site or navigation to the site (for example, as a result of following a link); and
Strict, which omits sending the cookie unless for requests originating from the same site (i.e., all external requests, including navigation will omit the cookie).

For most sites, SameSite=Lax offers a good balance between convenience and protection. This is the default behaviour in many major modern browsers (i.e., all cookies implicitly are SameSite=Lax unless the attribute is specified to be something else), with the notable exceptions at the time of writing being Firefox and Safari. In all other browsers, including legacy browsers that do not support the SameSite attribute, the default (or only) behaviour is that of SameSite=None.

As a first line of defence, all sensitive cookies should explicitly set the SameSite=Lax (or SameSite=Strict) attribute to guard off against classic CSRF and Secure attributes to guard against classic CSRF attacks. In addition, sensitive pages should use HTTPS and set the Secure attribute to protect against other attacks, as well as the HttpOnly attribute if the cookie does not need to be visible to client side scripts.

TLS 1.3 + `SameSite`

Because there is a nearly complete overlap between browsers that support TLS 1.3 and browsers that support SameSite, making the site available only through TLS 1.3 (or higher) can help ensure that SameSite is effective by preventing older browsers from making requests at all. Do note that this isn't an absolute guarantee, since older browsers may still access the site with some non-standard configurations.

CSRF tokens

The most common technique for CSRF protection, which works in older and newer browsers alike, consists of including some kind of token along with the request. This token needs to be difficult to guess for third parties, so that attempts at CSRF fail validation.

Double-submit cookies

This technique has the advantage of being stateless (i.e., the server does not need to maintain any state related to CSRF protection). It works by:

Setting a cookie on the client that has sufficient entropy (i.e., a long random value), and having all requests to resources that need to be protected include this value (or a value derived therefrom).
Upon receiving a request, requiring that the two values match

For example, upon visiting a site, it could set a the cookie _CSRF to the value 01234567-89ab-4000-8000-0123456789ab. Then, upon receiving requests that result in some sensitive action, the value 01234567-89ab-4000-8000-0123456789ab (or some derived form) is included as an additional field, which is checked to verify it matches the value included in the cookie. This additional field could, for example, take the form of a hidden form input field or a custom header.

Sites that may have untrusted subdomains or subpaths (for example, because they are under the control of third parties) should consider restricting access to cookies with appropriate use of the Domain and Path attributes. However, when this is not practical, an appropriate defence is setting the double-sent value to be a value derived from the CSRF cookie, in a way that only the target server can validate it. For example, the hidden field can be set to HMAC(_CSRF cookie, secret) or even HMAC(_CSRF cookie + path, secret).

Additional restrictions

To reduce the timeframe available to an attacker, the CSRF token cookie could be digitally-signed or integrity-protected and validity window set, with the token being rotated before it expires. This has the disadvantage of making validation slightly more demanding on the server as well as potential inconvenience to users when the token expires.

Synchronised tokens

As an alternative to double-submit cookies, the value used to derive the token can be stored server-side on the server as part of the session information. This allows for finer-grained control of the token and its validation, because it remains opaque to the client and any potential attackers. However, it can require more server resource and result in inconvenience to users when the token expires or is rotated.

Avoiding cookies

Since classic CSRF attacks rely on how cookies are sent in the HTTP protocol, a viable protection against attacks can be simply not using cookies, opting for local or session storage instead. This may be especially appropriate for sites which require client-side scripts to function and require or can have session identifiers or information available to client-side scripts.

CORS

Generally, classic CSRF attacks are not prevented by Cross-Origin Resource Sharing (CORS) because for compatibility reasons many cross-origin requests are allowed to go through without CORS checks, such as embedding external resources or submitting forms.

For applications that require client-side scripts, one way simple way of triggering CORS checks is by including a custom header or submitting payloads with a content-type header that is not allowed in HTML forms, such as application/json. The backend must in this case check that incoming requests have the expected headers as well as block cross-origin requests as appropriate.

When using CSFR tokens, it is imperative that CORS be set up correctly, or else a malicious actor could take advantage of an ineffective policy to retrieve the token value directly.

`Origin` and `Referer` headers validation

Browsers prevent spoofing the values of the Origin and Referer headers (although they do allow to omit the Referer header in some circumstances). The Referer header is by default included in all cross-origin requests while generally the Origin header is included in all CORS requests as well as POST requests.

One could protect against CSRF attacks by verifying that the value of the Origin header (when present) or else the origin in the Referer header (when present) matches an expected value, which usually should be the host same as that of the Host header.

In some situations, it could happen that neither the Origin nor the Referer headers be set, one such situation being a GET request caused by direct user navigation. When this happens, one might decide to either block (erring on the side of caution) or allow (erring on the side of availability) the incoming request. A good guideline might be blocking all non-GET / non-HEAD requests with neither header, while deciding on GET (and HEAD) on a case-by-case basis.

`Sec-*` headers validation

Many, but not all, modern web browsers implement various Sec--prefixed request headers, which can provide additional information that is helpful in preventing CSRF. In particular, the Sec-Fetch-Site having the value cross-site (different site altogether) or same-site (different subdomain) may be indicative of a CSRF attempt, and these requests may be blocked as appropriate.

Since not all browsers (notably, Safari) send this header along, requests lacking this header should be allowed through to prevent service disruption to legitimate users.

Built-in CSRF protection

If you're using some sort of web framework, chances are that it comes with CSRF protection out-of-the-box or that third-party modules exist that provide the functionality. Before implementing your own, you may want to familiarise yourself with the built-in protection to see if it covers your needs, since chances are that it will be a robust and well-tested solution that will save you time.

Protection against Client-Side CSRF

Unlike classic CSRF, client-side CSRF can be more challenging to protect against because rather than checking for the correctness of certain predetermined values, one needs to check for the well-formedness and correctness of application-dependent data.

Some useful guidelines against client-side CSRF can be:

When possible, avoid client-side actions that affect the application state or are potentially sensitive upon without prior user interaction. In other words:
- If possible, do not automatically make sensitive requests if the user has not interacted with the page (e.g., moving the mouse, clicking on something, typed something, scrolled, etc.)
When dynamic endpoints are used in a way that could be manipulated by a malicious actor (for example, as part of the URL), implement validation rules that are as strict as feasible for the application (for example, a whitelist or pattern checking).
When the data are sent as dynamic client-side requests that could be manipulated by a malicious actor (for example, as part of the URL), implement validation rules that are as strict as feasible for the application (for example, schema validation or data signing).

Other defence-in-depth measures

Content Security Policy (CSP)

While not effective against all forms of client-side CSRF (specifically, direct linking or somehow making a user open a link), a well-crafted content security policy can close some attack venues. Specifically:

frame-ancestors (and the header X-Frame-Options) can be used to prevent embedding of our site into a different site, thus forcing an attacker to make the user open a link, which may be more obvious than, e.g., a hidden <iframe>.
connect-src to restrict the origins our site is allowed to connect to, thus preventing leaking data to an attacker-controlled site
img-src, script-src, default-src, etc.: similar to connect-src, if we might leak information through dynamically inserted resources.

Apeleg join the W3C

Ricardo Iván Vieitez Parra — Wed, 19 Oct 2022 00:00:00 +0000

In July, we announced our partnership with Smidyo to provide Vector Express. We view this as a strategic move to support our mission of facilitating interoperability and producing value through automation.

Today, as part of our long-standing commitment to open standards and furthering innovation in the web, we are pleased to announce our becoming a W3C member.

The World Wide Web Consortium (W3C) was founded in 1994 and is one of the chief international standard organisations for the web, including the ubiquitous HTML, CSS and SVG standards. The organisation is composed of over 470 members and is the arena for discussion and development to the latest innovations in the web space, such as WebAuthn, a standard for superseding password authentication, the Web Cryptography API, an API for accessing cryptographic operations on web browsers and the WebGPU Shading Language, a standard shader language that allows for bringing rich user experiences such as games and advanced animations to the web.

We strongly believe in interoperability, open standards, collaboration and transparency and as a W3C member, Apeleg will be able to stay informed of new standards and advancements and participate in the various working groups to shape the future of the web.

The areas that are of special interest to us are accessibility, HTML, service workers and web authentication, as these closely relate to our areas of expertise and the solutions that we commonly work on. We plan on bringing our experience to the table to aid in producing high-quality standards as well as gain insights that can help us deliver even better services and solutions to our customers and the Internet community in general.

As one of the few Norwegian-registered W3C members (not including members' subsidiaries), we feel a strong responsibility to ensure that we can represent the voices of our Norwegian customers and foster innovation and accessible standards worldwide as well as in Norway.

Modern and robust hotlink protection in 2022

Ricardo Iván Vieitez Parra — Mon, 10 Oct 2022 00:00:00 +0000

What a hotlink is

Hotlinking refers to the practice of third-party web properties loading resources (most commonly images) directly from your own server.

For example, if you operate the website yourbusiness.example, you may have an image at https://yourbusiness.example/infographic.png that you use within your website. A hotlink is when an unrelated property (for example, the site anotherbusiness.test) embeds that image directly on their website by reference to your server, for example using the following HTML code:

<img
    src=https://yourbusiness.example/infographic.png
    alt="Widget purchases per capita"
>

Unauthorised hotlinks are generally undesirable, not only because they can facilitate reproducing your content without permission but also because, since the resources are being loaded directly from your server, they can burden you with additional server costs in computing and bandwidth.

TL;DR: If your website all your resources are in the same domain, add the Cross-Origin-Resource-Policy: same-site response header to your resources. If you use a CDN or serve some resources from an external domain, add the Cross-Origin-Resource-Policy: same-origin and Access-Control-Allow-Origin: https://yourbusiness.example response headers to your (external) resources and force a CORS request by using the crossorigin attribute.

Older approach to hotlink protection

Hotlink protection has historically relied on the HTTP Referer header, which indicates the source of the document loading the resource.

When a request is made to https://yourbusiness.example/infographic.png from your website, this referrer header would look something like Referer: https://yourbusiness.example/statistics.html, whereas when it's embedded in a third-party website, it might look like Referer: https://anotherbusiness.test/widgets-consumption/.

This approach is usually implemented in various webservers as follows.

Example configuration for `Referer`-based protection

Apache: `.htaccess`

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^https?://(.+\.)?yourbusiness\.example [NC]
RewriteRule \.(jpe?g|png|gif)$ - [NC,F,L]

nginx

location ~ \.(jpe?g|png|gif)$ {
    valid_referers
        none blocked
        server_names
            *.yourbusiness.example
            yourbusiness.example;

    if ($invalid_referer) {
        return 403;
    }
}

Microsoft IIS

<rule
    name="Hotlinking Prevention"
    stopProcessing="true"
>
    <match
        url=".*\.(jpe?g|png|gif)"
    />
    <conditions>
        <add
            input="{HTTP_REFERER}"
            pattern="^$"
            negate="true" />
        <add
            input="{HTTP_REFERER}"
            pattern="^https?://(.+\.)?yourbusiness\.example/.*$"
            negate="true" />
    </conditions>
    <action
        type="CustomResponse"
        statusCode="403"
    />
</rule>

Issues with `Referer`-based protection

Hotlink protection using the Referer has a number of issues that make it inappropriate in many scenarios.

Hotlinking vs direct linking

Perhaps the most compelling argument against relying on the Referer server is that doing so also breaks regular direct links, which may not always be desirable. This is because the Referer header is not only sent when loading resources but also when a navigation action happens. Hence, a citation like From YourBusiness' <a href=https://yourbusiness.example/infographic.png>widget purchases per capita</a> have increased steadily over the last century would result in a Referer header being sent from an external domain, which will trigger hotlink protection and result in blocked content. This is in most cases a bad user experience.

`Referer` omission

Another reason why this way of protection is ineffective is that the Referer header is not sent along requests in all cases.

One common case is that generally resources loaded from a plain HTTP location (i.e., with the URL starting with http://) will not include a Referer when they are being loaded from an HTTPS site. Therefore, an image loaded from http://yourbusiness.example/infographic.png at https://example.com will typically not have a Referer header.

Although the above scenario is not as relevant nowadays as most sites use HTTPS, that is just one example of how the Referer header can be omitted. A much more relevant argument is that the embedding site is in full control of whether a Referer header is sent along, which is as simple as including the referrerpolicy="no-referrer" attribute to the img tag.

It may be tempting to address this issue by requiring that a Referer header be present. However, there are other reasons why a Referer header may not be sent, such as direct navigation by a user, browser configuration and extensions that block this header or proxies that remove it. Thus, simply blocking incoming requests missing this header could result in poor user experience for some users, as your page will look 'broken' since resources won't load.

Additional processing

A third disadvantage of Referer-based protection is that it's dynamic by nature, meaning that every incoming request needs to be evaluated by your server to check whether the Referer contains an allowed value. This is a relatively small concern, but it introduces a tiny amount of latency to the response and results in more complex cache management.

As the web is moving more and more towards static or pre-rendered content and serverless platforms, effective protection that involves as little server logic as possible is ideal.

A more robust approach

Web standards and browsers have come a long way in the last few decades, and all of the tools for effective and robust protection against hotlinking in the most common scenarios. Specifically, developments in the fetch standard regarding cross-origin requests arm us with request and response headers that we can use to implement browser-enforced hotlink protection.

Some relevant headers

`Origin`

The Origin HTTP request header is in many respects similar to the Referer header with some enhancements that make it more suitable for requests across different web properties.

One key difference between Referer and Origin is that the former typically is a full URL (for example, https://www.example.com/page/) whilst the latter is always just an origin, or the first part of the URL, like https://www.example.com.

The Origin request header is sent for requests with methods other than GET or HEAD, or for requests that are explicitly marked as cross-origin (i.e., from one site to another).

`Cross-Origin-Resource-Policy`

The Cross-Origin-Resource-Policy HTTP response header is used to define a policy for cross-origin requests made in no-cors mode.

In practical terms, this header alone is sufficient in most cases for effective hotlink protection.

Cross-Origin-Resource-Policy can take one of three values: same-origin, same-site or cross-origin. All we need to do is include the header Cross-Origin-Resource-Policy with either of same-origin or same-site, and embedding of your resources by external websites will be blocked.

Difference between `same-origin` and `same-site`

The value same-origin specifies stricter policy than same-site.

Consider the site at https://example.com and the resources https://example.com/a.png and https://images.example.com/b.png. While both resources are part of the same site (i.e., example.com), each resource has a different origin: https://example.com and https://images.example.com, respectively. While both same-origin and same-site will allow for the site to use the first resource, only same-site will allow it to embed the second resource, as the origins are different.

`Access-Control-Allow-Origin`

The Access-Control-Allow-Origin HTTP response header is relevant for so-called CORS requests, which are requests that include an Origin header. Furthermore, the CORS protocol defines some versatile mechanisms that allow sites to define policies for cross-origin resource sharing.

Normally, most resources that we would like to protect against hotlinking (for example, images) are not loaded using CORS requests. However, CORS requests can be made explicitly by including the crossorigin or crossorigin="anonymous" attribute to the resource tag, for example like this: <img crossorigin alt=Example src=sample.jpg>.

The Access-Control-Allow-Origin instructs the browser that a certain origin actually allowed to make a cross-origin request, and it should take either of two values: *, indicating that the resource can be requested by all origins, or the value of the Origin header included with the request. Other values or absence of this header tell the browser that the request is not allowed in the context of the CORS protocol.

Hotlink protection for sites using a single origin

Many smaller sites serve all of their content from a single origin. For example, if you use WordPress, you may have the site https://yourbusiness.example and have most of your images under the https://yourbusiness.example/wp-content/uploads directory.

For this simple case, an effective way to implement hotlink protection is to include the header Cross-Origin-Resource-Policy: same-origin along with your responses, and this will prevent hotlinking by any other sites.

It is important that the Access-Control-Allow-Origin header not be sent, or if it is, that it be set to the origin of your site (e.g., https://yourbusiness.example). Sending Access-Control-Allow-Origin with any other value, especially * or replying back with the Origin header included in the request will allow hotlinking if a CORS request is made.

Configuration

Apache `.htaccess`

This policy can be implemented in Apache by using the .htaccess file (or alternatively the main configuration file) with something along these lines:

<FilesMatch "\.(jpe?g|png|gif)$">
    <IfModule mod_headers.c>
        Header set Cross-Origin-Resource-Policy "same-origin"
    </IfModule>
</FilesMatch>

The FilesMatch directive can be adjusted as needed depending on the files that require hotlink protection. Because this technique does not have many of the limitations of the Referer-based one, it is even possible to skip this check and include the header with all responses.

nginx

In the relevant server block, this policy can be implemented as follows, adjusting the location part as needed:

location ~ \.(jpe?g|png|gif)$ {
    add_header cross-origin-resource-policy same-origin;
}

Hotlink protection for sites using subdomains

Oftentimes sites serve resources from a subdomain. For instance, the main site could be available at https://yourbusiness.example while images and other static resources are hosted at https://static.yourbusiness.example. It may also be the case that certain parts of the site reside in a subdomain (for example, https://store.yourbusiness.example) and subdomains share resources.

For these scenarios, hotlink protection can still use the Cross-Origin-Resource-Policy header, except that the same-site value (instead of same-origin) is likely the most appropriate choice.

Hotlink protection for sites using external resources

It is increasingly common for sites to use CDNs to serve static resources, which often are accessed through a separate domain. For example, https://yourbusiness.example might load images from the origin https://yourbusiness.cdnprovider.example.

Hotlink protection in this scenario is slightly more involved than when all resources are part of the same site because then requests are by definition cross-origin and cross-site and CORS policies are only enforced for requests that use the CORS protocol.

While want to load resources from a different origin, we can't rely on the Cross-Origin-Resource-Policy alone. This is because the only value that would seem appropriate is cross-origin, which does not provide any hotlink protection whatsoever: adding the header Cross-Origin-Resource-Policy: cross-origin to CDN responses would result in anyone being able to load the resource in question from any origin, which is exactly the situation that we are trying to avoid.

Fortunately, we can force requests to use the CORS protocol and this way have more granular control over whom has access.

Counter-intuitively, an appropriate value for the Cross-Origin-Resource-Policy header in CDN responses is same-origin. By using this value, non-CORS requests to the CDN will fail, which leaves only CORS requests as a way to load resources, which equips us with more granular ways of defining an access policy through the Access-Control-Allow-Origin header.

For the CDN or external domain case, hence we need three elements for hotlink protection:

Cross-Origin-Resource-Policy: same-origin in the external resource response. This blocks non-CORS requests
Access-Control-Allow-Origin: https://yourbusiness.example in the external resource response (where https://yourbusiness.example is the origin the resource will be loaded from). This tells the browser that the response is intended for use by this origin and this origin only.
crossorigin (or crossorigin="anonymous") attribute in the document referencing the resource. This means that <img src=https://yourbusiness.cdnprovider.example/image.webp> becomes <img crossorigin src=https://yourbusiness.cdnprovider.example/image.webp>.

This approach will effectively block hotlinking from origins other than https://yourbusiness.example, which is exactly what we are looking after. As an added bonus, this same configuration also works for the single-origin case discussed earlier and, with the caveats that follow, for the single-site case.

Advantages, caveats and limitations

The solution presented is simple (requires adding a few HTTP headers to the response and a small change to the HTML markup), robust, has good browser support¹ and because the policy is enforced by the browser itself, it degrades gracefully, meaning that the resources will still load normally in the few browsers still in use that don't support these headers.

Moreover, this solution for hotlink protection is in many cases stateless meaning that no conditional logic is required in the server, as the values for the headers are predetermined in advance.

The main caveat that applies, which is most relevant to this sites that make use of multiple domains or subdomains, and

load content served from an external domain (such as a CDN), or
use the crossorigin attribute

Since the Access-Control-Allow-Origin can only contain a single origin (and there is no syntax for allowing subdomains), these sites will require some server-side logic give an appropriate value to the Access-Control-Allow-Origin header. The Access-Control-Allow-Origin must contain the value of the Origin sent in the original request, and this value must be validated first to ensure that it's an allowed value. Moreover, it's likely that these sites will need to add Access-Control-Allow-Origin to their Vary response header to allow for proper caching.

Direct linking protection

In certain scenarios, it may be desirable to prevent direct access or direct linking to certain resources. The new Sec- headers allow for controlling these actions.

Images and other embeddable content

The Sec-Fetch-Dest request header is useful in these cases to have fine-grained control over when a browser is allowed to download a certain resource.

For example, to prevent direct access to an image, you can check if this header is set to image. To allow direct access and embedding as an image, but not other uses (for example, a fetch or XHR request), only the document and image values could be allowed. To block direct access, just image would be allowed.

Note that this header is not yet supported by all browsers (most notably Safari), so it's advised that, if you decide to make decisions based on this header, you allow requests that do not have it set.

External direct links

You may want to prevent or discourage external websites directly linking to certain files on your site (for instance, a large PDF file) while still allowing your users to access this content by directly linking files internally.

The Sec-Fetch-Site request header can be helpful for taking different actions based on where a user came from. For example, if this header is set to cross-site, then you might decide to issue a 303 See Other redirect to a page discussing the resource in question.

Like Sec-Fetch-Dest, Sec-Fetch-Site as of yet does not have wide enough support and may not be present in all requests. It's recommended to allow through normally requests without this header.

As of the time of writing, Cross-Origin-Resource-Policy is supported by over 93% of global users. ↩

Types of Execution Environments, Attestation and SGX

Ricardo Iván Vieitez Parra — Tue, 06 Sep 2022 00:45:55 +0000

Execution Environments

OMTP refer to an execution environment as the combination of hardware and software components that can be used to execute and support applications. A typical execution environment consists of a processing unit, memory, input and output (I/O) ports and an operating system. Because application execution requires an execution environment, applications are ultimately limited by any constraints placed onto them by their execution environment. This means, for example, that applications can use only as much memory as their execution environment allows, and that they can only perform tasks supported and allowed by their execution environment.

Rich Execution Environments (REEs)

Traditionally, computing is effected in execution environments that not only permit the loading and execution of arbitrary programs but may also themselves be manipulated in arbitrary ways, such as by attaching a hardware debugger to observe and change the internal state. As it is impossible for such environments to make any verifiable assertions as to their state, they are inherently untrustworthy. Alternatively, the term Rich Execution Environments (REEs) is sometimes also used to refer to this type of execution environments, such as by GlobalPlatform.

To understand the difficulty of establishing trust in REEs, consider the example of a messaging application that requires uploading a person's contacts to its developers' systems for the purpose of allowing that person to discover contacts already using said application. While this can be a valuable feature, the main disadvantage of such a system is that the users do not have a means of auditing or controlling how the information about their contacts will be used. While honest developers might use this information only for the stated purpose of contact discovery, unscrupulous developers might instead collect this information, aggregate it and then surreptitiously sell social graphs in the open market. Because at this point a sceptical user has only the word of the application developers that contact information is used for the stated purpose and only for the stated purpose, convincing prospective sceptical customers to upload their contacts requires demonstrating that their execution environment is such that the contacts uploaded cannot possibly be misused. However, providing such proof is impossible with an REE because a malicious developer could alter the execution environment in any number of ways such that contact information is exfiltrated.

For example, suppose that the application developers hire an independent, objective and trusted auditor to inspect their systems for compliance to their stated policy and that not only does the auditor find the systems compliant, but also inspects the systems on a regular basis to ensure ongoing compliance. If the developers had malicious intent, they could attempt to fool the auditor (and their users) in various ways. A crude approach could be changing their server configuration files after each audit, but more sophisticated techniques are possible, such as modifying the hardware in their servers to exfiltrate the desired information.

Furthermore, even if the developers were honest and trustworthy, the nature of REEs results in the contact-use policy still being difficult to enforce by technical means because other actors involved might still interfere in ways that result in the misuse of contact information, such as the developers' employees and suppliers. Some of these breaches could include an employee reading the server's memory directly, a malicious operating system transmitting data on user activity, or even an unauthorised third-party accessing the server and modifying the software being executed.

Trusted Execution Environments (TEEs)

TEEs address the inherent untrustworthiness of REEs by providing an isolated environment for user applications, the properties of which are well-known. Continuing the previous example of contact discovery, consider messaging application developers willing to go to great lengths to substantiate their contact use policy. They may accomplish this by limiting the execution environment to prevent arbitrary state manipulation and by providing a means of establishing a verifiable chain of trust. One of the ways they can accomplish this is to configure the contact discovery service and then hand it over to a trusted third party (trusted third party in this context means an entity that is trusted to be honest by the application users). The trusted third party then proceeds to thoroughly audit the entire execution environment (including both hardware and software components) and publish the relevant findings. Then, the entire machine is secured¹ so that the execution environment cannot be modified further. When a user uploads their contact information to the contact discovery service, they also consult the trusted third party as to whether the service they are connecting to corresponds to the audited system.

Intuitively, this scheme provides a high degree of confidence that the system a user is connecting to corresponds to one that behaves as per the published audit findings providing the user trusts the auditor. However, the example described is not practical for several reasons. Firstly, the increasing complexity of computer software and hardware and the interactions between them make it extremely difficult for an auditor to be decisively confident that a system behaves in a certain way, as doing so would require evaluating each of the components that comprise the system. Secondly, the example relies on an idealised notion of securing a machine that is not practical; physical access to the machine may be needed to effect repairs and other maintenance tasks, logical access may be needed to perform software updates. Thirdly, because of the restrictions placed on the entire system, any modifications would require going over the impractical processes of a full audit and securing once again.

A more practical approach would be to isolate certain critical system components (such as software for contact discovery) from the rest of the system, and then audit these critical components. Hardware manufacturers are in a unique position to provide such isolated execution environments that allow users to restrict the interactions between sensitive components from the rest of the system, and that facilitate auditing by providing just enough functionality for implementing these critical operations. In GlobalPlaform terminology, these environments are known as Trusted Execution Environments (TEEs) and are defined as follows by M. Sabt et al.:

[A] Trusted Execution Environment (TEE) is a tamper-resistant processing environment that runs on a separation kernel. It guarantees the authenticity of the executed code, the integrity of the runtime states (e.g. CPU registers, memory and sensitive I/O), and the confidentiality of its code, data and runtime states stored on a persistent memory. In addition, it shall be able to provide a remote attestation that proves its trustworthiness for third-parties. The content of TEE is not static; it can be securely updated. The TEE resists against all software attacks as well as the physical attacks performed on the main memory of the system. Attacks performed by exploiting backdoor security flaws are not possible.

TEEs have numerous privacy-enhancing applications that may benefit users. One of them is, as discussed earlier, private contact discovery; the Signal application uses a contact discovery service enhanced using Intel SGX, a TEE technology, to protect its users' privacy. A similar application of TEEs is performing malware analysis in a remote cloud service, so that the service may not identify users by the contents of their devices, such as the applications they have installed, especially important as 98.93% of users may be uniquely identified by the list of applications they have installed.

Sealing and Unsealing

In the context of TEEs, sealing and unsealing refer to cryptographic primitives providing authenticated encryption and decryption, respectively, with a secret that exists only within the TEE to securely store some of the TEE state in untrusted storage. Some TEE platforms, such as Intel SGX, provide a mechanism for deriving sealing secrets deterministically based on the TEE identity, which allow for unsealing information in different instances of the same TEE. Sealing can be useful to reduce the state information that is needed within a TEE, such as when the TEE is memory-constrained or for persistence when the TEE state may be lost (such as due to a power failure). A high-level overview of the sealing primitive is shown below.

Attestation

A crucial component of TEEs is their providing a protocol to provide attestations, verifiable assertions about the state or properties of the TEE; in particular, they can prove the presence of a particular TEE. Note that attestations are indispensable for TEEs, as without them it is impossible for an observer, including local observers with full access to a system, to differentiate between a TEE and another execution environment, whether it be another TEE or an REE. Hence, although it is indeed possible to use TEEs without using attestations (for example, for their isolation properties), they provide weaker security guarantees.

The trusted part of a TEE stems from the concept of a root of trust, that is, the notion that TEEs are trusted because some entity makes assertions about the properties of the TEE, and that these assertions are believed because the entities verifying the assertion trust the one making them. Trust in attestations entails both trusting the processing environment used in the TEE to meet certain specifications (a static component) and the specific state or application in the TEE (a semi-dynamic component).

Attestation Example

Although various attestation schemes are possible, it may be helpful to consider a simple attestation model to understand the different parts that make an attestation. A TEE manufacturer produces TEE hardware in which each device δ is assigned a unique per-device asymmetric key pair (δₚ, δₛ) stored in the device. The public part of this key, δₚ, is signed with the device manufacturer's key pair (𝙼ₚ, 𝙼ₛ) to produce a signature σ. When a TEE is initialised, the secret key (δₛ) is used to sign the TEE state 𝑺² to produce a signature Σ. An attestation could consist of the tuple (𝙼ₚ, Σ, δₚ, Σ, 𝑺). This would allow anyone to verify that the state 𝑺 was present in the TEE δ. In particular, Σ establishes trust on the device δ because the manufacturer is trusted; then, the trust on the presence of state 𝑺 is possible because the device δ is trusted; lastly, if the state 𝑺 corresponds to a well-known and application, the entire TEE can be trusted to execute this application according to the device specifications.

Intel Software Guard Extensions

Intel Software Guard Extensions (SGX) is an extension to the Intel architecture, available since the Skylake microarchitecture, that provides a TEE for user software in the form of isolated software containers known as enclaves. SGX has the stated goal of reducing application attack surface and protecting applications from potentially malicious privileged code, such as the operating system and a hypervisor. An SGX enclave is a protected region within the memory space of an application, which after initialisation can only be accessed by software resident in the enclave. Except where noted, the background information on the operation of SGX enclaves in this section is based on the Intel® 64 and IA-32 Architectures Software Developer's Manual.

Enclave Properties

Enclave Signing

Each SGX enclave is required to be signed with a 3072-bit RSA key, along with including identifiers of the enclaveprogram and its version. The public RSA key is used to identify the enclave developer, and a 256-bit identifier is derived from measuring this public key, which is called MRSIGNER. By signing several enclaves with the same key, some convenience features are available, such as the ability to derive common sealing keys for exchanging information between enclaves made by the same developer.

Debug and Production Modes

SGX enclaves exist in two flavours, which are determined at build time: debug and production enclaves. These two behave identically, except that external processes may use the EDBGRD and EDBGWR instructions for reading from and writing to debug enclave memory, and may similarly set breakpoints and traps within debug enclaves. As a result, the state of debug enclaves can be arbitrarily compromised and they are not TEEs. While any SGX-capable processor may instantiate debug enclaves, only enclaves signed with a key whitelisted by Intel may be executed in production mode.

Enclave Identity

The initial enclave state is measured upon initialisation by applying a hash function on every memory assigned to the enclave, which includes its code and data. This providing an enclave identity called MRENCLAVE, which is represented as a 256-bit value. This identity, along with MRSIGNER, represents the enclave identity. Both measurements may be used to derive enclave- or signer-specific keys (for example, using the EGETKEY instruction) that depend on a processor-unique secret.

Execution Model

An enclave may be accessed by using the EENTER instruction, which must point to a well-known entry point in the enclave, each of which is called an ecall. Once inside an enclave, the enclave may transfer control back to the calling process by using the EEXIT instruction or its execution may be interrupted by an Asynchronous Exit Event (AEX), caused by external events (such as interrupts) that occur while an enclave is being executed. If an enclave execution is executed as a result of an AEX, the enclave execution may be resumed using the ERESUME instruction. An enclave may call EEXIT to return once it has finished its current task, or to perform an ocall, that is, to execute code outside the enclave, such as a system call. In the latter case, the function performing the ocall may not have finished executing and may expect the called code to return by following some convention to call back into the enclave.

Memory Model

The processor manages access to any memory pages assigned to SGX enclaves (referred to as trusted memory) and triggers page faults for unauthorised accesses, and any memory pages swapped to disk are sealed to provide confidentiality and integrity, even from privileged processes. By contrast, memory pages not assigned to enclaves (referred to as untrusted memory), do not offer these protections any may be arbitrarily manipulated by the operating system or other processes.

Attestation Model

SGX provides local attestations, which depend upon a processor-unique secret and can only be verified by enclaves executing on the same processor, and remote attestations, which can be verified by anyone, and do not require an enclave for verification.

Local Attestations

Any SGX enclave may produce a report that can be verified by other enclaves executed on the same processor, referred to as local attestations. These attestations enable secure information exchange between enclaves, and may be used both for synchronous communication (e.g., for establishing a secure channel between enclaves) and asynchronous communication (e.g., for storing information that may be later used by another enclave). The local attestation process is illustrated in the figure below.

Generation

Local attestations are generated using the EREPORT instruction, which takes two inputs, TARGETINFO and REPORTDATA, and produces a report as an output. TARGETINFO identifies the enclave intended to verify the generated report and includes the target enclave MRENCLAVE as well as other attributes, such as whether it is a debug enclave. REPORTDATA is a 512-bit data structure that may contain arbitrary information intended to be verified by the target enclave. The generated report is authenticated with a symmetric key derived from the provided target MRENCLAVE and contains REPORTDATA and information about the attesting enclave, including MRENCLAVE, MRSIGNER and other attributes, such as whether it was a debug enclave.

Verification

Local attestations can only be verified by the target enclave, which must be executed in the same processor that was used for generating the report. The verification process consists of deriving a report key using the EGETKEY instruction and verifying that the MAC attached to the report is valid.

Remote Attestations

In addition to local attestations, SGX supports the generation of remote attestations. Unlike local attestations, verifying remote attestations does not require the use of an enclave nor is it limited to the same processor.

Generation

The process for generating a remote attestation is similar to that for local attestations but involves some additional steps. A remote attestation begins with a report made for an Intel-provisioned quoting enclave (QE). The quoting enclave then receives and verifies the report, and converts the report into a quote, which is signed with the QE private key.

The quoting enclave (QE) is provisioned by Intel as part of the Intel SGX Platform Software (PSW). The QE is tasked with verifying a local attestation report (described earlier) and using its public key to sign the report, which can be verified later. The signature uses a group signature scheme called Intel Enhanced Privacy ID (EPID) to ensure that a signature cannot be linked to a particular signer, but just to a member of the group.

Verification

Remote attestation quotes may be verified by validating whether the signature they contain is valid, which can be done by using either an EPID public key certificate or an attestation verification service, such as the Intel Attestation Service (IAS).

Plausible Deniability

The ISO non-repudiation framework states that the goal of non-repudiation is ‛to collect, maintain, make available and validate irrefutable evidence concerning a claimed event or action in order to resolve disputes about the occurrence or non-occurrence of the event or action’.

Plausible deniability is a concept closely related to non-repudiation in that it has the opposite stated goal: it is desired that any evidence regarding a disputed event be plausibly refutable. In particular, a protocol executed between two entities, Alice and Bob, and observed by an external observer Eve is plausibly deniable if its execution does not generate any evidence that can be used to irrefutably demonstrate Alice's or Bob's involvement. Note that this definition permits mutual authentication of Alice and Bob insofar as this mutual authentication is non-transferrable (i.e., it has no meaning for a third party).

Following Y. Dodis et al. and M. di Raimondo et al., we can model deniability in the paradigm of simulation. For this, we assume an arbitrary protocol π in which there is a sender 𝐴, a receiver 𝐵, an informant 𝐸 who observes the protocol execution, a misinformer 𝑀 who does not observe the protocol run but executes a simulated protocol π* and a judge 𝐽 who will rule at the end of the protocol whether communication took place between 𝐴 and 𝐵. The protocol π is deniable if 𝐽 is not able to distinguish between 𝐸 and 𝑀.

Authentication

Authentication was presented earlier in the context of authenticated encryption as a cryptographic process conferring integrity. In the context of communications, authentication has a broader meaning that encompasses the entire communication, including provenance. Generally speaking, for a message ℳ, the authenticator 𝓐 allows a verifier 𝑉 to assess not only whether the message ℳ was tampered with but also whether it originated from the expected party, that is that ℳ was sent by some party 𝐴 and not from an unauthorised party 𝖢. Although authentication is useful to verify that messages are genuine, the way that ℳ and 𝓐 are tied together, possibly to a sender 𝐴, may reveal information to parties external to the protocol as to 𝐴's involvement.

Working in the deniability framework set forth above, authentication is deniable if it does not give the informant 𝐸 any advantage over the misinformant 𝑀 in convincing the judge. We will examine three different authentication schemes and see how deniability is affected.

Scenario 1: No authentication

In this case, the protocol π provides no authentication. Note that it is impossible for the judge to tell which messages are genuine (meaning that the misinformant is able to forge messages that are indistinguishable from any real messages sent by the sender). This protocol is therefore deniable.

Scenario 2: Digital signatures

In this case, the protocol π uses digital signatures for authentication. The sender has a key pair (𝐈ₐ, 𝐢ₐ), where 𝐢ₐ is used for verification and is public and 𝐈ₐ is used for generating an authenticator and is a secret known only by the sender. Assuming that generating a valid authenticator without knowledge of 𝐈ₐ is infeasible, the judge can now distinguish the informant from the misinformant because the latter is unable to furnish messages that validate correctly.

Scenario 3: Symmetric authentication

In this case, the protocol π uses symmetric authentication, such as that provided by AES-GCM. The sender and the receiver agree on a key 𝐾 and the sender transmits messages encrypted and authenticated under this key, which are verified and decrypted by the receiver using the same key. Note that because 𝐾 is a shared secret, the receiver (or anyone who knows 𝐾) can forge messages that are indistinguishable from genuine messages from the sender. Because forgeries are possible, the judge cannot distinguish the informant from the misinformant.

Securing the machine refers in this case to applying a combination of physical and technical tamper-evident seals such that any modifications to the execution environment be noticed if not completely prevented. As an example, administrator-level software access could be disabled, the entire software stack could be placed in read-only memory and the entire machine could be encased in concrete and then shipped to a secure location. ↩
For the purposes of this example, the state 𝑺 may be assumed to contain a nonce, a timestamp or some other similar measure that assures freshness. ↩

Benchmarking in C (for x86 and x64)

Ricardo Iván Vieitez Parra — Mon, 05 Sep 2022 19:43:51 +0000

Benchmarks are extremely useful to see how performant some code or operation is and a requirement for any empirical decision making. After all, how can we know with any certainty if some library is faster than another one without testing?

The basics of benchmarking

At its core, benchmarking is quite a simple idea: we want to know how long a certain operation takes. Some simple (and naïve) pseudocode representing this can look like this:

start <- getCurrentTime()
runOperation()
end <- getCurrentTime()

print("Operation took: ", end - start)

Astute readers will notice a few issues with the above snippet. Let's go over some of these.

Operation time

As with any real-world operation, how long runOperation() takes is variable, and by having a single measurement we risk that it may not be representative. We can mitigate this by having several measurements and computing statistics, such as the average time.

A second and related issue is that in many applications there is a warm-up, meaning that the first few times runOperation() is executed it may take longer than subsequent executions, and that, because the operation is executed repeatedly, the time it takes after warming up may be more representative.

Time measurements

Another consideration is how exactly getCurrentTime() works, as we need to understand what we are measuring. Ideally, we are after time measurements that don't interfere with runOperation(), that are high-resolution (so that our measurements are accurate) and that are monotonic.

Out of order executions

A further complication is that just because we wrote our code to execute runOperation() in between getCurrentTime() calls, it doesn't mean that the the code will actually be executed in this manner, because processors often implement out-of-order execution. With benchmarks, we want time measurements to be executed exactly as specified and not while runOperation() is running.

Towards an implementation

Most modern operating systems provide some sort of abstraction for time measurements, such as clock_gettime(2) in POSIX systems or QueryPerformanceCounter in Windows systems. While these abstractions can be useful, it is possible to use native instructions to run accurate benchmarks without relying on any system or library calls.

In the x86 (and the nowadays more common 64-bit x86-64 aka x64 aka amd64) architecture, we can use the rdtsc instruction to read the Time Stamp Counter (TSC) register. If we read the TSC before and after executing our code of interest, we can determine how many CPU cycles elapsed, which we can use for comparisons.

Using the TSC

We can see how the TSC can be used for benchmarks in Gabriele Paoloni's white paper. This white paper presents a good introduction to some of the challenges discussed when carrying out measurements and a viable approach for an implementation.

The basic approach can be summarised as follows:

CPUID ; serialise
RDTSC ; read the clock
MOV start_cycles_high, edx
MOV start_cycles_low, eax
; Call the function to benchmark here
RDTSCP
MOV end_cycles_high, edx
MOV end_cycles_low, eax

Note that cpuid is used for serialisation, i.e., ensuring that all previous instructions before the benchmarking code have been executed. In addition, rdtscp is called for the end measurement. This is because, unlike rdtsc, rdtscp waits for all previous instructions to finish executing before reading the counter. The reason why we need cpuid when starting the benchmark and we can't just use rdtscp is that we also want the function to benchmark to start executing after all other code has finished executing.

Improving measurements

One of the first things we can address is that while a serialisation instruction is needed, cpuid is a relatively onerous instruction. lfence is an less onerous instruction that is also serialising, and in non-Intel manufacturers, mfence, which can be even more performant, is also serialising. Hence, we can use one of these instructions instead of cpuid (let's call it xfence).

A second change we can make is that, to make it more explicit that the mov instructions following rdtscp must be executed in that order, and after the code to be benchmarked, we can call xfence between rdtscp and the first mov. Thus we get:

XFENCE ; serialise
RDTSC ; read the clock
MOV start_cycles_high, edx
MOV start_cycles_low, eax
; Call the function to benchmark here
RDTSCP
XFENCE ; serialise
MOV end_cycles_high, edx
MOV end_cycles_low, eax

Warm-up measurements

As previously discussed, executing instructions the first few times might take a different time than executions that follow them. We can address this by calling rdtsc and xfence a few times before our benchmark code, so that we get something like this:

XFENCE
RDTSC
XFENCE
RDTSC
XFENCE
RDTSC
XFENCE ; serialise
RDTSC ; read the clock
MOV start_cycles_high, edx
MOV start_cycles_low, eax
; Call the function to benchmark here
RDTSCP
XFENCE ; serialise
MOV end_cycles_high, edx
MOV end_cycles_low, eax

Measuring in a loop

Now that we have established how to conduct measurements, we can start writing some code to run benchmark our code.

for(j = 0, k = 0; j < ITERATIONS; j++) {
        START_MEASUREMENT(cycles_high_s, cycles_low_s);
        ret = runOperation();
        if (likely(0 == ret)) {
                END_MEASUREMENT(cycles_high_e, cycles_low_e);
                start = ( ((u64) cycles_high_s << 040) | cycles_low_s );
                end   = ( ((u64) cycles_high_e << 040) | cycles_low_e );
                deltas[k++] = end - start;
        } else {
                fprintf(stderr, "Unexpected error occurred benchmarking type %s.\n", "NULL");
                printf("[%s] ERROR\n", "NULL");
        }
}
compute_statistics(deltas, k, &min, &max, &mean, &variance);

This code won't yet compile, because there are many things yet undefined, but we can see that it calls runOperation() for ITERATIONS time, calling START_MEASUREMENT (which we can later define as a macro) before starting and calling END_MEASUREMENT once it's done. In addition, we're calling compute_statistics at the end of the loop, which will compute some statistics about the operation (such as the minimum, maximum and mean time), which is our desired result from the benchmark.

Putting it all together

Different compilers

Since we're using C and executing native instructions, we need to use the asm keyword. Unfortunately, this is implementation defined. We'll address the Microsoft and GNU dialects, which will cover most implementations relevant for the x86 architecture.

For Microsoft, the syntax is __asm INSTRUCTION, whereas for GNU the syntax is __asm__ ("INSTRUCTION").

C standard versions

A further challenge is that different C versions (for example, ANSI C and C99) provide different types, and for compatibility we might want our code to use the latest types when available while still being able to run on older compilers.

Baseline measurement

Lastly, we want to run a baseline measurement for a do-nothing operation to determine the overhead of our measurement.

Final implementation

#include <math.h>
#include <stdio.h>
#include <stdlib.h>

#ifdef HAVE_CONFIG_H
#include <config.h>
#endif

#ifndef ITERATIONS
#define ITERATIONS (2048)
#endif

/* C99 types */
#if defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* The compiler supports C99 */
#define SIZE_T_FMT "z"
#define SIZE_T_FMT_TYPE size_t
#define LONGLONG long long
#define LONGDOUBLE long double
#define LONGDOUBLE_FMT "L"
#elif defined(_MSC_VER) /* Some MSVC versions don't fully support C99 */
#define SIZE_T_FMT "I"
#define SIZE_T_FMT_TYPE size_t
#define LONGLONG __int64
#if _MSC_VER >= 1310
#define LONGDOUBLE long double
#define LONGDOUBLE_FMT "L"
#else
#define LONGDOUBLE double
#define LONGDOUBLE_FMT ""
#endif
#else
#define SIZE_T_FMT "l" /* Old compiler, use long */
#define SIZE_T_FMT_TYPE long
#if defined(__GNUC__) /* GCC supports long long as an extension */
#define LONGLONG long long
#else
#define LONGLONG long
#define __extension__ /* */
#endif
#define LONGDOUBLE double
#define powl pow
#define sqrtl sqrt
#define LONGDOUBLE_FMT ""
#endif
/* End C99 types */

/* Compatibility */
#if !defined(__has_attribute) /* Clang & newer GCC */
#define __has_attribute(x) 0
#endif /* !defined(__has_attribute) */

#if !defined(__has_builtin) /* Clang */
#define __has_builtin(x) 0
#endif /* !defined(__has_builtin) */

#if !defined(INFINITY)
#define INFINITY ((double) 1.0e120)
#endif

#if defined(_MSC_VER) /* MSVC-specific */
#define NOINLINE __declspec(noinline) /* MSVC extension */
#elif __has_attribute(__noinline__) || (defined(__GNUC__) && (__GNUC__ > 3) || (__GNUC__ == 3 && defined(__GNUC_MINOR__) && __GNUC_MINOR__ >= 1))
#define NOINLINE __attribute__ ((noinline)) /* GNU extension */
#else
#define NOINLINE /* not supported */
#endif /* defined(_MSC_VER) */

#if __has_builtin(__builtin_expect) || (defined(__GNUC__) && (__GNUC__ > 3) || (__GNUC__ == 3 && defined(__GNUC_MINOR__) && __GNUC_MINOR__ >= 1))
#define likely(x) __builtin_expect((x),1) /* GNU extension */
#else
#define likely(x) x
#endif /* __has_builtin(__builtin_expect) */

#if defined(_MSC_VER) /* MSVC inline assembler */
#if !defined(intel) /* Use LFENCE for Intel processors and MFENCE for other manufacturers */
#define xFENCE __asm _emit 0x0f __asm _emit 0xae __asm _emit 0xf0 /* MFENCE */
#else
#define xFENCE __asm _emit 0x0f __asm _emit 0xae __asm _emit 0xe8 /* LFENCE */
#endif
#define RDTSC rdtsc
#define RDTSCP rdtscp

#define WARMUP_MEASUREMENT() __asm { \
        xFENCE \
       __asm RDTSC \
        xFENCE \
        __asm RDTSC \
        xFENCE \
        __asm RDTSC \
        xFENCE \
}

#define START_MEASUREMENT(HI, LO) __asm { \
        xFENCE \
        __asm RDTSC \
        __asm mov LO,eax \
        __asm mov HI,edx \
}

#define END_MEASUREMENT(HI, LO) __asm { \
        __asm RDTSCP \
        xFENCE \
        __asm mov LO,eax \
        __asm mov HI,edx \
}

#define ZERO_REGISTER(OUT) __asm mov OUT,0
#elif defined(__GNUC__) /* GCC-style extended asm */
#if !defined(intel) /* Use LFENCE for Intel processors and MFENCE for other manufacturers */
#define xFENCE ".byte 0x0f, 0xae, 0xf0\n" /* MFENCE */
#else
#define xFENCE ".byte 0x0f, 0xae, 0xe8\n" /* LFENCE */
#endif
#define RDTSC ".byte 0x0f, 0x31\n"        /* RDTSC */
#define RDTSCP ".byte 0x0f, 0x01, 0xf9\n" /* RDTSCP */

#define WARMUP_MEASUREMENT() __asm__ volatile ( \
        xFENCE \
        RDTSC \
        xFENCE \
        RDTSC \
        xFENCE \
        RDTSC \
        xFENCE \
        : : : "%eax", "%edx", "memory" \
)

#define START_MEASUREMENT(HI, LO) __asm__ volatile ( \
        xFENCE \
        RDTSC \
        : "=d" (HI), "=a" (LO) : : "memory" \
)

#define END_MEASUREMENT(HI, LO) __asm__ volatile ( \
        RDTSCP \
        xFENCE \
        : "=d" (HI), "=a" (LO) : : "%ecx", "memory" \
)
#define ZERO_REGISTER(OUT) __asm__ volatile ("xor %0, %0" : "=r"(OUT) : : )
#else /* Unsupported compiler. */
#define WARMUP_MEASUREMENT() /* */
#define START_MEASUREMENT(HI, LO) HI = 0; LO = 0
#define END_MEASUREMENT(HI, LO) HI = 0; LO = 0
#define ZERO_REGISTER(OUT) OUT = 0
#endif /* defined(_MSC_VER) */

/* End Compatibility */

typedef unsigned LONGLONG u64;
typedef unsigned long u32;

/* Function signatures */
NOINLINE static int do_nothing(void);
NOINLINE static int do_something(void);
static void compute_statistics(u64 const data[], size_t const len, LONGDOUBLE * min, LONGDOUBLE * max, LONGDOUBLE * mean, LONGDOUBLE * variance);
/* End function signatures */

int do_nothing() {
    int r;

    ZERO_REGISTER(r);

    return r;
}

int do_something() {
    int r;

    ZERO_REGISTER(r);

    return r;
}

void compute_statistics(u64 const data[], size_t const len, LONGDOUBLE * const min, LONGDOUBLE * const max, LONGDOUBLE * const mean, LONGDOUBLE * const variance) {
    LONGDOUBLE sum1 = 0.0L, sum2 = 0.0L, mean_, variance_, min_ = (LONGDOUBLE)INFINITY, max_ = (LONGDOUBLE) -INFINITY;
    size_t i;

    for (i = 0; i < len; i++) {
        LONGDOUBLE const t = (LONGDOUBLE) data[i];
        sum1 += t;
        if (t > max_) max_ = t;
        if (t < min_) min_ = t;
    }

    mean_ = sum1 / (LONGDOUBLE) len;

    for (i = 0; i < len; i++) {
        sum2 += powl((LONGDOUBLE)data[i] - mean_, 2);
    }

    variance_ = sum2 / (((LONGDOUBLE)len) - 1.0L);

    if (NULL != min) *min = min_;
    if (NULL != max) *max = max_;
    if (NULL != mean) *mean = mean_;
    if (NULL != variance) *variance = variance_;
}

int main(int argc, char **argv) {
    int ret = 0;
    size_t j, k;
    u64 * deltas;
    LONGDOUBLE min, max, mean, variance;
    u32 cycles_low_s, cycles_high_s, cycles_low_e, cycles_high_e;
    u64 start, end;

    (void)argc;
    (void)argv;

    deltas = (u64 *) malloc(sizeof(*deltas) * ITERATIONS);

    WARMUP_MEASUREMENT();

    /* Nothing */
    {
        printf("[%s] Starting benchmark\n", "NULL");
        for(j = 0, k = 0; j < ITERATIONS; j++) {
            START_MEASUREMENT(cycles_high_s, cycles_low_s);
            ret = do_nothing();
            if (likely(0 == ret)) {
                END_MEASUREMENT(cycles_high_e, cycles_low_e);
                start = ( ((u64) cycles_high_s << 040) | cycles_low_s );
                end   = ( ((u64) cycles_high_e << 040) | cycles_low_e );
                deltas[k++] = end - start;
            } else {
                fprintf(stderr, "Unexpected error occurred benchmarking type %s.\n", "NULL");
                printf("[%s] ERROR\n", "NULL");
            }
        }
        compute_statistics(deltas, k, &min, &max, &mean, &variance);
        printf("[%s]\t [%" LONGDOUBLE_FMT "e - %" LONGDOUBLE_FMT "e] mean: %" LONGDOUBLE_FMT "e (std. dev.: %" LONGDOUBLE_FMT "e) N:%" SIZE_T_FMT "u\n", "NULL", min, max, mean, sqrtl(variance), (SIZE_T_FMT_TYPE)k);
    }

    /* Something */
    {
        printf("[%s] Starting benchmark\n", "NULL");
        for(j = 0, k = 0; j < ITERATIONS; j++) {
            START_MEASUREMENT(cycles_high_s, cycles_low_s);
            ret = do_something();
            if (likely(0 == ret)) {
                END_MEASUREMENT(cycles_high_e, cycles_low_e);
                start = ( ((u64) cycles_high_s << 040) | cycles_low_s );
                end   = ( ((u64) cycles_high_e << 040) | cycles_low_e );
                deltas[k++] = end - start;
            } else {
                fprintf(stderr, "Unexpected error occurred benchmarking type %s.\n", "NULL");
                printf("[%s] ERROR\n", "NULL");
            }
        }
        compute_statistics(deltas, k, &min, &max, &mean, &variance);
        printf("[%s]\t [%" LONGDOUBLE_FMT "e - %" LONGDOUBLE_FMT "e] mean: %" LONGDOUBLE_FMT "e (std. dev.: %" LONGDOUBLE_FMT "e) N:%" SIZE_T_FMT "u\n", "NULL", min, max, mean, sqrtl(variance), (SIZE_T_FMT_TYPE)k);
    }

    free(deltas);

    return !!ret;
}

Smidyo x Apeleg = Vector Express

Ricardo Iván Vieitez Parra — Fri, 12 Aug 2022 11:07:00 +0000

Apeleg have partnered with Smidyo in a joint venture to provide Vector Express. Vector Express is a SaaS platform that addresses and simplifies many common tasks when working with vector files, providing conversion between a wide range of vector image formats as well as processing and analysis.

Vector Express is a unique offering because of the diversity of formats that it can handle, from AutoCAD DXF and DWG files to HP-GL to more common formats SVG and PDF and because of its flexibility: it can be used manually with the frontend at Vector Express or integrated with other systems with a flexible and well-documented API.

At Apeleg, we view this project as a long-term partnership that will further our goal of offering infrastructure to our customers to automate and improve their business processes and are excited to be able to take a more active role in developing Vector Express.

History

Vector Express was born as an internal project in Smidyo to support Smidyo's main offering, Smidyo's Quotation Portal, a platform for quote automation. Vector Express played a vital role in the platform for businesses in the laser cutting sector, which require handling a large volume of vector images and where an accurate and reliable analysis of vector files is essential to providing accurate price estimates.

As the platform grew so did Vector Express and soon it was clear that it could be useful as a service in its own right. Apeleg was involved in some of the design and development of Vector Express version 2, which introduced a new API and a more modular design for greater scalability.

At the time, Vector Express ran on a server with a graphical interface to run the different operations. The new version of Vector Express maintained and extended the existing functionality while eliminating the need for a graphical interface. It was also engineered with containers in mind so that it can be easily deployed in a way that is scalable and that its functionality can be extended for users requiring bespoke functionality.

Since Vector Express version 2 was released, it has grown in use and adoption, becoming an independent offering of Smidyo's and it has been extended with new functionality, from being a conversion tool to becoming a tool that can also process images and run various analyses.

Roadmap

First and foremost, our priority is ensuring the continued operation of the Vector Express API with the reliability and accuracy that the service has become known for among its free and paid users. To attain this, we are migrating Vector Express to dedicated infrastructure to improve response time and processing capacity.

Once this first step is completed, we already have some features planned for improving the service. Although Vector Express has been designed with modularity in mind, it runs mostly synchronously and has a complex setup of interrelated dependencies. As we have gained experience from Vector Express and other projects, we believe quality of service can be improved by switching to an event-based and asynchronous model powered by microservices, which will not only improve response times but will also provide significantly enhanced horizontal scaling capabilities and isolation.

We also want to include svgcut as a processor, as well as improve the frontend to make it a standalone application that can be used for those that don't require using an API and need more capabilities than the free plan offers.

Handling IP addresses

Ricardo Iván Vieitez Parra — Fri, 12 Aug 2022 11:04:35 +0000

IP addresses pose unique technical challenges because they are sent in the clear with every network request and because they may in some cases identify or help identify an individual, making them personal data. In the EU, this was affirmed already in case C-582/14 and, with the coming into force of the GDPR, this criterion has been applied to the transfer of IP addresses to the United States in the popular website analytics tool Google Analytics by data protection agencies in several countries already, such as Austria, the Netherlands and France, and others that may soon follow suit, like Norway. Similarly, in a recent case in Germany, a website operator was ordered to compensate a visitor for embedding Google Fonts and thus violating the visitor's privacy by sharing their IP address with Google.

Since IP addresses are sent along with every packet on the Internet and precisely because they identify the sender to some extent, they also have many practical uses. For instance, they are a valuable data point for detecting and investigating data breaches and other security incidents. They can also be used as input to automated moderation or filtering systems for purposes like blocking spam or online fraud and can also provide vital analytics for responding to traffic, like finding optimal route paths for transmitting content to our users.

Fortunately, regulations like the GDPR contemplate these use cases under categories like 'overriding legitimate interest', which means that not all use or processing of IP addresses is disallowed without explicit user consent. However, since IP addresses are personal information, it is important to ensure that they are only used in connection to essential purposes that don't require explicit consent or that consent is obtained prior to processing. The particular challenge of IP addresses compared to other personal information is that, by the nature of how the Internet works, they can be very easily relayed to third parties by things as simple as having the user download a file.

To avoid this situation, we are left with three options: not using external resources, obtaining the user's consent or self-hosting.

Not using (certain) external resources

While this addresses the concern of sharing the user's IP address without consent, it's not always a viable option, because those resources may very well provide value to our website and our users, for example, jQuery might what our website uses to provide an interactive page, or those resources might be fonts that are important to use for consistent branding. Nonetheless, the option of not using an external resource is a valid one to consider because, if feasible, it is generally the easiest alternative. This option is valid for resources that are not really used or which are easily replaceable.

Apart from not sharing IP addresses with third parties, the advantage of not using certain resources is that we get to improve the user's experience by having a small page that loads faster.

Obtaining the user's consent

While this is a valid option to consider, it might not always be practical or a preferred solution, because the user consent needs to come first, before any resource is loaded. Take the example of public CDNs, which host resources that may be essential for having our website work, or at least to provide a rich and high-quality user experience. If we seek to have the user's consent, depending on what we use those external resources for, in practice it means that we will deliver a degraded user experience until the consent is obtained. While this could be acceptable for certain sections or functionality of our page, it probably isn't for core functionality. More importantly, it can unnecessarily pressure the user into giving consent (removing the 'freely given' element of it) since doing otherwise implies having a poorer experience interacting with our site.

Self-hosting

Self-hosting, as the name implies, is hosting resources ourselves in our own infrastructure. While doing this can in some cases be an involved process, especially if we are implementing self-hosting on a site that currently loads several resources from third party origins, when done well it has the advantages of not leaking visitors' IP addresses to others, not requiring explicit consent and not resulting in limited functionality or interactivity.

Server logs

Self-hosing doesn't address all of the challenges that come with processing personal information, but it does put us back in control and from that position we can implement policies that protect our users' privacy. One of those challenges is that most web servers by default store incoming IP addresses in their various logs. The fact that IP addresses are stored is not necessarily a problem, as they can provide useful information for, for example, adequately responding to a security incident.

However, it's crucial to treat these entries with IP addresses as containing personal information and ensuring the right procedures and policies are in place, starting with the use to be given to the information. For example, while it is most certainly appropriate to keep logs for security purposes, using the IP entries in log files to also correlate sessions and derive behavioural analytics is likely to require the users' informed and actively given consent.

A second consideration when keeping log files with IP addresses is elaborating a data retention policy. The benefits of having logs of IP addresses for security purposes diminish over time, and need to be weighed against the disadvantages, namely that storing information opens us up to leaking that information (for example, by accident or due to server compromise) and that users might request their personal information be deleted, and handling these requests manually can be time-consuming and expensive. Hence, in many cases the optimal solution is retaining logs for a period long enough to respond to incidents but short enough to avoid the information turning into a liability.

Different jurisdictions

Different jurisdictions have different regulations and requirements about processing personal data, and what is permissible or even required in one place can be disallowed in another. Therefore, in as much as it is possible, the preferred solution is avoiding transferring personal information between jurisdictions, especially between those that don't have comparable levels of data protection. One example are data transfers between the EU and the US, which after the ECJ invalidated the Privacy Shield framework require additional considerations to ensure that all EU regulations are being followed.

If we host our server in the United States and receive visitors from the EU, safeguards need to be in place to ensure that all personal data being transferred, including IP addresses, are going to be handled appropriately and in accordance with the EU regulations. Since this can be burdensome, a solution for organisations using a CDN with many local points of presence might be to simply implement firewall policies at the edge and strip requests of all personal information, thus avoiding this requirement. For example, Cloudflare offers request transform rules that can remove information present in request headers, like IP addresses.

Analytics services

Analytics provide insightful information about how our web properties are used, which can in turn be valuable feedback for improving user experience and identifying areas for improvement. On the other hand, when implementing analytics, as with everything else, we must consider how our visitors' personal data will be processed and the basis for that processing. There exist a wide range of services, including the popular and ubiquitous Google Analytics, that enable publishers to collect and process various types of metrics.

However, using third party tools presents many of the same challenges as loading content from external sources do. Although Google Analytics has an opt-mechanism to avoid collecting users' IP addresses, as mentioned earlier, these measures have been found to be inadequate in many countries in the EU. The solutions are also in many ways the same as for loading external content.

Since analytics services are in most cases non-essential, the alternative of collecting users' consent could in this case be more viable than it is for loading essential libraries, and is implemented in many websites nowadays in the shape of consent prompts. It is still however not necessarily good for neither user experience nor user trust having to present these prompts and warn of the risks of cross-border data transfers.

When using analytics services provided by a third party, it can be worth considering to proxy connections to the service through our own infrastructure instead of having users connect directly to the third party analytics collection endpoint. This way, it is possible to scrub the data being sent and remove any personal information. In the case of Google Analytics, this means proxying requests to https://www.google-analytics.com/collect with our own domain.

Analytics services can also be self-hosted. While this is a more expensive solution from a management perspective, it is the solution that affords the greatest flexibility for implementing and enforcing privacy protections. There are many offerings in this space, including established projects like Matomo (formerly Piwik) and Open Web Analytics.

DEV Community: Ricardo Iván Vieitez Parra

Privacy Pass: The Revolution in CAPTCHA Mitigation and User Privacy

Introduction

Issues with CAPTCHAs

Introducing Privacy Pass

Moving on and into the browser

When and where can I use this?

How will this not lead more centralisation?

Closing remarks

Conclusion

Understanding Identity and Access Management (IAM)

Introduction

Why IAM is important

Authentication

Authentication methods

Username and password

Magic links

Cryptographic authentication

Multi-Factor Authentication (MFA)

Implementation options

Per-system authentication

Dedicated authentication services

Single Sign-On (SSO) and federated identity

Authorisation

Authorisation strategies

Role-Based Access Control (RBAC)

Policy-Based Access Control (PBAC)

Attribute-Based Access Control (ABAC)

Context-Aware Access Control (CAAC)

Access Control Lists (ACLs)

Implementing authorisation

Internal rules

Delegated authorisation

Delegation authorisation patterns

Conclusion

Opaque IDs: the ultimate protection against enumeration attacks

Why IDs are used

How IDs might be exploited to gain access

German tank problem

Enumeration attacks

Prevention

Timing attacks

Mitigation

Encrypted IDs

Example solution in TypeScript

Approach taken

Progressively loading CSR pages

Progressive enhancement

Progressively loading web applications

Server-side rendering vs. client-side rendering

<noscript>

Handling errors

<script> attributes

Putting it all together

Effectively mitigating CSRF

What CSRF is

Examples

Classic CSRF: a concrete example

Bonus classic CSRF example: logging out

Client-Side CSRF: a concrete example

CSRF mitigations

SameSite cookies

TLS 1.3 + SameSite

CSRF tokens

Double-submit cookies

Additional restrictions

Synchronised tokens

Avoiding cookies

CORS

Origin and Referer headers validation

Sec-* headers validation

Built-in CSRF protection

Protection against Client-Side CSRF

Other defence-in-depth measures

Content Security Policy (CSP)

Apeleg join the W3C

Modern and robust hotlink protection in 2022

What a hotlink is

Older approach to hotlink protection

Example configuration for Referer-based protection

`<noscript>`

`<script>` attributes

`SameSite` cookies

TLS 1.3 + `SameSite`

`Origin` and `Referer` headers validation

`Sec-*` headers validation

Example configuration for `Referer`-based protection

Apache: `.htaccess`

Issues with `Referer`-based protection

`Referer` omission

`Origin`

`Cross-Origin-Resource-Policy`

Difference between `same-origin` and `same-site`

`Access-Control-Allow-Origin`

Apache `.htaccess`