DEV Community: BlackMagiq

Beware of Your Static Database Credentials

BlackMagiq — Thu, 04 Apr 2024 15:15:08 +0000

Like it or not, every static database credential, and more broadly authentication token, is a secret waiting to be leaked and exploited by a bad actor. Be it one or ten years from now and as careful as you may be, the credential is bound to get leaked through one of many avenues and the proof is all over the news. From Mercedes-Benz to AstraZeneca, it almost seems that no company is immune to credential leaking. That said, there may be a solution after all, one that is common enough to be named but clearly not widespread enough to be employed everywhere.

In this article, we dive into everything wrong with static database credentials and discuss how we can improve to mitigate any risks associated with leaking them.

What’s wrong with static database credentials?

Most users and applications in the wild authenticate with databases using manually-provisioned, static database credentials. This approach is often problematic for a few reasons:

If you happen to leak a static database credential, then you give an attacker an indefinite window of opportunity to exploit your database. Moreover since your applications may share database credentials, they may collectively increase the overall risk of leaking secrets throughout your development cycle.
In the lucky event that you discover the incident, then you may try to manually revoke the database credential as soon as possible. The revocation process, however, may take time and hence introduce yet another shorter window of opportunity for an attacker to exploit your database.
In any case, you will want to audit the incident to check the details of how it transpired. Depending on whether or not parts of your infrastructure share database credentials, however, you may find it difficult to map a good audit trail.

In a world where one database credential leak could mean an attacker gaining access to a treasure trove of data, it’s clear that we need better practices.

Introducing the Moving Target Defense

As the Department of Homeland Security states, many systems have been “built to operate in a relatively static configuration. For example, addresses, names, software stacks, networks, and various configuration parameters.” Seeing associated issues, the Moving Target Defense (MTD) was a concept introduced with the intent of “controlling change access multiple system dimensions to increase uncertainty and apparent complexity for attackers, reduce their window of opportunity and increase the costs of their probing and attack efforts.”

While MTD covers a broad range of systems, for the sake of our discussion, it can be seen widely applied in the context of web applications. For example, one common moving target defense mechanism is the refresh token mechanism that is commonly used in web application authentication or protocols like OAuth 2.0.

In the refresh token mechanism, an application may redeem a long-lived token (the refresh token) for a short-lived access token and use that to authenticate with a target service. By doing so, it reduces any risk associated with leaked access token exploitation because an access token would likely be expired by the time it is found anywhere by a bad actor. Put differently, the risk of exploitation reduced with an access token developed into a moving target with short time-to-live (TTL).

How does Moving Target Defense relate to static database credentials?

When dealing with static database credentials, we can use two moving target approaches to reduce the window of opportunity that an attacker may have of exploiting database credentials: secret rotation and dynamic secrets.

Secret rotation at interval: Secret rotation is the practice of rotating database credentials at a specific interval where rotating implies invalidating a set of database credentials in favor of newly-issued ones. For example, you may have a script or application rotate a database credential every 30 days to completely remove the risk of an attacker being able to use a leaked, old database credential. Since database (and other types) credentials often get mishandled, forgotten about, and eventually leaked somewhere like in a public repository where it is swept up and exploited by a botnet; it’s surprisingly common to run into instances of leaked and exploited long-lived credentials, cases that would’ve easily been avoided with proper secret rotation practices in place.
Secret rotation on-demand: In cases where a secret does get leaked and you need to revoke it immediately, secret rotation can be manually triggered to immediately invalidate and issue a new set of database credentials. Ultimately, both interval-based and on-demand secret rotation are useful to ensure that database credentials are switched out periodically and immediately in the event of an incident.
Dynamic secrets: Lastly, dynamic secrets is the practice of issuing each engineer or application that needs access to database credentials a unique, short-lived credential on-demand that is when requested. This removes the accumulated risk associated with shared database credentials, isolates each party’s access, and most importantly retains the moving target principle for database credentials. In a dynamic secrets scenario, an application requests access to a database from another application (we’ll get to this in a bit) after which it is granted a short-lived access token to do so; the scenario maintains that the application’s database access can be cut off independently and audited if needed.

How are secret rotation and dynamic secrets implemented in practice?

As you may have guessed, implementing secret rotation and dynamic secrets is not trivial and requires introducing a third-party script or application to orchestrate the mechanisms. The third-party application would be responsible for updating the values of your database credentials as in the secret rotation case and/or issuing database credentials downstream to your applications that request them.

While it’s possible to implement your own secret rotation and dynamic secrets functionality, it’s strongly encouraged to instead use a secrets manager. Using purpose-built tooling ensures that you reap all the benefits of good secrets hygiene including comprehensive visibility over secrets scattered throughout your infrastructure and moving targets applied to your database credentials.

When using a tool like Infisical, for example, you set up dynamic secrets by delegating privileged access to the platform so it can issue temporary database credentials (i.e. dynamic secrets) bound under “leases” to applications that request them; upon lease expiration, Infisical automatically revokes the dynamic secrets, so applications must request a new dynamic secret to continue accessing the intended database.

Conclusion

Many companies fall victim to using static database credentials only to find out, one or ten years from now, that they have accidentally leaked them somewhere across their sprawled infrastructure. From small to large companies and organizations, there’re countless stories to show that this is no rare occurrence. That said, by employing Moving Target Defense principles and specifically practices like secret rotation and dynamic secrets, you can improve your organization’s overall security posture and maximally reduce the window of opportunity that a bad actor may have of exploiting database credentials in the event of a leak.

Onward and upward!

— -

Infisical — The open source secret management platform

Infisical (11.8K+ ⭐️)helps thousands of teams and organizations store and sync secrets across their team and infrastructure.

GitHub repo: https://github.com/Infisical/infisical

The Great Migration from MongoDB to PostgreSQL

BlackMagiq — Fri, 29 Mar 2024 12:23:11 +0000

Infisical has grown rapidly in the past year with the platform now processing north of 50 million secrets daily that is sending application configuration and secrets data to teams, CI/CD pipelines, and servers/applications that need them.

With usage continuing to grow, we’ve had to continuously upgrade our stack. More recently, Infisical underwent a full database migration from MongoDB to PostgreSQL. This entailed deliberating the initiative, adopting new tech, creating new database schemas, rewiring logic, re-writing queries, to migrating millions (if not billions) of database records over to PostgreSQL. This was an elaborate process but one that was by all means necessary and for the betterment of the platform.

This is the story of our decision-making behind why we moved from MongoDB to PostgreSQL and how we did it. Hopefully, this makes for an interesting read and is useful for others who may find themselves one day considering a similar database migration.

Where we started

When we first built Infisical, we built it with the stack that felt most familiar with the team. As part of that stack, we chose MongoDB + Mongoose ORM because the combination presented least overhead and allowed us to ship quality features quickly. As Sir Tony Hoare states, “premature optimization is the root of all evil,” and there was certainly no need for further optimization at the time.

At the time, we were also more focused on building Infisical Cloud, the managed SaSS offering, and given this focus, we didn’t anticipate as many users self-hosting the product and hence it wasn’t designed with that use-case in mind.

Why not MongoDB?

While MongoDB served Infisical well in the early days, it started showing signs of shortcoming when the use-case of our product evolved beyond the managed service. As time passed, we found that many organizations, especially ones operating at the intersection of compliance and security, preferred self-hosting Infisical as opposed to using Infisical Cloud; others had on-prem requirements that needed to be met.

With demand growing for self-hosting Infisical, we found ourselves shipping many features catered to reducing the learning curve needed to self-host Infisical and, as part of that, we ended up leaving MongoDB in favor of PostgreSQL.

In practice, we and our customers often ran into constraints around the capabilities and usability of MongoDB like the lack of support for transactions, clean-up, inconsistent versioning across managed offerings by cloud providers, not to mention issues associated with schema-less database design structure.

I elaborate more on a few of these challenges below:

Difficulty configuring database transactions: With MongoDB, setting up transactions was not trivial because it required running MongoDB in cluster mode with various configuration overhead; this made it extremely difficult, for instance, for customers to run a simple POC of Infisical because it required a production setup of MongoDB. For a product dealing with highly-sensitive data where data integrity is a must, this was a no-go.
Missing out on relational features: With MongoDB, we lost out on many nice features from the relational world like CASCADE which, when specified, deletes all referenced resources across other tables whenever a target resource is deleted; this hurt in particular because our data was very much relational. As a result, we employed hefty delete functions in our old codebase that never fully did the job and left dangling resources in the MongoDB databases.
Lacking support across cloud providers: After MongoDB’s license change to SSPL, many cloud providers opted to offer older versions of MongoDB. As a result, we found it difficult to ensure the availability of features of Infisical for customers running on everything other than the latest stable version(s) of MongoDB.
Lacking experience with MongoDB: Since more people were familiar with deploying SQL-based databases, they often struggled to scale and properly configure MongoDB; this led to a disproportionate uptick in the amount of support we needed to provide for customers specifically because they weren’t familiar with MongoDB.

Amongst a dozen more reasons, we came to the realization that a full database migration to something more universal was the ultimate feature needed to make Infisical more accessible to teams and organizations around the world.

Why PostgreSQL?

When searching for a new database, we began by listing out what aspects mattered most to us: ease of management (i.e. configuration, deployment, and scaling included), built-in support for transactions, and relational capabilities. As part of the deliberation, we also contemplated whether or not we should build our own integrated storage or pursue an external storage solution.

Here’s what that meant for each option:

Integrated storage: We could package in a database system like SQLite directly into Infisical and pursue a horizontal replication strategy to reduce latency by avoiding extra network hops. In this model, scaling the system would mean deploying multiple instances of Infisical and have them communicate with each other via some consensus algorithm like Raft. While this seemed like an excellent solution since customer’s wouldn’t need to connect any dependencies to run Infisical, the tooling ecosystem to execute this vision felt immature and the engineering effort required for it felt nothing short of overwhelming.
External storage: We could simply replace MongoDB with another database(s) like PostgreSQL or MySQL and use its built-in scaling capabilities. Although this solution didn’t fully eliminate friction associated with needing external dependencies to use Infisical, we felt that it already delivered significant benefits by virtue of not being MongoDB. When it came to supporting one or multiple databases, we felt that supporting multiple would mean missing out on the unique advantages of each solution; it would also add to our engineering overhead.

After careful consideration, we chose PostgreSQL. Beyond having a vibrant community, extensive documentation, and a myriad of solutions and extensions available, we appreciated most its open source nature and how the vast majority of cloud providers offered managed services of PostgreSQL.

Above all, this meant that users of Infisical could more easily self-host our platform on any cloud provider and pair it with its corresponding managed PostgreSQL service. Moreover, given how widely-adopted the database has become, we were confident that users would have less trouble operating it when using Infisical.

What about the ORM?

After settling on PostgreSQL, we needed to figure out how our application would interact with the database. Right off the bat, we wanted something comparable to our experience with MongoDB where we used Mongoose ORM. So, we began evaluating candidates on the basis of maturity, visualization and migration support, and appropriate level of abstraction; we primarily considered Drizzle ORM, Prisma ORM, TypeORM, and Knex.js, a query builder.

At the end, we decided to use Knex.js, a query builder, instead of a ORM to maintain better control over the database. While admittedly, going with raw SQL would be most versatile with least abstraction in place, we felt the approach would be far too error-prone and frankly cumbersome to maintain, especially without proper TypeScript support. Moreover, beyond being close to bare SQL, Knex.js came with its own toolkit for seeding and migration, had a mature ecosystem with excellent documentation and answers for almost any possible query. Coupled with some custom Zod integration work, we managed to get it to a satisfactory level for TypeScript support.

Having decided on the database and ORM, we kicked off a process that would ultimately result in a re-write of dozens of data structures and hundreds of queries across the application.

How did we plan the migration?

Toward the end of the code-rewrite, we started to think about how we would conduct the migration operation to map our MongoDB data to PostgreSQL with minimal disruption to the Infisical Cloud platform.

Given Infisical’s critical role in customers infrastructure, we immediately ruled out the possibility of having any absolute downtime. The part where we had to make a compromise was in disallowing write operations during the brief migration window (i.e. customers would not be able to create or update application configuration) in exchange for higher guarantee of data integrity. This tradeoff seemed acceptable given that customers primarily fetched back secrets from Infisical and, to a much lesser extent, updated their application configuration on a second-by-second basis.

Next, regarding the actual migration operation, we needed to dump data from MongoDB, transform it carefully, and insert it back into PostgreSQL. As we audited the migration sequence, we grappled through challenges like making sure that various tree-like structures from NoSQL were correctly transformed to their relational counterparts; this was particularly sensitive for data structures like folders that had recursive considerations. We also found that we needed a persistent way to store and map identifiers in MongoDB to those in PostgreSQL; doing so in-memory would not work considering how much data we were dealing with. In the end, we settled for using the LevelDB key-value store to assist with identifier storage and lookup operations. With it, we would move data table-by-table into PostgreSQL.

The Great Migration

Finally, we were ready to conduct the migration. At this point, folks not directly involved in the codebase re-write had spent a much-needed quarter improving other aspects of Infisical including making frontend changes, performing maintenance patches, extending client functionality, and writing up better documentation. We now all reconvened to prepare for the migration itself that is replacing the application codebase with the new one and transferring data over from MongoDB to PostgreSQL.

As part of the preparation, we drafted a detailed migration checklist with an expected timeline.

On a high-level, the plan looked something like this:

In the weeks building up to the migration, we would communicate in advance via both email and in-app banner to let users know about the impending database upgrade. We would conduct thorough testing of every feature flow on the platform and perform trial runs for the migration.
The migration itself would occur within a six-hour window where only read operations would be allowed to the platform. During this window, we would run the migration script to move data from MongoDB to PostgreSQL, check that no data was lost, and if successful switch the DNS to the new instance. There were of course backup plans in place in case things went south.
Finally, after the migration, we would iron out any residual issues and start rolling out new documentation for working with Infisical and PostgreSQL.

With the plan in hand, we proceeded to the execution.

Results

Fortunately, the migration execution turned out smooth with zero data loss and only a few non-essential incidents of feature malfunction; we ironed out these bugs out in the following 36 hours with minimal impact to customers.

Following the migration, we observed many benefits:

The platform experienced significant performance gains largely attributed to query optimizations with joins. With MongoDB, the platform often made inefficient aggregation queries and multiple network hops to achieve needed functionality. For instance, due to the relational characteristics of our core data, we often had to perform many $lookup operations to simulate joins in SQL; such operations were inefficient and often required us to scale up both the database and application instances accordingly. Having moved to PostgreSQL, we avoided these inefficient operations which also resulted in a 50% cost reduction on our database bill.
The platform now employed better data validation rooted at the database level rather than at the application layer. Since MongoDB was schemaless by design, it relied on Mongoose’s framework to define data types, required fields, and validation rules. With PostgreSQL in place, we no longer faced data inconsistencies that would otherwise previously occur if the database was ever accessed or modified outside of Mongoose’s purview.
Lastly and most importantly, we believe that Infisical is much easier to self-host now with customers able to conduct POCs with no additional configuration overhead such as dealing with replica sets in MongoDB to enable transaction capabilities.

Overall, we considered the initiative to be highly successful given the objective at hand, the scope of the task, and the resulting execution of it. We intend on publishing more concrete results in the future once we have more data on hand.

Conclusion

The decision to move from MongoDB to PostgreSQL was not an easy one from the get-go. All in all, the initiative took us 3–4 months to perform with careful planning and discussion around why we needed to perform it, how we were going to do it; and then to execute it all with care. For anyone reading this, I’d highly recommend thinking through the use-case and implementation deeply before attempting such a big endeavor. Overall, I’m extremely happy that everything went according to plan and we were able to deliver such a huge update that will make a large difference to users of Infisical moving forward.

Many thanks to Akhil Mohan for absolutely crushing the migration initiative and everyone else at Infisical for assisting with the process.

— -

Infisical — The open source secret management platform

Infisical (11.8K+ ⭐️)helps thousands of teams and organizations store and sync secrets across their team and infrastructure.

GitHub repo: https://github.com/Infisical/infisical

Guide to Building Great Audit Logs for Application Software

BlackMagiq — Tue, 15 Aug 2023 18:15:46 +0000

It’s 2023 and audit logs are a core component of any enterprise product offering. As simple as they seem, audit logs can be tricky to implement.

Having made this feature twice as part of my work at Infisical, I discuss everything about audit logging in this article and specifically how to ship them properly for yourself.

What are audit logs?

To begin, audit logs are a centralized stream of user activity used by security and compliance teams at enterprises to monitor information access in the event of any suspicious activity or incident review. At first glance, they are entries consisting of events, time-stamps, and payloads. On a deeper level, however, they can be seen as summaries that, when taken in tandem, capture a frame-by-frame narrative of what happened at any given period of time in great detail (we call this a fine-grain audit trail).

Making great audit logs

In this section, we outline the data structure we’d expect from an audit log and the principles we wish to uphold.

The general data structure

As a reminder, audit logs are purpose-built for security and compliance teams to monitor and inspect the context of an incident at a given point in time. Given this, a good set of audit logs should, at the every least, contain fields that answer your basic interrogative pronouns:

Event: What happened? — For Infisical, this was, for example, an action associated with a secret such as it being fetched or created.
Actor: Who triggered the event? This may be the name, email, or identifier for the entity responsible for performing or causing the underlying event.
Timestamp: When did the event occur? — Source (User agent + IP): Where did the event occur? If you’re building an API, both user information and IP address can typically be obtained from the request itself. Frameworks like Express, for instance, expose the request’s user agent under req.headers[“user-agent”] and req.ip .
Metadata: Additional data to provide context for each event. In our case, this could’ve been the path at which a secret was fetched from etc.

Beyond these fields, it can be useful to store the following data (these may or may not be applicable):

Source Type: The source category of the event. In our case, requests could be made from a Web UI, CLI, Kubernetes Operator, or Other (i.e. anywhere else). We’ll discuss this more later but including such a field helps from a querying standpoint.
Description: A human-readable description of the action taken. The defining characteristics.
Status: Like HTTP requests, it’s possible for events to succeed or fail. If applicable and you have a way to detect this status information, you should include it in the entry. This would be helpful to identify, for instance, if someone attempted to access a forbidden resource.

A good audit logging system should not only record event data but also consider various usability criteria for the teams inspecting the logs (i.e the end users) and engineers of the system itself:

Immutable: Users of the system should only have the ability to read data from it and not write to it. Having write-ability would compromise the integrity of the audit logging system since it must reflect the exact state of events in the past.
Queryable: Users of the system must be able to efficiently search through the audit logs by applying any one or more filters; filtering should be quick. In case of a security incident, an administrator should be able to filter out all events where actor A performed action B from source C between the dates D and E.
Exportable: Data must be exportable and/or streamable to a dedicated logging solution like Splunk, AWS, etc. This is useful for security teams to centralize their security logs to be analyzed externally.

Lastly, you may consider some additional nice-to-have qualities:

Lightweight: Audit logs should capture brief summaries, snippets, and/or references to data rather than include all of it. So, if you find yourself copying an entire data structure over as if you’re capturing a database backup/snapshot, then you’re probably doing it wrong (e.g. you don’t need to attach every detail about the actor for an event; you may only need their identifier to link them to the event).
Parsable: A great audit log data structure should also be easy to parse and, more generally, work with. The shape and structure of your data can have implications on how easy it is to render it over in the platform UI as well as to manipulate for some future engineering endeavor.

How we did it at Infisical

When we first built the audit logging functionality for Infisical, we didn’t fully consider the target user at hand.

Instead, we viewed the objective in an overly-simplified way: create a chronologically-ordered ledger of events for teams to inspect the past. We also mistook audit logs for complete snapshots of data and state that, in retrospect, should be the responsibility of database backups/snapshots.

As a result, we ended up with a impractical audit logging system that took us back to the drawing board.

The data structure we used

After giving it a lot of thought, we opted for the following audit log schema:

export interface IAuditLog {
    actor: Actor;
    organization: Types.ObjectId;
    workspace: Types.ObjectId;
    ipAddress: string;
    event: Event;
    userAgent: string;
    userAgentType: UserAgentType;
    expiresAt: Date;
}

const auditLogSchema = new Schema<IAuditLog>(
    {
        actor: {
            type: {
                type: String,
                enum: ActorType,
                required: true
            },
            metadata: {
                type: Schema.Types.Mixed
            }
        },
        organization: {
            type: Schema.Types.ObjectId,
            required: false
        },
        workspace: {
            type: Schema.Types.ObjectId,
            required: false
        },
        ipAddress: {
            type: String,
            required: true
        },
        event: {
            type: {
                type: String,
                enum: EventType,
                required: true
            },
            metadata: {
                type: Schema.Types.Mixed
            }
        },
        userAgent: {
            type: String,
            required: true
        },
        userAgentType: {
            type: String,
            enum: UserAgentType,
            required: true
        },
        expiresAt: {
            type: Date,
            expires: 0
        }
    },
    {
        timestamps: true
    }
);

Fundamentally, we split the necessary components into 3 parts adjusted to our needs:

Actor: Since events in Infisical could be triggered by human and non-human (i.e. services) actors, we added actor type to the schema. Moreover, since human and non-human actors had different identifier metadata like email, user ID, service ID, etc., we also created a well-typed, mixed type metadata field to store this information for each actor. The defining thought process here was query-ability.
Event: The name of the event and any associated metadata unique to each event.
Other: Request information such as the IP address and user agent of the inbound request as well as a timestamp of when it came in.
Equipped with this data structure, creating and querying for audit logs became easy as cake.

The UI and UX

As you’d expect, we organize audit logs into a paginated table view, equipped with filters that users can apply together to narrow their search.

We made a ton of improvements since our first iteration of audit logs not limited to:

Filter options: You can filter audit logs by event, user, source, date, or any combination of these filters.
Metadata: We display only relevant metadata and nothing excess. For instance, we don’t store and perform any crypto operations on audit logs when it comes to secrets; we just offload that to another part of the application that’s responsible for it.
Timestamps: We format time-stamps in “YYYY-MM-DD hh:mm am/pm” format.

Closing thoughts

Audit logs are a high-demand feature when it comes to selling enterprise software and it’s important to get them right. By choosing a versatile data structure and considering how they are used in practice, you can design great audit logs for your software that help users gain insight for security reviews and incidents.

— -

Infisical — The open source secret management platform

Infisical (8.4K+ ⭐️)helps thousands of teams and organizations store and sync secrets across their team and infrastructure.

GitHub repo: https://github.com/Infisical/infisical

Guide to pentesting (what, why, and how)

BlackMagiq — Wed, 21 Jun 2023 10:46:25 +0000

In May 2023, Infisical commissioned cybersecurity firm Oneleet to perform a full-coverage, gray box penetration test (pentest) against the application's entire attack surface to identify vulnerabilities, according to industry standards (such as OWASP ASVS, WSTG, TOP-10).

In this article, I share my experience spearheading the pentesting initiative for Infisical that is our motivation and how the procedure went down for other companies that may be considering undergoing a pentest.

What is pentesting?

Pentesting or ethical hacking is a simulated cyber attack against a computer system, network, or web application to identify vulnerabilities that could be exploited by malicious actors. The goal of pentesting is to identify weak spots in security posture which can be used to fine-tune security policies, patch vulnerabilities, and enhance overall security measures.

Why should you do a pentest?

There are plenty of compelling reasons to do a pentest. Here are a few that stood out to us and that may be relevant to you:

To proactively uncover and remediate security vulnerabilities before malicious actors discover them and, in doing so, strengthen your organization's security posture.
To ensure compliance with standards like SOC 2 and ISO 27001 where policies may require your organization undergo pentests on a regular basis.
To increase trust between you and existing/prospective customers, especially when your business handles sensitive data.

As an open source secret management solution with security at the heart of everything we do, Infisical undergoes full-coverage pentests at least twice per year to ensure that the codebase is kept secure. That said, it may or may not make sense to do a pentest early on depending on the nature of your product/service, the sensitivity of data handled by it, the complexity of your systems, your organization's risk tolerance and compliance requirements, etc. The stage of your company may also matter since, for example, a pre-product market fit company may be concerned more with shipping product out the door quickly to ascertain demand rather than securing it.

Ultimately, the decision is multifaceted and must be judged on a case-by-case basis.

What firm should you hire to conduct the pentest?

You should hire a firm with a demonstrated history and track record of pentesting; ideally, it is also affordable for your company. With nearly a decade of experience performing pentests for 100+ companies and positive reviews/feedback from within the startup community, the team at Oneleet felt like a strong fit for Infisical and we were quick to get started.

What is the structure of a pentest?

Depending on the scope of the engagement, you can expect a multi-step sequence spanning one or more months. In our case, we followed a four-phase roadmap:

Information gathering: This step involved a call with Oneleet and assigned testers to determine the scope and timeline of the engagement, provide background information about the platform stack and infrastructure, and prepare for the test such as by providing the team access to a staging environment.
Pentest start: The pentest itself involved two testers manually performing a large number of tests on the application with extra attention paid to vulnerabilities that could cause serious damage to Infisical.
Reporting: After the pentest, Oneleet delivered a report consisting of an executive summary, a detailed finding section, and recommended remediations for the findings.
Remediation & retesting: Finally, the team at Infisical remediated any findings within the following week. During this period, Infisical maintained regular communication with Oneleet who were very responsive to inquiries. Following remediation, Oneleet provided a remediation report and letter of attestation for the conducted pentest.

Do you recommend doing a pentest?

Definitely — We had a pleasant experience working with Oneleet.

On a final note, we recommend doing a pentest at least once a year; we also recommend shaping a comprehensive security strategy early on and getting certified for compliance standards like SOC 2 and ISO 27001 early on. Prioritizing these initiatives early on help maintain your organization's security posture in the face of malicious actors and strengthen trust between you and your customers.

Scan your codebase for leaked environment variables instantly

BlackMagiq — Wed, 24 May 2023 13:13:09 +0000

All it takes is for one developer on your team to forget to .gitignore their .env file or remove a hardcoded environment variable they used in development for sensitive data to leak to source control. In the modern era where bad actors use bots to monitor repositories on GitHub, leaking environment variables can be a fatal mistake.

In this article, I introduce a CLI tool that your team can use to scan your existing codebase for any leaked environment variables instantly. It can scan your existing commit, past commits, and be configured as a pre-commit hook to prevent developers from committing any unintended secrets to source control. By the end, you should be able to detect leaked environment variables in your codebase and set up a pre-commit hook to automatically scan for leaks and prevent committing them to source control.

Why should I care?

It's important for developers to know the best practices for managing environment variables - to be more precise, managing secrets that is sensitive data like, for example, an API key that we wouldn't want bad actors to get ahold of.

Perhaps the first step to better managing secrets is to adopt the tooling necessary to prevent leaking secrets to begin with.

What's the tooling?

While there are numerous CLI tools that you can use to detect secrets in your codebase, I'm going to be using the Infisical CLI in this article because it's simple and has synergies with Infisical, a popular open-source secret management stack that can also be used to store your environment variables.

The CLI itself is free to use, and the Infisical platform itself can also be self-hosted on your own infrastructure as well.

Installation

To begin, you need to install the Infisical CLI onto your machine.

MacOS (using brew):

brew install infisical/get-cli/infisical

Windows (using scoop):

scoop bucket add org https://github.com/Infisical/scoop-infisical.git

scoop install infisical

Arch Linux (using yay):

yay -S infisical-bin

Check out the documentation for more installation options like Redhat/CentOS/Amazon and Debian/Ubuntu if your system is missing above.

Great! You're now ready to detect secrets in your codebase.

Scanning a directory, file, or full Git history for secrets.

The scan command can be run in a directory to detect any secrets within it; it looks out for 140+ secret types like potential API keys etc.

# scan your codebase for secrets
infisical scan

# display the full secret findings
infisical scan --verbose

infisical scan simply prints how many leaks were found across your commits.

11:13AM INF scanning for exposed secrets...
11:13AM INF 932 commits scanned.
11:13AM INF scan completed in 2.23s
11:13AM WRN leaks found: 32

Meanwhile, infisical scan --verbose prints fuller details for each secret such as what was leaked, who leaked it, and where it can be found.

Finding:     JWT_SECRET=138817529bf4c73772243214896b168df91d3501d56757dbcbeda31d3da7acad
Secret:      138817529bf4c73772243214896b168df91d3501d56757dbcbeda31d3da7acad
RuleID:      generic-api-key
Entropy:     3.593139
File:        .env.example
Line:        7
Commit:      622ed0d1b294bd31bd771eb40115d337
Author:      John Doe
Email:       johndoe@gmail.com
Date:        2022-12-24T19:21:21Z
Fingerprint: 622ed0d1b294bd31bd771eb40115d337:.env.example:generic-api-key:7

Scanning uncommitted files

Scanning your entire git history is nice but what if you only wanted to scan your uncommitted files while developing?

No worries, the infisical scan git-changes subcommand has you covered.

# scan your uncommitted files
infisical scan git-changes

# display the full secret findings for the uncommitted files
infisical git-changes --verbose

Setting up automatic scanning before each commit

While scanning your codebase for leaked secrets manually works, the best teams have pre-commit hooks set up that automatically detect any leaked secrets before allowing you to commit any code. This one-time setup makes it impossible for developers to forget to manually run the scan command before each commit and, in doing so, enforces that committed code is clear of leaking any secrets.

To install the git hook, run the following command in your local git repository.

infisical scan install --pre-commit-hook

That's it! You now know how to detect secrets in your codebase and prevent leaks from happening.

That said, I'd encourage you to check out the full documentation for more information and advanced options on using the CLI for secret scanning. It includes how to configure the scan to ignore select secrets or even those matching a pattern.

Conclusion

Leaking secrets can happen to the best of us, and this risk scales with the size of a team; the more engineers you have onboard, the higher the risk of one developer accidentally introducing a new secret and accidentally committing it to source control.

However, there are practices that you and your team can enforce to prevent secrets leaks from happening and using a CLI to detect secrets in your codebase is one such practice. With minimal effort, you can set up a pre-commit hook that automatically scans for leaked secrets and prevents developers from committing code before removing that secret or acknowledging it as intentional.

Magically use environment variables without a .env file

BlackMagiq — Thu, 18 May 2023 13:51:26 +0000

TLDR: You can build or use a pre-made CLI together with your application's start command to inject environment variables for local development and avoid using a .env file altogether.

Recently, I wrote about fetching environment variables for applications so teams always have access to the right variables. The method was simple: store environment variables in a secret management platform, then have the application fetch them back and cache them at runtime. This, however, came with a nuance of having to manage yet another token to fetch these variables.

In this article, however, I discuss an enhanced approach specifically for local development to inject environment variables into an application without needing a .env file at all.

It sucks to fetch environment variables back for an application if the method still requires managing one environment variable

This is a common complaint and debate amongst developers considering to use a secret management platform to store their environment variables only to find out that they still need to manage a token to gain access to the platform.

While the approach of storing a token in a .env file and using it to fetch back environment variables tamed secret sprawl, it did little to improve on the security front. After all, you could still leak the token to source control and, if not revoked, bad actors could use it to access your environment variables.

I'm glad to say, however, that you can use a secret management platform and fetch environment variables back without needing to store another proxy token. Here's how:

You can inject environment variables into your application using a CLI

Let me explain.

It turns out that your application running in local development is just a process and it's possible to pass environment variables into that process as it starts up. This means that you can build a platform-agnostic CLI to fetch back environment variables from a centralized source and wrap it around your application's start command. Couple the CLI with authentication logic and proper credential storage on your system and you can have a secure workflow that pulls and injects environment variables into your local development process.

Moving forward, members of your team only need to install the CLI onto their machine, authenticate with the secret management platform via the CLI, and start up their application via CLI with environment variables injected - Wizardry.

That's enough, show me your recommendation, gimme the code

While you can code the CLI yourself and connect it to an existing secret management platform, I wouldn't recommended it for most developers since this isn't a quick endeavor. Instead, I'm going to be using the Infisical CLI to pull and inject environment variables in local development from Infisical, a popular open-source, end-to-end encrypted secret management platform that you can store environment variables with.

It's worth noting that Infisical is self-hostable on your own infrastructure and well-documented.

Getting started

Before we can fetch environment variables back into your application, you need to add them to a project in Infisical Cloud or in a self-hosted instance of Infisical.

Okay, let's get started.

First, install the CLI onto your machine.

MacOS (using brew):



brew install infisical/get-cli/infisical

Windows (using scoop):



scoop bucket add org https://github.com/Infisical/scoop-infisical.git



scoop install infisical

Arch Linux (using yay):



yay -S infisical-bin

Check out the documentation for more installation options like Redhat/CentOS/Amazon and Debian/Ubuntu if your system is missing above.

Next, authenticate with Infisical using the CLI:



infisical login

This command authenticates the CLI with Infisical Cloud or your self-hosted instance and stores credentials in your system keyring.

Next, navigate to the root of your project and run the initialization command:



infisical init

This command creates a infisical.json file holding a reference to your project in Infisical Cloud or self-hosted instance; the file can be committed to source control.

Finally, start your application in local development with the help of the CLI:



infisical run -- <your application start command>

If you're using a self-hosted instance of Infisical, you can run this command instead:



infisical --domain="https://your-self-hosted-infisical.com/api" run -- <your application start command>

This command fetches environment variables back from the development environment of your project in Infisical and starts up your application with the variables injected. The environment variables become available in the application environment. For example, they're accessible on process.env in Node.js or on os.environ in Python.

While the CLI comes with a few helpful subcommands and flags that you can read more about in the documentation, in practice, the basic command above is all you need 99% of the time.

Here're some examples for common frameworks:



# example with node
infisical run -- node index.js

# example with node (nodemon)
infisical run -- nodemon index.js

# example with Next.js
infisical run -- npm run dev

# example with flask
infisical run -- flask run

# example with django
infisical run -- python manage.py runserver

# example with laravel
infisical run -- php artisan serve

# example with rails
infisical run -- bin/rails server

# example with .NET
infisical run -- dotnet run

That's it!

Now you can use environment variables in local development without a .env file and feel confident that your team will not leak anything to source control. You're now also able to view all the environment variables for your application from one central place and avoid any missing environment variables.

How do I trust a secret manager?

I'm including this section because it has been helpful to developers in past articles.

Adopting a secret manager can be daunting at first and rightfully so. You have to trust that the solution you're working with is secure, that the vendor won't view or tamper with your data, and that the underlying infrastructure will work and deliver environment variables reliably when needed. There is, however, a right tool for every task and using dedicated tooling to manage environment variables is surely better than any manual process that's either not efficient or prone to error.

The analogy I like to use is password managers like 1Password and Bitwarden. Before using one, I was content with managing dozens of passwords manually and occasionally forgetting them. The minute I started to face password sprawl, however, I was compelled to upgrade my workflow and sign up for a password manager.

Similarly, in the realm of secret managers, when you're a solo developer, you can use manual approaches to store your environment variables. The minute, however, you start to scale to above five engineers, you start to face secret sprawl and that's where a secret management solution can really help. My suggestions for selecting one:

Pick a solution that's either open source or made by the big cloud providers; it will be more resilient that way and you can inspect and, in the open source case, make changes to the code if needed.
Pick something that provides a good developer experience for the size of your team. If you have site reliability engineers and teams for process, you can afford to go with more complex tools. Otherwise, if you're a smaller team, go frictionless and focus on building the rest of your application.

Conclusion

We developers often debate the practicality of using a secret management platform when faced with the need to yet manage another token to access environment variables or secrets. This unfortunately comes off as counterintuitive to many developers who then hesitate on adopting a secret management solution.

In this article, however, I demonstrate a workflow using a CLI that fetches environment variables and injects them into any application upon starting up; the method avoids storing any proxy token in a .env file. With this CLI approach, development teams stop risking leaking environment variables to source control by not storing any variables in plaintext in .env files on their machines.

Node.js: Replace your .env file with this awesome tool at scale

BlackMagiq — Thu, 11 May 2023 14:41:46 +0000

When developing applications, handling sensitive information like API keys, database credentials, and configuration settings is crucial. Ensuring these environment variables are managed securely, efficiently, and in sync across the development lifecycle is a common challenge and let’s face it, .env files don’t cut it anymore, well, for many reasons.

In this article, I discuss the optimal way to manage environment variables for your Node application with Infisical at scale.

What is Infisical?

Infisical, an open-source, end-to-end encrypted secret management platform that you can store environment variables with. It’s fully self-hostable on your own infrastructure, well-documented, and insanely beautiful.

Its Node SDK lets you fetch back environment variables at runtime whether it be in local development or production.

Getting started

Before we can fetch environment variables back into your Node application, you need to add them to a project in Infisical Cloud or in a self-hosted instance of Infisical.

Okay, let’s get started.

First, install the infisical-node package in your project:

$ npm install infisical-node --save

Next, import the SDK and create a client instance with your Infisical Token:

import InfisicalClient from "infisical-node";

const client = new InfisicalClient({
  token: "your_infisical_token"
});

To ensure optimal performance, I’d recommend creating a single instance of the Infisical client and exporting it to be used across your entire app. The reason is because the Node SDK caches every secret and updates it periodically, reducing excessive calls; this built-in caching makes syncing environment variables seamless at scale.

I’d also recommend storing the Infisical Token in a .env file in local development or as the only environment variable in production. This way, you don’t have to hardcode it into your application and can use it to fetch the rest of your environment variables.

Now you can use the client to fetch secrets for your application on demand:

app.get("/", async (req, res) => {
  const name = await client.getSecret("NAME");
  res.send(`Hello! My name is: ${name.secretValue}`);
});

That’s it!

Now whenever your application needs an environment variable, it can request it from Infisical on demand. You’re now able to view all the environment variables for your Node application from one central place and avoid any missing environment variables.

I’d recommend reading into the documentation more to learn more about how to manage environment variables effectively.

But, you’re still using a .env file…

One question came up when I first posted this article being: “If the Infisical Token used to fetch other environment variables is stored in a .env file, then doesn’t that defeat the purpose of the tool?”

The answer is no.

As mentioned, a big point of using the recommended approach is to keep your environment variables in sync across your team. Oftentimes, new environment variables get introduced to a codebase and .env files don’t get updated across the team; as a result, applications crash. The issue compounds when your infrastructure gets big and a problem known as “secret sprawl” emerges. As such, Infisical provides you the ability to centralize your environment variables so you can update them in one place and have them delivered back to your team and infrastructure from development to production. This is different from what a lot of people do which is directly store dozens of environment variables in .env files.

Lastly, from a security perspective, leaking a revokable token is much better than leaking a dozen set of raw environment variables; you avoid leaving any direct traces in source control.

Conclusion

Infisical is an awesome platform to streamline environment variables for you and your team. Its open-source and has a handy Node SDK that can be used to fetch environment variables back to your Node applications on demand.

Doing much better than your .env file

BlackMagiq — Wed, 10 May 2023 13:21:26 +0000

Six months ago, I advocated for everyone to stop using .env files in favor of approaches using secret management platforms. Since then, I've learned and experienced much more about how to manage environment variables effectively. In this article, I argue for why we should reduce the practice of storing all your application environment variables in a .env file and move to a more sophisticated approach for managing them in local development.

Before you roast me, I want to make clear that my position is not that you have to ditch the .env file entirely. I'm asserting that your application's environment variables don't need to be stored directly in a .env file. This doesn't mean, however, that you can't store a token in it that pulls in the rest of your environment variables at runtime.

I also, by the way, want to make clear that this article is intended for software development teams and not security and devops teams that already have this figured out; this article is also not intended for solo developers who can feel free to use .env files.

Don't fix what's not broken

In the beginning, developers hardcoded environment variables into their codebase. After realizing that hardcoding them into source control was suboptimal, we introduced .env files for separation of concerns that is to split sensitive data from the rest of code.

In practice, we'd create a .env file, add environment variables to it, and .gitignore the file. We'd start up our applications and read the environment variables into them in local development. Unfortunately, we still teach developers this as a staple of software development and proponents tend to say "don't fix what's not broken."

If you're one of these stubborn proponents, then stop reading; if you're open to improvement and breaking tradition, go on.

Developers have unmet needs

At its core, .env files are broken and we've all experienced it to some degree depending on the scale of our applications and infrastructure; folks deep in the security and devops department know it. In order to understand my argument for why the current usage of .env files is broken, it's important to unlearn tradition and break the topic of managing environment variables locally down to first principles. Let's explore what developers need:

Reliability: We want to make sure that developers have access to the right environment variables at all times. Oftentimes, developers introduce new environment variables and forget to update other developers on the team about the new values. As a result, developers start up their applications and … crash.

Security: We want to keep our environment variables safe and out of source control. While we want to keep bad actors away from our environment variables, since this article focuses on .env files that are typically used in development and contain less critical environment variables, I make this my second point.

Note that my points above generalize across the entire development cycle from development to production and these needs grow with the scale and complexity of application infrastructure. In any case, we seek reliability and security when dealing with environment variables. We want the right environment variables in the right hands and places safely.

Okay so what's wrong with .env files?

While .env files prevent developers from hardcoding and committing environment variables to source control, they fail the reliability criteria.

In a two person team, it's easy to maintain .env files because you don't have to share environment variables with many other developers. As your team size grows, however, you start to run into the secret sprawl problem that I mentioned earlier and it becomes unwieldly to manually share environment variables.

Take for example a microservices architecture where now you have many services, each demanding their own set of environment variables. Here, developers frequently face a phenomenon known as secret sprawl which is when environment variables get scattered across infrastructure and it all becomes unwieldy to manage.

Developers need to centralize their environment variable management

Naturally, the solution is to store environment variables in one centralized location and have developers fetch them from that place back to their applications. It turns out that centralization is indeed what bigger teams do but even then there is still variation. Sometimes, teams store environment variables in password managers; other times, teams encrypt them and store them directly in a private repository on GitHub.

These methods work but they typically present an extra step of friction that is having to manually grab or trigger refetching environment variables before starting up an application; you could build a custom script that does the job of pulling secrets and injecting them into an application but that in itself presents a step of friction for you.

What developers need is a frictionless tool that automatically pulls and injects environment variables into an application. For this, smarter teams use a dedicated tool that's called a secret manager, built to manage environment variables. These solutions like Infisical, Vault, AWS Parameter Store, Google Secret Manager help securely store environment variables and deliver them back to your infrastructure reliably. When set up correctly, these tools can serve environment variables efficiently across any complex microservices infrastructure from local development through CI/CD pipelines and to production.

Most developers don't know better

Unfortunately, beyond the security and devops crowd, most developers have never heard of secret managers and the proof is in the number of YouTube videos and views for the topic of secret management versus anything else. Let's face it, secret management isn't considered to be a sexy topic the way artificial intelligence and ChatGPT is.

The reason secret managers aren't popular is because they are unwieldy. That's right, the tools built to solve the secret sprawl problem I outlined earlier have histoically introduced new problems with undesirable characteristics. On one hand, there is difficulty in getting started; no one wants to watch a course and read through tens of pages of documentation just to learn the basic concepts of how store and fetch back environment variables from a secret manager. On the other hand, the easy tools are closed-source and, well, how can we trust them?

Okay, I know that I'm going to get a lot of backlash now from die-hard Vault fans but let's be real here. If any secret management tool was as simple as installing the dotenv package, importing it, and starting up your app, then we'd be seeing mass adoption of secret managers everywhere.

But we're not.

Instead, we're stuck with 30M+ downloads per week of the dotenv package for Node.js alone and a sad statistic of 10% of organizations worldwide having adopted secret managers.

Developers would love secret managers, if only they were easy to use and trustworthy

At the beginning of this article, I made clear that my position is not that you have to ditch the .env file entirely. I'm asserting that your application's environment variables should not be stored directly in a .env file.

While I have many tips and strategies for managing environment variables, the major one I'll allude to for this article is a hybrid approach that is to use a secret manager in conjunction with a .env file.

Instead of storing your application environment variables directly in a .env file, you can store them in a dedicated secret management platform, preferably one that's easy to use with little overhead. Then, store an access token in the .env file that your application can use to fetch environment variables from the secret manager at runtime. That's a simple adjustment you can make to improve the reliability of receiving the right set of environment variables at all times across your team.

That's enough, show me what you're suggesting, gimme the code

To keep things simple, I'm going to use Infisical, an open-source, end-to-end encrypted secret management platform that you can store environment variables with. You can also use another secret manager like Vault or AWS Parameter Store that each have their own SDKs that you can use to achieve the recommendation.

I'm picking Infisical because it's self-hostable on your own infrastructure, well-documented, and beautiful. It's also worth noting that it's self-hostable on Fly.io and Render as well as AWS, Kubernetes etc.

Anyways, we'll be using the Node SDK to fetch back environment variables at runtime with this example, though there are other SDKs and methods available.

Getting started

Before we can fetch environment variables back into your Node application, you need to add them to a project in Infisical Cloud or in a self-hosted instance of Infisical.

Okay, let's get started.

First, install the infisical-node package in your project:



$ npm install infisical-node --save

Next, import the SDK and create a client instance with your Infisical Token:



import InfisicalClient from "infisical-node";



const client = new InfisicalClient({
  token: "your_infisical_token"
});

To ensure optimal performance, I'd recommend creating a single instance of the Infisical client and exporting it to be used across your entire app. The reason is because the Node SDK caches every secret and updates it periodically, reducing excessive calls; this built-in caching makes syncing environment variables seamless at scale.

I'd also recommend storing the Infisical Token in a .env file in local development or as the only environment variable in production. This way, you don't have to hardcode it into your application and can use it to fetch the rest of your environment variables.

Now you can use the client to fetch secrets for your application on demand:



app.get("/", async (req, res) => {
  const name = await client.getSecret("NAME");
  res.send(`Hello! My name is: ${name.secretValue}`);
});

That's it!

Now whenever your application needs an environment variable, it can request it from Infisical on demand. You're now able to view all the environment variables for your Node application from one central place and avoid any missing environment variables.

How do I trust a secret manager?

Adopting a secret manager can be daunting at first and rightfully so. You have to trust that the solution you're working with is secure, that the vendor won't view or tamper with your data, and that the underlying infrastructure will work and deliver environment variables reliably when needed. In many ways, it does feel easier to store environment variables in a decentralized fashion or perhaps centralized but encrypted in a private repository on GitHub.

There is, however, a right tool for every task and using dedicated tooling to manage environment variables is surely better than any manual process that's either not efficient or prone to error; this is especially true in cases where your infrastructure is sprawled out.

The analogy I like to use is password managers like 1Password and Bitwarden. Before using one, I was content with managing dozens of passwords manually and occasionally forgetting them. The minute I started to have many different passwords across different services, each satisfying a different requirement, I faced a password sprawl that compelled me to sign up for a password manager because I couldn't keep hitting the "forget my password" button.

The same is true for secret managers. When you're a solo developer, you can use manual approaches to store your environment variables. The minute you start to scale above five engineers, you should start thinking about maintaining your secret management hygiene. While there is no 100% guarantee for the reliability of any process, you can get close to 99.99% and my suggestions for selecting a secret manager are as follows:

Pick a solution that's either open source or made by the big cloud providers; it will be more resilient that way and you can inspect and, in the open source case, make changes to the code if needed.
Pick something that provides a good developer experience for the size of your team. If you have site reliability engineers and teams for process, you can afford to go with more complex tools. Otherwise, if you're a smaller team, go frictionless and focus on building the rest of your application.

Conclusion

For the longest time, developers defaulted to storing all their environment variables in a .env file largely because it was the most frictionless way to get up and running locally. Alternatives like other secret management platforms were too cumbersome to adopt and never became mainstream because they were too difficult to use/learn and intended for the security and devops crowd.

In this article, however, I make clear that the approach of storing all of your environment variables in a .env file lacks reliability when it comes to keeping your team's environment variables in sync. By using a dedicated secret manager, preferably one with little overhead, and instead keeping a single token in your .env file that your application can consume and use to fetch the rest of your environment variables, you can preserve the security benefits of separating sensitive data from your codebase whilst ensuring reliability of environment variables across your infrastructure.