DEV Community: Ziad

How does Bokeh, the Python Interactive Visualization Library, Secure its Open-Source Repositories?

Ziad — Tue, 09 Nov 2021 12:31:45 +0000

Open-source is everywhere, it is one of the driving forces of software innovation from the academic to the enterprise world (75% of codebases audited by Synopsys in the 2021 OSSRA report rely on open-source components. Its prevalence in commercial software is reaching unprecedented levels, to the extent that the European Commission has recently identified it as a public good, in a recent study assessing its impact on the region's economy.

But the interstitial nature of open-source in modern software also makes it a subject of security and compliance concerns, as it is capable of exposing organizations that use it to a host of unknown risks and vulnerabilities. Most discussions we are hearing today around security in this space are focused on the identification, fixing, and remediation of vulnerabilities - all seen from the "consumer" perspective.

This time, we decided to go on the other side of the fence. We had the pleasure to exchange a few words with Bryan Van de Ven, co-creator and core maintainer of the Bokeh project, a Python library for data visualization. Bryan gave us an insider look at how open-source maintainers such as himself shield their projects against the attempts of malicious actors trying to exploit security gaps. The goal of attackers is straightforward: introduce vulnerabilities downstream, and in turn, attack the software supply chains that depend on the same open-source packages and libraries.
Bokeh, the interactive visualization library for the modern browser

Bokeh (pronounced /ˈboʊkeɪ/ BOH-kay) is an interactive visualization library for modern web browsers, written in Python. It provides elegant and concise construction of plots while maintaining high-performance interactivity over large datasets. Bokeh can help anyone who would like to quickly and easily make interactive plots, dashboards, and data applications.

Before starting his endeavor with Bokeh in 2012, Bryan was no stranger to open-source libraries. He authored the conda package manager and worked full-time at Anaconda on its distribution, simplifying package management and deployment for more than 25 million users worldwide. Inspired by his previous contribution to Chaco (Python data visualization library) and the rise of JavaScript-heavy frameworks for frontend in the early 2010s, Bryan teamed up with Peter Wang to offer an alternative for Python developers, who were working on interactive data applications for the modern browser. The rest is history.

Why is security important for open-source projects like Bokeh?

With 37,000+ public GitHub repositories declaring its use and 2.5 million monthly downloads, Bokeh has made a name for itself. Bryan believes this has a lot to do with the project's early days, where patience, responsiveness, and receptiveness to the contributions from a community in its embryonic stages play a determining role in later successes.
Now comes the difficult part, ensuring Bokeh is open-source code on which individual developers and enterprise teams alike can safely build. Together, we discussed some of the threats keeping open-source maintainers up at night:

Typosquatting

This attack implies bad actors pushing malicious packages with similar names to the original one to a trusted registry and crossing fingers for users to fall for their dirty trick. Packages hosted on the npm and PyPI registries have been notable targets, reminding us that developers too can fall prey to a different breed of phishing.

With more than 2.5 million monthly downloads shared between conda and pip and ~150 million requests for BokehJS resources at cdn.bokeh.org every year, the bokeh library looks like it is ripe for the picking, at least from an attacker's perspective.
Once the malicious packages are installed and executed in runtime, attackers can:

Siphon environment variables for further lateral movement (crossenv attack)
Hijack cloud computing resources for crypto-mining (PyPI attacks in 2021)

Unfortunately, there is not much the package maintainers can do here. It is recommended users scan their project dependencies to verify their integrity with tools such as WhiteSource, Sonatype, Snyk or Vdoo.

Compromised build & deployment pipelines

Like other developers, core maintainers may not be application security experts. They are not particularly immune to introducing bugs in their code or inadvertently committing secrets in their project's git repository. Insecurely handling credentials can lead to their public broadcasting on GitHub, with consequences for projects like Bokeh ranging from Cloud Jacking (e.g abuse of AWS cloud computing resources) to compromising their publishing rights on PyPI or Anaconda.

With a team of 5 to 6 core maintainers and a total number of ~500 contributors spanning the project's life, things can get messy for Bokeh if credentials are spilled. Bryan assures us only a handful of contributors are cleared to handle the project's secrets and the team uses GitHub's Encrypted Secrets to store and retrieve credentials in their GitHub Actions CI pipelines.

In addition to this measure, Bokeh is hardening the CI pipelines with GitGuardian, effectively catching any leaked secrets as soon as the code wrapping them reaches the build stages. More on this later.

Threats to Asset Integrity

Open-source software is not just a public code repository hosted on GitHub. It is also a collection of assets, guiding both starters and experienced members of the community through their learning journey. To name just a few from Bokeh:

Website
CloudFront CDN for distributing the BokehJS runtime at cdn.bokeh.org
Public documentation
Blog (hosted on Medium)
Social media accounts (YouTube, Twitter…)
Community and support (Discourse)
Other tools (Zapier for automation, Pingdom for forms and surveys)

Unlike open-source code, such services will never run in the users' production environments. But they are equally important in terms of the protection they deserve since they could offer openings to attackers with a strong taste for Social Engineering tactics.

For everything other than the documentation, Bokeh is using the free tier 1Password offers to open-source projects to securely store and manage all their credentials.

How does GitGuardian shield Bokeh's public repositories?

Bryan tells us his first encounter with spilled credentials was in a coffee shop, coincidence? While testing new automation functions and uploading documentation to the CloudFront CDN, he accidentally pushed an AWS secret key and its token.

Luckily, GitGuardian had his back. Our pro-bono service caught this secret in the public bokeh repository and sent him a timely alert with the incident's details.
Bryan was able to remediate in time, revoking and rotating his secret, and finally rewriting the git history to remove all evidence of the leak. In less than an hour, his fast response allowed him to nullify the damage and close the window for any exploits.

Since this incident, his team has had GitGuardian's CLI, gg-shield, running on every Pull Request, in their GitHub Actions workflows. The secret scanning checks and their results are also displayed in the VCS, making sure all contributors have full and equal visibility - without ever leaving their developer environments.

What's next for Bokeh and GitGuardian?

The Bokeh team is actively adding core contributors and one of the recent joiners is a git hooks power-user, which works out perfectly since GitGuardian offers secret scanning at the pre-commit level. The Bokeh team can now shift further left and prevent secrets from reaching their VCS and CI pipelines.
Bryan was also excited about our latest product updates, namely the Presence and Validity checkers.

These features provide hints as to whether the leaked secrets are still present in the git history or not and if they are valid, meaning they can still be exploited by malicious actors.

Interesting! This will help teams modulate the fervor of the response to incidents and focus their efforts where they are most needed.

What about you? Are you actively maintaining an open-source project or looking to make your source code public?

Here are a few resources on securing open-source code that can get you started:

How is Carrot getting Bitcoin to the next billion people with the help of Forest Admin? 🥕

Ziad — Wed, 23 Dec 2020 12:04:35 +0000

It is really our sort of secret plan, to get Bitcoin to the next billion people!

Tyler Evans, co-founder & CTO of BTC Inc.

The mission at hand is daunting, if not impossible. But those who have met Tyler Evans know he wouldn't flinch at such a challenge.

Tyler is the co-founder and CTO of BTC Inc - the media group behind Bitcoin Magazine, providing news and commentary on Bitcoin and blockchain technology since 2012 to millions of readers across the globe. With a deep background in the technology and an enthusiastic community, BTC Inc. was in the perfect position to build the next on-ramp for Bitcoin.

In 2020, he and his team at BTC Inc. decided to launch Carrot, a rewards platform where users can earn bitcoin simply by supporting the brands and creators they love. Creators of all sorts (writers, podcasters, open-source collectives...) can publish challenges to be carried out by their audiences and offer bitcoin for their completion.

For its public release, Carrot chose Forest Admin to build its admin panel. Here's why and how they did it.

Carrot needed serious internal tooling to launch its app 🚀
Carrot's alpha phase saw it distribute millions of sats* to more than 10,000 users, proving the demand for a micropayment rewards platform.

Launching its public release however, is a different matter. "We needed to stop interacting directly with the database and APIs and build the right set of internal tools for our team", said Tyler. This would prove critical for the Carrot team to provide responsive customer support at scale, mainly:

Reviewing the challenges and bounties sent by creators
Moderating user submissions to the challenges
Triggering bitcoin payouts for the winning submissions

Fortunately, Forest Admin was here to help.

The day-to-day operations on Forest Admin 🌲

Reviewing the bounties sent by creators

The first conversations we had with Carrot revolved around making the bounty creation process as streamlined as possible. For this to happen, external partners had to be granted access to Forest Admin to submit challenges with little help or no support.

Leveraging Forest Admin's team-based permissions, Carrot restricted their access to a user-friendly form where it was possible to write a description, set the reward size and upload a few images.

Once submitted, the challenges were reviewed by the Carrot Operations team and the approved ones would become visible to all users on the platform.

Moderating users' submissions and triggering payouts

Users scrolling through the platform's available bounties were then able to pick their favorite and give it a try. One typical challenge was to complete a product testing survey where a user provides valuable feedback in return for a fraction of a bitcoin.

This type of submission goes through Typeform and its content is viewed in Forest Admin by the creators and Carrot team. Submissions are moderated to filter out the spammy and unhelpful ones.

Once a submission is accepted, it's payday for the user! Carrot operators fire a call from Forest Admin to their APIs, sending the bitcoin micropayment over a Lightning Network channel.

Measuring the traction 📈

Aside from streamlining operations, measuring the platform's traction was essential to nailing the public release. For this, the Carrot team created dashboards using Forest Admin's no-code chart builder.
By displaying the total number of users, live bounties and tracking bitcoin payouts amongst other KPIs, they were able to quickly gain insights before committing to any important decision.

Delivered in record time

Forest Admin was a no-brainer for Carrot. It provided them with the right set of tools for a stress-free public release. Its ease of set up and customizability also allowed them to build all business workflows in record time; less than 14 days were needed to go live!

We were able to go so quickly with Forest Admin's help from an out-of-the-box solution to something fairly custom that did exactly what we wanted for Carrot without us having to change our workflows or adapt our technical stack to fit!

Tyler Evans, co-founder & CTO of BTC Inc.

All we can hope for Tyler and the Carrot team now is to run a successful launch and distribute more bitcoin through Lightning Network ⚡ than it has ever been done before!

Is your company operating in the Crypto & Blockchain space? Let's get in touch!

While we continue to power the most exciting startups in the FinTech industry, we're also excited to build our presence in the Crypto space and are on track to become one of its cornerstone tools.

We're eager to learn what use cases you have in mind and would love to take a chance at demonstrating what Forest can do for you.

💻 Book a 1:1 demo with our team here

Curious to know why other FinTechs chose Forest?

Glossary

sats or satoshis* = smallest bitcoin unit, equal to 100 millionth of a bitcoin

Build internal tools with privacy in mind using Forest Admin 🌲

Ziad — Fri, 04 Sep 2020 11:34:39 +0000

Introduction

Internal tools like admin panels are oft overlooked. Because time spent on building them equates to time not spent on shipping the next shiny feature.

So what happens when non-technical teams need to see or manipulate an app's data? They are stuck with rushed and frustrating tools.

Forest Admin empowers developers to build and ship admin panels. Ones that are easy for non-developers to use, easy to adapt, and easy to extend. For a fraction of the time it would cost to develop them in-house.

Here we'll explain the architecture we chose to provide such a service. Without forcing you to choose between the privacy and security of an in-house solution and the convenience of a SaaS. Yes, you can get the best of both worlds and no, there's no catch.

How does Forest Admin work?

Forest Admin provides you with an admin panel for your internal teams to see and manage your data. But what happens under the hood to build the interface for your teams?

The whole architecture consists in 4 different components as shown below. The database, the admin backend, the Forest Admin API server and the Forest Admin UI server.

1. Your database

The database is the most vital part of any stack. It's where your application data lives and resides. It reflects real world transactions from your customers.

It's also the first building block needed to create your admin panel on Forest Admin. Because it will be the main data source for it.

Forest Admin currently supports RDBMS such as MySQL, Postgres & MsSQL. In the NoSQL databases family, it supports MongoDB.

2. Forest admin backend

When you install Forest Admin, you generate a node.js application on your local machine. It includes a RESTful API that connects to your database. We call this app the admin backend. It feeds all the data to your admin panel interface.

To get more technical:

it translates client requests (from the user browser) into queries to your database.
it also provides the Forest Admin API Server with the information needed to build the User Interface. This information includes table names, column names and types, and relationships.

A JSON file called the forestadmin-schema.json carries this metadata within the admin backend.

3. Forest Admin API server

The Forest Admin API Server stores the information to build the user interface. This includes both the database structure (sent by the admin backend) and the UI customization made by the user.

To get more technical, the information stored includes:

Display & Order - Which tables and columns should be displayed or hidden? In what order should the columns appear in the 'Table' view?
Collection Settings (permissions) - Are the records in this table read-only? Can they be deleted? Can they be exported in a .csv file?
Widget preferences - Which UI component should be rendered for each column (e.g a file viewer for a column that contains images urls).
Chart configurations - How are the dashboard charts configured and in which position should they appear?

The Forest Admin API Server also manages the Forest Admin app's logic like user authentication or billing.

4. Forest Admin UI server

The Forest Admin UI server stores static assets. These include HTML documents, CSS stylesheets and JS script files. It provides the UI components needed to build the interface that displays the data.

Now that you have a good overview of the architecture, you may be wondering how the pieces actually interact to make it work.

How do all the pieces come together?

Let's figure it out by checking the http calls made between each of the above-mentioned elements when operating a Forest Admin project. Namely calls made:

between the user's browser and the Forest Admin servers (both UI and API servers),
between the user's browser and the admin backend,
between the admin backend and the Forest Admin API servers.

Calls made from the user's browser

The following details the calls made by the browser when a user accesses the admin panel from his browser (at app.forestadmin.com).

To the Forest Admin UI servers

Calls need to go out to the Forest Admin UI server to fetch static assets including:

HTML documents
CSS stylesheets
JS scripts
A map of the assets

To the Forest Admin API servers

Calls need to go out to the Forest Admin API servers to retrieve information regarding:

the user logged in,
the project he's logged into,
the environment he's logged into,
the configuration of the rendering to be displayed (i.e. the configuration of the UI),
the widgets configuration,
the billings info of the project,
any updates happening on the UI configuration. This is done through websockets to notify the user a new version of his tool is available.

To the admin backend

Calls need to go out to the admin backend to retrieve/ modify data from the database including:

GET calls to retrieve a list of records, the count of a list or the details of record,
PUT calls to modify a record,
POST calls to create a new record or trigger a custom action,
DELETE calls to delete records.

Calls made from the admin backend

To the database

When calls are made from the browser to the admin backend, the latter translates the call into a database query.

To the Forest Admin API servers

In order to ensure that the UI reflects the structure of the database, the admin backend needs to send calls containing the information from the forestadmin-schema.json to the Forest Admin API servers. This file is sent upon every restart of the admin backend server.

At the startup of the admin backend and periodically afterwards, calls are also made to the Forest Admin API servers to retrieve permissions. This protects the data from being accessed by unauthorized users through curl requests for example.

Now that we've detailed how Forest Admin works, let's discuss why we went that way.

What are the benefits of such an architecture?

Build internal tools with privacy baked in right from the start

We can't store any of your data because we never get to see it. You own your admin backend. That means no Forest Admin backend (on our servers) proxying requests to your database.

You can host it in the cloud architecture you want. You can shield your admin backend to comply with any of your security requirements - VPN, VPC, DMZ...you name it.

Your internal tools, your rules

You own the admin backend and can extend it like any node.js app. Using JavaScript and npm packages, you can add any custom logic you want. When you create a record you want to add an automatic email alert on top of inserting a row in your database? You can do so by easily overriding the default create route from your admin backend for example.

This means you can also keep track of any changes using Git, containerize your app using Docker, deploy on your favorite Cloud Hosting Provider... You remain in control.

You do half the work, we take care of the other half

This architecture lets you benefit from improvements and new features by updating to our newer versions of the API. All it takes is one command npm install forest-express-[sequelize|mongoose]@latest.

You also benefit from continuous updates brought to the UI by refreshing your browser page. Again, one command Cmd+Shift+R.

So, what next?

Want to try our admin panel framework? Head here!🌲
Curious what it looks like? Take a peek at our live demo 💻
Interested in deep diving into the tech behind Forest? Peruse our documentation 📚