DEV Community: Copple

pgvector 0.6.0: 30x faster with parallel index builds

Copple — Wed, 31 Jan 2024 15:30:41 +0000

pgvector 0.6.0 was released today, with a significant improvement: parallel builds for HNSW indexes. Building an HNSW index is now up to 30x faster for unlogged tables.

This release is a huge step forward for pgvector, making it easier to tune HNSW build parameters and increase search accuracy and performance.

HNSW indexes in pgvector

We explored how HNSW works in an earlier post, so as a quick recap: HNSW is an algorithm for approximate nearest neighbor search. It uses proximity graphs and consists of two parts: hierarchical and navigatable small world. It operates over multiple layers with different densities or distances between nodes, where layers represent different connection lengths between nodes. Thus allowing HNSW to search, insert, and delete in linearithmic time.

pgvector parallel index builds

Prior to 0.6.0, pgvector only supported building indexes using a single thread - a big bottleneck for large datasets. For example, building an index for 1 million vectors of 1536 dimensions would take around 1 hour and 27 minutes (with 'm'=16, 'ef_construction'=200).

With parallel index builds you can build an index for the same dataset in 9.5 minutes - 9 times faster:

Performance comparison: pgvector 0.5 vs 0.6

We tested index build time with the dbpedia-entities-openai-1M dataset (1 million vectors, 1536 dimensions) to compare the performance of parallel and single-threaded index HNSW builds. At the same time, we verified that the resulting indexes are the same in terms of accuracy and queries per second (QPS).

We ran benchmarks on various database sizes to see the impact of parallel builds:

4XL instance (16 cores 64GB RAM)
16XL instance (64 cores 256GB RAM)

4XL instance (16 cores 64GB RAM)

This benchmark used the following parameters:

	0.5.1	0.6.0
mainenance_work_mem	30GB	30GB
max_parallel_maintenance_workers	-	15

max_parallel_maintenance_workers controls how many parallel threads are used to build an index. In further sections we will refer to the total number of workers, including the leader.

The index build time is 7-9 times faster for 0.6.0, while queries per second and accuracy stay the same for both versions:

v0.5.1: averaged 938 QPS and 0.963 accuracy across all benchmarks.
v0.6.0: averaged 950 QPS and 0.963 accuracy across all benchmarks.

16XL instance (64 cores 256GB RAM)

You can further improve index build performance using a more powerful instance (up to 13.5x for these parameters).

The index build time is not linearly proportional to the number of cores used. A sensible default for max_parallel_maintenance_workers is CPU count / 2 , the default we set on the Supabase platform. Accuracy and QPS are not affected by max_parallel_maintenance_workers.

Embeddings with unlogged tables

Building time can be reduced even further using unlogged tables.

An unlogged table in Postgres is a table whose modifications are not recorded in the write-ahead log (trading performance for data reliability). Unlogged tables are a great option for embeddings because the raw data is often stored separately and the embeddings can be recreated from the source data at any time.

One of the steps of index creation is the final scan and WAL writing. This is generally short but not parallelizable. Using unlogged tables allows you to skip the WAL, with an impressive impact:

ef_construction	Build time: v0.5.1	Build time: v0.6.0 (unlogged)	Improvement
64	38m 08s	1m 38s	23x
100	1h 06m 59s	2m 10s	31x
200	1h 27m 45s	3m 37s	24x

Getting started

pgvector 0.6.0 was just released and will be available on Supabase projects soon. Again, a special shout out to Andrew Kane and everyone else who worked on parallel index builds.

🚀 Learn more about Supabase

Supabase: what's new in September 2021

Copple — Tue, 05 Oct 2021 09:22:15 +0000

Did you know it's been 2 years since the first commit to Realtime, our real-time engine for Postgres? Before we even existed as a company!
We spent this month improving docs and content content, improving UX, and onboarding Developer Advocates!

Hackathon v2

To kick off Hacktoberfest, another Supabase Hackathon is happening right now. You've got another 7 days to be in to win a limited edition t-shirt.

Abort Requests

We added support for AbortController in our Javascript library so that you can abort long-running queries. [Docs]

Improved table management

We've made a number of changes to the Dashboard to expose some great features of PostgreSQL including:

Column types

We've improved the column Type field so that it supports your custom types.

Is Unique

We've made it simple to add a unique constraint.

Edit columns

By popular request, you can now view all columns in a table at a glance and edit them in bulk.

Cross-schema relationships

We updated our grid to support relationships across multiple schemas.

Improved Auth Docs

We've revamped the Auth docs - The docs are now broken down into Authentication and Authorization, and organized alongside our Deep Dive series.

Low Code demo

Low Code demo, Using Supabase with Clutch.io - @_dijonmusters ran a session at General Assembly showing how to use these two tools together to create apps using a low code approach.

https://youtu.be/5fsKMTeBKKY

Community

There was a lot of new content this month.

Videos

How I built the #10 product in 30 days - Gary Tokman [video]
Github OAuth flow with Supabase + React - Nader Dabit [video]
The Best Stack for Web Developers - Christopher Kapic [video]
Flutter + Supabase Course - Abhishvek [video]
React Native Mobile Auth - Aaron Saunders [video]
Self Hosting Supabase - Kelvin Pompey [video]
Teta Flutter Frontend builder - Teta Team [product]
Magic Link + Route Controls in Next.js - Nader Dabit [Video]
Supabase in 6 minutes - Nader Dabit [video]

Twitter

We hit 16.5k followers. Follow us there for advance frontend tips and 👁️⚡👁️

GitHub

Not far from 20K stars: github.com/supabase/supabase

Source: repository.surf/supabase

Discord

Our Discord is growing fast. Come hangout with 3500+ developers building on Supabase today: discord.supabase.com

Hiring

We're Hiring SREs. We're fully remote and we love Open Source. See open roles.

Coming Next

We're warming up for another Launch Week! Last time was "Launch Week II: the SQL". We're going to need another month to come up with a good pun again, so we'll aim for November.

Get started

Start using Supabase today: app.supabase.io
Make sure to star us on GitHub
Follow us on Twitter
Subscribe to our YouTube channel
Become a sponsor

Daily email of Wikipedia's Current Events

Copple — Sun, 26 Apr 2020 12:04:37 +0000

tldr: I built https://currentevents.email

Intro

For many years I haven't been reading any news at all. ^{*HackerNews doesn't count}

While this is great for my cortisol levels, I do feel a bit uninformed.

I recently discovered that wikipedia has daily summary of key events happening around the world.

I decided that is the perfect replacement for news. I've been reading this every day to keep updated. Although it's not hard, I do find the processes of opening the page everyday a bit tedious.

Wikipedia supports RSS, but I've never been much of an RSS user. And so while thinking of something to build this weekend I figured I would create a daily, automated email of the events listed on Wikipedia.

Here is the result: https://currentevents.email

How I built it

The build didn't actually take too long. These are the things I used.

1. Get the content

First I fetch the html content using cheerio. The content requires a bit of cleaning - removing some of the styling etc - but it was a fairly simple process. Here are the key parts:

const request = require('request')
const cheerio = require('cheerio')
const moment = require('moment')

const currentMonth = moment().format('MMMM_YYYY')
const URL_TO_PARSE = `https://en.wikipedia.org/wiki/Portal:Current_events/${currentMonth}`

request(URL_TO_PARSE, async (err, response, body) => {
  const $ = cheerio.load(body)
  $('*').removeAttr('style')
  cleanseLinks($) // Not included here
  styleHeaders($) // Not included here
  styleLists($) // Not included here
  styleLinks($) // Not included here

  // Loop through all days this month
  var startOfMonth = moment.startOf('month').format('YYYY-MM-DD')
  var today = moment().subtract(1, 'day')
  for (var m = moment(startOfMonth); m.isBefore(today); m.add(1, 'days')) {
    const currentDayEvents = $('.description', `div#${m.format('YYYY_MMMM_d')}`)
    console.log(currentDayEvents.html())
  }
})

2. Create the email template

Anybody who has tried to create a nice email quickly finds that it's nearly impossible to design something which works on all email clients.

Luckily i found this email template on Github which did all the hard work.

I simply removed the content in the email template and replaced it with the currentDayEvents.html() in the code snippet above.

3. Sending it to subscribers

I needed a transactional email provider for people to subscribe and to send the emails. I chose SendGrid. They provide a hosted signup page and also an API for sending the email.

4. Sending it every day

The solution to this one was quite cool. I discovered that Github Actions can be run on a schedule, so I created a action that runs at 1 minute past midnight every day. Here is the action:

name: 'Current Events Bot'

on:
  schedule:
    - cron: '1 0 * * *' # Run at 1 min past midnight every day

jobs:
  bot:
    runs-on: ubuntu-16.04 # Download the server
    steps:
      - uses: actions/checkout@v2 # Check out the code
      - uses: actions/setup-node@v1 # Set up node
        with:
          node-version: '12.x'
      - run: npm install # Install the modules
      - run: npm run build # Run the "build" script that I created
      - name: Commit files # Commit all the changes
        run: |
          git config --local user.email "action@github.com"
          git config --local user.name "GitHub Action"
          git commit -m "Today's events" --allow-empty
      - name: Push changes # Push the changes to the repo
        uses: ad-m/github-push-action@master
        with:
          github_token: ${{ secrets.CURRENT_EVENTS_TOKEN }}
      - run: SENDGRID_API_KEY=${{ secrets.SENDGRID_API_KEY }} npm run send # Run the email script that I created

Conclusion

It took a few hours to build, even though it was quite simple. I also bought the domain name and hosted the content on Netlify, so the set up took a bit of time.

Feel free to comment if you want to get the full code.

Loading JSON into Postgres

Copple — Thu, 16 Apr 2020 14:00:03 +0000

Today I had to load some JSON data into Postgres.

Postgres' COPY command it expects one JSON object per line rather than a full array.

For example, instead of a JSON array:

[
    { "id": 1, "name": "John Conway" },
    { "id": 2, "name": "Richard Feynman" },
]

It needs to be this format:

{ "id": 1, "name": "John Conway" }
{ "id": 2, "name": "Richard Feynman" }

It took me a surprisingly long time to get the data into Postgres, but the solution was fairly simple.

Here's how I did it.

Convert the data

This is done with one command:

cat input.json | jq -cr '.[]' | sed 's/\\[tn]//g' > output.json

Explanation

cat input.json - read the contents of the file
| jq -cr '.[]' - pipe JSON into jq and split it onto every line
| sed 's/\\[tn]//g' - [optional] remove tabs, newlines etc
> output.json - output to a new file

Importing the data

From here it's easiest to ingest the data into a JSONB column.

## Create a table
psql -h localhost -p 5432 postgres -U postgres -c "CREATE TABLE temp (data jsonb);"

## Copy your data in
cat output.json | psql -h localhost -p 5432 postgres -U postgres -c "COPY temp (data) FROM STDIN;"

That's it

20 seconds of reading, and 1 hour of my time. To get the data out of the table now you can use any of Postgres' amazing JSON support. For example:

select data->>'name'
from temp;

-- John Conway
-- Richard Feynman

Enjoy.

Quick tip: Clone a single folder from Github

Copple — Fri, 03 Apr 2020 09:31:04 +0000

Sometimes you find a repository on Github and you only really need a single folder.

How do you get it without cloning the entire repo?

Luckily Github just implemented sparse-checkout. This is a super-quick example of how it works.

Example repo

Let's say I wanted to clone Supabase's example Slack clone, which lives in the following folder:

├── connectors
├── examples
│   ├── slack-clone-basic # <-- github.com/supabase/supabase/examples/slack-clone-basic
├── libraries
├── web
└── README.md

Sparse Checkout

Here are all the steps I would need:

# 1. Copy an empty repo
git clone --no-checkout https://github.com/supabase/supabase

# 2. Move into the empty repo
cd supabase

# 3. Initialize sparse-checkout
git sparse-checkout init --cone

#4. Checkout the folder
git sparse-checkout set examples/slack-clone-basic

And viola! Now you have just the folder you want.

Troubleshooting

sparse-checkout was introduced in Git version 2.25. If you get see an error "git: 'sparse-checkout' is not a git command" then you probably need to update. You can use git --version to see what version you're on.

If you're on Mac (and used homebrew to install git) you can run brew upgrade git.

Design Tips for Developers

Copple — Sat, 14 Mar 2020 02:37:05 +0000

Developers aren't well-known for their design skills. I suspect that poor design/UX in open source projects is the primary reason they aren't more successful. Usually the projects are technically brilliant - fast, well tested, and well maintained. But without a loveable interface users won't get hooked.

Perhaps you're the type of developer who is put off by any blog post with "Design" in the title. If you are, then I will say this up-front: design is about doing less rather than doing more. I don't advocate over-design, and I recommend erring on the side of simplicity. This blog post will only elaborate this point.

As with anything, being "20% good" at design will get you 80% of the way there. This article outlines some of the tools and tips I have picked up over the years that will make you a "20%" designer. I'm not going to go into UX (even though it's more important) because I think learning how to think in simple design patterns is the baseline for creating usable products.

Note: this article was modified from my original blog post: https://paul.copplest.one/blog/design.html

Contents

Building a design-y sense
Creating a logo
- Designing an Icon
- Designing the logo text/name
Colors
Fonts
Icons
Images and Illustrations
- Illustrations
- Images
Branding
Using a CSS framework
Design tools
More reading, tips, and tools

Building a design-y sense

This step isn't necessary but it's the thing that gave me the most improvement for the least effort. It will improve your "design sense", and give you an idea of your personal design style. The steps are:

Visit Dribbble (I have no affiliation) and sign up for their weekly newsletter
When you receive the weekly newsletter, click the "Most Popular Shots" link
Browse through the first page of the most popular shots for that week
"Favourite" any that you like

This takes about 5 minutes each week. After a few weeks you can go back and look at your favourites. You'll likely find a theme in your favourites - perhaps the same color, or you like angles, etc.

You'll also notice the up-and-coming trends. In the past it was soft shadows and sharp angles. The current trend is illustrations of people. Start feeding these trends into your projects. Even if you aren't a big fan of them, it's good to remember that design is effective when it appeals to your target audience and not yourself.

Creating a logo

Logos are a great place to start since the logo design carries over to your software/site. Logos vary for different mediums (print, digital, etc), but since you're reading I'll assume you're a developer, and so your logo will be used on a website, on an app, or in software.

If this describes your use-case it means your logo should consist of two parts, and should follow a few basic patterns. You should start with:

An icon
Logo text/name

Designing an Icon

Here are some useful guidelines:

Square: Make sure it fits evenly into a square. This is important because it will be in a favicon or possibly an app icon.
Symetrical: By symetrical, I mean that it should take up an even amount of space within the square. Logos can look a bit awkward when they are squashed to one side or if there is lots of space in one area of the square.
Simple: This is by far the most important rule. There is no such thing as an icon which is too simple. Make sure it is legible when scaled down to 32x32 px. This is how it will look in the tab of a browser. Also keep the colors simple - you can use color, but it's best if you stick with just one.

Designing the logo text/name

You have a bit more freedom with this one. I would recommend, however, that you make this just a simple color - for example, black or dark grey. Let the icon carry the logo, and let font that you choose for the text set the brand. Whichever font you choose for the text, you should (probably) use it in all the headings of your site/app/software.

Colors

The best advice I can give for colors is to choose one which is vibrant and that's all. Every other color should be white or black.

Ok, now that I have got that out of the way, you can use other colors (for things like alerts, warnings, etc), and a few different grey tones.

If I'm building a tech product, I usually begin my designs with what I call "startup blue". You've probably seen this color on a million different tech sites. The hex code for startup blue is somewhere close to#2B70F6. There are lots of reasons for this, but none more important than it's safe. After you have created your site, feel free to play with this color.

For everything you need to know about colors, I highly recommend reading Building your color pallete by Refactoring UI. After you have a good grasp on the concepts, you can use colorbox.io to build your custom pallete.

Fonts

There are three guidelines I could give here:

Aim for only one or two different fonts, three if you absolutely need it.
Sans-serif is safest for headings and the font should (probably) be bold, symmetrical
Content can be serif but it's fine to be san-serif. Serif fonts are ostensibly easier to read for long paragraphs because the little "tails" on the letters make it easier on the eyes, but this is more true in printed media than on pixilated screens.

As I mentioned in the logo section, you (probably) want your heading font to match your logo font, but it's not critical.

Fonts are something that you have to play with a lot to find something you like. Google Fonts is useful here (if you don't have an aversion to Google) since you don't have to go through the hassle of downloading/uploading the font to try it out. I'm personally trying to move away from Google so I'm slowly building up a site to achieve the same thing, except the fonts are served from GitHub.

Icons

Try to use only a few icons. Icons actually add very little, practically speaking, unless you are able to replace some words completely with the icon. A great example is the ≡ 'hamburger icon' which completely replaces the need to have "Show menu" in navbars. Quite honestly, I think even "good" designers use too many icons.

That said, a few icons can make a site look more finished. Just make sure the icons have a consistent "theme": they should have the same stroke thickness; if some are rounded, they should all be rounded; keep the colors consistent.

And once again, simplify! The point of an icon is to reduce the cognitive load on a user. Try to stick with icons that have very little detail.

Here are the best (free) resources that I commonly use

Images and Illustrations

Images and illustrations largely serve the purpose of filling space and emphasizing your message. Which one you choose is a matter of preference, but try not to mix the two.

Illustrations

Illustrations can really bring a website/app to life. They are less important in software (in fact they would probably look stupid in most software) but work well on landing pages. However illustrations are the hardest to nail, because they are hard quite bespoke which makes it difficult to have a consistent look across all of them.

Try to create your site first without illustrations (I doubt this will be a problem for most readers), then as a final step add a few illustrations if you have gaps to fill. There are some really great resources starting to pop up which have some free, lightweight illustrations to try. These are:

humaaans.com --> you've probably started seeing these everywhere
undraw.co
icons8.com/ouch
avataaars.com
freepik.com

Images

Images are much easier than illustrations because the quantity of stock photos is so much larger than stock illustrations, and if you want to move the designs in-house in the future then it's far easier to find a competent photographer than a competent illustrator.

Once again, images are far more effective on websites than software (which probably goes without saying, but I'm staying it anyway).

It can be difficult to make images feel "part" of the site - quite often they just seem like they are floating in place. It just takes some small things to mitigate this. For example, you can make your images stretch full-width. Or if your site uses lots of rounded corners, add rounded corners to your floating images. If your site uses drop-shadows, add them to your images.

Adding text onto images is a little bit risky since it can make the text hard to read. If you are going to do this, add a very subtle drop shadow to the text.
Alternatively you can consider adding an overlay between the text and the image.

The best sites I use for images are:

Branding

Consistency is the only thing you need to remember here. For a product to feel cohesive, it must be consistent. Some guidelines are:

Choose one color and use it everywhere. For everything else use black or dark grey.
Choose one font and use it everywhere
Text sizes should always be the same (with the exception of headers). Don't make one paragraph 12px and another 14px
Use consistent padding
Make things line up. When we see things for the first time, it's easier to make sense of fewer "blocks" of things. We can achieve this by grouping things together and aligning them.

Don't fall into the trap of "redesigning sections". When you start designing it feels like every part of your site should look slightly different. In fact the opposite is true. Try to use the same layouts everywhere. When a person sees a something for the first time their brain has to make sense of everything they are seeing. There is a cognitive load while they determine what is the important information, and this is mentally an uncomfortable process. If you can reuse the same layouts throughout your site then your users' cognitive load is lessened, and they only have to feel uncomfortable once.

Using a CSS framework

If you're building a website, I highly recommend keeping your CSS framework and JS framework separate. Wherever possible avoid using "locked in" frameworks like ReactStrap, and instead opt for "class based" CSS frameworks. Once you become comfortable with this then your designs can be implemented anywhere - whether you use HTML, React, Vue, or whatever latest and greatest framework you decide to use.

I would also recommend sticking with a "utility-first" CSS framework. Take your pick of either Bulma or Tailwind and invest a day into learning how they work. This day will pay off in spades - prototyping with either of these is quicker than designing then implementing from scratch. Avoid ever using the style tag. Just use classes for everything from layout to look and feel.

Design tools

Figma. That's all you really need. Forget about Sketch and Invision. Figma has vector design capabilities, clickable prototypes, commenting, and it's collaborative. And it's free(mium)! Invest a little bit of time into using it - video tutorials are quite helpful here - and then live inside it for anything that you need to do for design and editing.

I'm really a fan-boy of Figma, but for good reason. I've tried a lot of other tools in the past and none are as good (and holistic). If you're working with photos then you may also need Photoshop but as a developer the types of things that you need to do with photos are usually just compressing and resizing, which can be better achieved with an asset pipeline.

How to Agile

Copple — Wed, 11 Mar 2020 01:18:20 +0000

Ask 3 developers "what is agile" and you'll get four different answers. That's because there is no "definitive agile", and the Agile Manifesto which started it all is extremely loose (on purpose!).

This article will give you a brief background on agile, but more importantly it'll give you a few tips on how to apply it to your own team.

Contents

How to do Agile
A brief history of Technical Project Delivery
The Agile Manifesto
Common Methodologies
Agile Toolset
Interesting Implementations

Don't start with processes

I won't bury the lede. If you're looking to build an agile team, the best approach is to:

Start with as little process as needed. I can't emphasis this step enough, start with the minimum possible process. This is the part that most teams get wrong.
Get regular feedback from your team on pain points and get their buy-in on ways to improve the process.
Start adding or removing processes whenever required. You can pull from a wide toolset of useful practices to make your team more effective. There is no need to be dogmatic about any particular methodology.

What might this look like when you are getting started?

Every day: Do a standup where everyone gets together. This will possibly be the only time your developers spend together each day. Don't be too uptight about how the standup is run, it's great for culture when the managers get out of the way.
Every week: Once per week do a retrospective where you ask your team what can be improved about the process. It's important to highlight the distinction of product vs process. You don't want to talk about product improvements in this meeting. Instead the team should think of activities that make them more effective at delivering. More/less meetings, more/less planning, more/less tracking of productivity, etc.

You'll probably end up with a process which is completely unique to your team, even within the same company.

A brief history of Software Project Management

Before digging deeper into Agile it might be worthwhile knowing how software development got to where it is.

In the not-so-distant past, a Waterfall development approach was ubiquitous. Quite simply it outlined the phases which a technical project would move through:

1. Gather all requirements 
2. Design the entire system 
3. Develop the entire system 
4. Test everything and patch any bugs 
5. Deploy everything in one go 
6. Maintain (sometimes)

It looks quite similar to current techniques, except that it was done in one "monolithic" effort.

No matter how big the project, or how long it was expected to take, the project would be shoved through this pipeline. Once a transition from one stage to another was made there was no going back.

The problem with Waterfall is that big projects would get delivered and everyone would discover that the requirements have changed. Or the requirements were incorrect to start with. Or the people who started the project were gone and nobody knew why they had those requirements to start with.

The Agile Manifesto

Eventually some smart techies decided that there needed to be a more effective way. So they got together in a ski resort and decided to develop the
agile manifesto.

It's extremely brief, but I rarely meet developers who've read the manifesto so I'll list the four core values here:

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

There are also 12 Principles which I'll let you read on your own.

If you've been using agile methodologies in your company, you'll probably be shocked to read the core values. People over processes? That doesn't sound much like the agile shops I worked in. Responding to change? Most companies stubbornly force teams to use their "one true agile".

OK, I'll admit "contract negotiation" isn't a common activity in startups, but if you adapt the third value to "listen to your customers" then you've got a pretty solid set of values for managing a software team.

Notice as well what isn't there. It doesn't say you have to do any specific process. In fact it's not far off telling you the opposite: throw away the processes, plans, and structure; expect change and let your people guide the way.

Agile Toolset

If you've religiously followed what I said up until now then you'll quickly find all your problems are not yet solved. It turns out that processes are important, it's just not clear at the start which processess they are.

Processes

Here are six broad practices that I use and adapt. Most agile methodologies seem to be some variation of these. Use them only if they solve specific problems in your team.

Stand ups: A daily meeting where everyone answers 3 questions: what they worked on yesterday; what they're working on today; and if they have any blockers. Great for culture and an opportunity to clear confusion.

Retrospectives: A meeting to discuss the process that the team is using and any improvements to consider. Typically there are 3 questions: what went well? what didn't go well? and what should we do differently? Great for improving your team's agile process.

Backlogs: A list of all the tasks that need to be completed. Great for deflecting unimportant tasks from your team's important work.

Sprints: A set timeframe for the team to try complete an agreed amount of work. These are typically short - one or two weeks. A meeting held at the start of sprint to discuss what tasks needs to be done. The tasks are usually pulled from the backlog. Great for team productivity (if done correctly).

Pair programming: When 2 programmers work together for a task or timeframe. Usually one person will be driving the interaction and the other is more passive. Great for building team knowledge.

Pull requests: A review process for code. When one developer has completed a task or needs feedback, they can create a pull request (using version control) for another developer to review. Great for code quality.

Roles

As your team grows it's sometimes usuful to introduce roles and delegate responsibilities. These are the roles that I find are the most useful.

Project Manager / Scrum master: this is a bit of a thankless job. Usually their key responsibility is to track how everything is going. However, developers hate being asked when things will be delivered. I like to reframe the role: this person should concentrate on reducing admin tasks for developers so they can focus on their work. New admin tasks should only be introduced by the developers (during the retro).

Product Owners: usually a member of company who isn't a techie. This role is for the business side to help priorities tasks. They don't have to make the final decisions on what to build, but their input can be invaluable. Try to pick someone who has a good social standing in the company and will "have the developers' backs". This person is the bridge between techies and the rest of the company and they can really help shield the tech team from unreasonable demands.

Methodologies

There are a lot of Agile Methodologies but two are predominantly used these days: Kanban and Scrum. If you're new to Agile it can be good to use these as a starting point and them slowly adapt them. Just make sure you remember the fourth agile value: "Responding to change over following a plan". I won't got into them in depth here but here is how I see their usefulness:

Kanban: great for consistent, continuous improvements. Usually you'd lay out the entire set of tasks into one board, and then have the developers pull from the top of the list. Slowly, steadily, the tasks get done without a mad rush for some arbitrary deadline.

Scrum: great for working towards deadlines. The tasks get chunked up into short sprints - periods of work usually one or two weeks long. At the end of the sprint you assess how much work was done and you can forecast or adjust the deadlines as required. Scrum makes it a bit easier to push out new features compared to Kanban, simply because you can stagger the dependencies.

Interesting implementations

Learning how other teams do agile is useful in a "the journey is the destination" sort of way. You can learn what worked for that company, at that time, with those people. If you ask these companies today what their agile processes look like then it will almost certainly have changes. So take only what you need from these case studies, but don't get too enamoured into thinking they have the perfect process.

Spotify: "Autonomous Tribes"

Spotify used to employ a fairly verbose framework where they organised their teams into Tribes, Chapters, and Guilds. They used a very decentralised approach to development, modelling their internal software architecture on the open source model of loose dependencies. If you wanted to identify which agile methodology they use, they would say "it depends which team".

Amazon: "Press Releases"

Not so much of a framework but a requirements gathering process. Amazon write "Press Releases" before working on any particular feature. The idea behind this is to be customer driven and to improve clarity. It also answers the question of "why". Much of the work tech teams is driven by blind requirements. Press Releases help to motivate teams by understanding why they are working on a feature and the impact it will have.

Early WhatsApp: "No process"

(I can't actually remember where I heard this. I think it was a video of a Whatsapp employee talking at a conference. Please share if you know which video I am talking about.)

In a video I saw, a Whatsapp employee said that every developer had their own backlog. They had full autonomy over what they would work on, with one rule: prioritise bugs over anything else.

He said that he felt so much empowerment and ownership with this approach, that when his friend told him about a bug during a party he pulled out his laptop and fixed the bug immediately.

Honestly I don't know how this worked when trying to push new features. But I can imagine that if you give your developers autonomy over what they want to work on (maybe pulling from a company backlog) and no set timelines, they will be the most loyal techies in the world. Meaning that they will stay with your company for three years instead of two.