LagoonCon Portland 2024 Recap

LagoonCon 2024 was held on May 6, in the Hyatt Regency across from the convention center. With registrations flying in through late Monday afternoon, we were pumped for a great event!

We were off to a great start with people so eager to attend that they showed up almost an hour early. The attendees mingled, enjoying drinks and appetizers while the room buzzed.

At 7 PM, Toby Bellwood, Lagoon Product Owner, warmed up the crowd, and gave an introduction to Lagoon and amazee.io. He introduced himself and gave the order of the evening.

Lagoon Recap, Roadmap, and Positioning

Toby gave an overview of what Lagoon does and why it’s great for both devs and ops. Then we got a recap of what Lagoon has done in the past 12 months. Here are some highlights:

• Focus on security, with more permissions for platform admins, less reliance on legacy tokens.
• Keycloak updated to v17, and then v21 - required major refactoring
• Implemented Organizations
• Brought a lot of self-service functionality
• Build image overhaul, better logs, customization, start of the Go migration
• Lots of component, dependency and version updates
• Changes in the internal image build, test and deployment systems
• Individual environment services - gives us a good building platform for “Insights”
• Documentation reorganization and documenting upgrades and deprecations better

Some other notable achievements this past year:

DDEV Support for Lagoon! Check out the webinar from February: https://www.amazee.io/local-development-with-ddev-and-lagoon

Legacy (Ruby) Pygmy finally deprecated! pygmystack/pygmy all the way now!

So what’s coming next for Lagoon?

• Lagoon Retention periods - using Lagoon to define how long to keep backups, stale images etc - represents not only a cost saving, but a security improvement.
• Lagoon Resource requests - allowing a Lagoon administrator to set specific scale/redundancy parameters for projects.
• Lagoon Labeling standards - providing stable, reliable Kubernetes labels to allow Ops teams to build functionality on top of Lagoon - metrics, logging etc.
• UI v2 - work on the next iteration of the Lagoon UI.

Keep an eye on the Lagoon blog, socials, and Discord to be the first to know when new features are released!

Highly Technical Content Meets Private LLM: Insights and Lessons Learned

Our next talk was from Christoph Weber, Solutions Architect and Sr. Product Extension Manager at Pronovix with Justin Winter, Business Intelligence Engineer at amazee.io. Pronovix builds developer portals, and hosts them on amazee.io.

This project was Inspired by Michael Schmid’s demo at LagoonCon 2023.

The introduction of a private LLM solution and demo of a simple implementation sparked the idea to prove the solution on very complex, highly technical content that is sourced from the real world.

The idea was a private LLM chatbot fed with highly complex technical developer documentation for their client’s developer portal. Ansys, their client, has extensive documentation that is very complex but gives very little context. Privacy was also very important, making amazee.io a great choice for the project.

For the proof of concept, the scope was limited to one client library package, a DPF C++ Client Library. They were working with raw HTML produced with Doxygen, migrated into Drupal. It was ugly, but it worked! Getting the right data was crucial to getting good results. As with everything in computing, garbage in, garbage out.

Justin joined in to discuss some of the implementation hurdles. He talked about the multi-step process of splitting Markdown into chunks, and then processing each chunk with a custom prompt to extract questions/answers, themes, and keywords that then get converted into a vector database. These chunks of text can then be searched when a user asks a question by finding stored text that has a similar semantic meaning, as defined by the embeddings of both the question and text chunks. This is known as Retrieval Augmented Generation (RAG), and can dramatically improve the results of LLM output.

He then demonstrated how this data is used to build a chat interface.

Justin asked a basic question that queries the documentation directly and receives a response. He then asks a more detailed question that requires the AI to understand the data. Finally, the chat generates a code example for implementing the library we’re discussing.

He explained that If you can feed the LLM the context it needs to answer questions, that’s where the real magic is. He also recommended that we think of Chat GPT not as a chat bot but as a search vector. The answers it gives are only as good as the data it has.

This project was done using Llama 3 and Grok LLM.

If you’re interested in the AI space, stay tuned for future offerings from Lagoon and amazee.io!

Why an Open Source Hosting Platform Makes Sense for the Open Website Alliance

Next up was Bryan Gruneberg, CEO, CTO, and Co-founder at Workshop Orange, and Consulting Solution Architect at amazee.io.

He gave a compelling and passionate talk about the Open Web Alliance, its goals, and how an open source hosting platform like Lagoon aligns perfectly with the goals of the alliance.

Bryan introduced himself and Workshop Orange and set forth the goals of his talk: that open source projects deserve to have open source hosting (and that Lagoon is a great choice).

What is the Open Website Alliance?

The Alliance has been founded to facilitate collaboration between Open Source Content Management System Projects (specifically Drupal, WordPress, TYPO3, and Joomla) to drive education and advocacy around open source benefits and principles. Essentially, it’s a community of communities.

The open web is more than a technology - it is a cause

The open web is built on freedom. Anyone can learn, build, and advance. It is defined by decentralization - no single person or entity controls the open web. It thrives on inclusion. Anyone can have a home on the open web, regardless of their background, identity, ability, wealth, or status. The open web requires participation - it is a shared resource and a shared responsibility, sustained by deliberate choice and collaborative effort. It exists for empowerment. The open web is fueled by the desire for information, connection, and progress, and strengthened by individual rights to choice, privacy, and security.

Bryan walked through the points of the Open Web Manifesto and how Lagoon stacks up.

1. Must be designed to protect - not exploit - personal data and public discourse.

   • With Lagoon being fully open source, you can run your own platform or hire a company to run your own platform, controlling data and discourse as you see fit, instead of having to pick a hosting company whose values might not align with yours.

2. Must enable the next generation of innovators & entrepreneurs to compete.

   • Scalability and Flexibility are key. Many hosting providers only allow for a couple types of applications on their platform. If you can put it in a container, Lagoon can deploy it, and scale for any level of traffic.

3. It must be resilient to a changing world and not controlled by a select few.

   • There are only ~4 major Drupal hosting platform-as-a-service offerings. When it comes to WordPress, almost 1% of all websites are hosted on Automattic. That’s not a lot of variety. As a fully open source platform, organizations are empowered and even encouraged to run their own platform-as-a-service with Lagoon.

Bryan wrapped up with an excellent question to think about:

If you buy into these values, embracing a fully open source hosting platform is a logical conclusion.
If your CMS is open source, your tooling is open source, and your contributions are open source…
Shouldn’t your hosting platform be too?

Lagoon Updates and a Hybrid Roundtable

To wrap up the evening, the Lagoon team joined from far and wide to introduce themselves, what they’re working on, and gave a few demos of exciting features being developed for Lagoon.

Matt Swann: UX and UI Updates

Introducing Lagoon UI v2! Matt gave us a sneak peak into the upcoming update to the Lagoon UI. Focused on simplicity and functionality, there are many new user-friendly features. Users will find quick access to common tasks, and overviews will give more information at a glance.

The UI is managed via Storybook, and a Figma design library is available for the community, which we’re excited about. We have fostered a culture of testing and evaluating the UI, creating a roadmap of continuous improvement. If you’re interested in being a UI tester, let us know! Reach out via email at [e-mail] or [e-mail].

UIv2: Problems Tab

UIv2: Deployments Tab

UIv2: Environment Overview

Davit Darsavelidze: Lagoon UI development improvements

Davit has been working behind the scenes on making UI development easier. Here are the highlights:

UI Component Library

A full set of React components, combined with NextJS to make atomic development of future UI functionality more straightforward.

Cypress Testing for the UI

Introducing a full Cypress test suite into the UI, and configuring both GitHub actions and local development environments to ensure all development retains functionality, improving iteration speed.

Full Storybook support for the UI, connected to Figma

Similar to Cypress, a development enhancement to allow faster design to code process, and improved testing/QA/iteration cycles.

Blaize Kaye: Lagoon Ecosystem/Insights Updates

Blaize has been focusing on our Insights suite, which allows users to see a variety of information about their project in the Lagoon UI. Here’s what he’s been working on:

• Lagoon's Insights feature categorizes valuable data into "Facts" and "Problems," aiding portfolio management.
• Automatically gathered during scans, Insights are processed within Lagoon, facilitating efficient issue identification.
• Custom Insights enrich the system by reporting portfolio-specific information, simplified by recent enhancements.
• New mechanisms, like in-cluster service endpoints, streamline interaction with the Insights API.
• Posting data to Insights APIs is straightforward, allowing seamless integration with various tools.

Blaize has also developed a Lagoon-Remote service for Insights, making it simple to add an Insight.

Alanna Burke: Lagoon Community Updates

Alanna gave a quick update on the Lagoon community. A major highlight has been the Lagoon Discord, which is growing and thriving - make sure to join if you haven’t yet: https://discord.gg/te5hHe95JE

We are still trying to figure out what works best for the community when it comes to community hours, so please fill out the community survey and help us help you! https://tinyurl.com/lagoonsurvey

Alanna also highlighted some of the documentation changes in the past year. We’ve done some reorganization to make it easier to find the content you need. We’ve documented the new Organizations features, added a section on how to Lagoonize your site, added a list of client libraries, and all releases are now listed in the docs. Check them out: https://docs.lagoon.sh/

Our community updates wrapped up with a plea for contribution - we’d love to have everyone contribute, whether it’s directly to the Lagoon project, a supporting project, documentation, or to our biggest need: examples! Lagoon can support just about anything, but we need folks to help us out by setting up and Lagoonizing sites in various applications, frameworks, and languages, so that we can add them to our list of examples. If you’re interested in contributing in any way, check out our contribution docs and say hello in the Discord!

Ben Jackson: Build deploy tool iteration, DBaaS Controller redesign, Lagoon Machinery

Ben works on so much of Lagoon, it was hard for him to pick just a few things to talk about! He gave some updates on the work that goes on behind the scenes to make Lagoon tick.

The build deploy tool was initially built in BASH, and is now being rewritten in a component-styled architecture that allows for implementing individual bits in stages.

Build logs are now more easily parseable, and have added smarts for warnings, errors etc.

A future development to keep an eye out for is more composable builds - quicker and more efficient builds will only build selected services, hot-reloading where applicable, allowing for variable-only updates.

The DBaaS controller was a much-needed overhaul, built in partnership with the amazee.io platform team. The DBaaS controller provides Lagoon environments with their databases, but it had a number of limitations - it’s non-updatable once deployed, database targets are hard to auto-select, and it can’t be adapted for other scenarios.

Finally, Ben talked about how he and Chris have been working to refactor Lagoon Services and the Lagoon CLI in Go, using best practices and ensuring that modules are reusable. They’ve taken this rewrite as an opportunity to ensure that everything is done the right way, resulting in clean, concise code.

Brandon Williams: Lagoon and Community

Brandon talked about “eating our own dogfood” - performance tuning for Lagoon core on amazee.io infrastructure resulted in updates to the mariadb images so that everyone can benefit.

Brandon has also been providing a lot of community support in our Discord, and spoke about how we are seeing how community members are benefiting from our years of support and Lagoon knowledge, as well as helping one another - which is exactly what we want to see in our community support channels!

As Toby mentioned, Brandon has been working with DDEV to get native support for Lagoon. DDEV is now a great option for anyone using Lagoon. Read more in the DDEV docs if you’re interested: https://ddev.readthedocs.io/en/stable/users/providers/lagoon/.

Chris Goodwin: Internal-services-test, Lagoon CLI & UI

We needed something to test the wide array of services that Lagoon supports, but what was the best way to do it? A gigantic mega-project with every possible option? No one wants to maintain that. So the team figured surely we could find a way to do this in Go.

Enter internal-services-test, an application that tests itself. It uses Go modules for the individual services, and depending on the path the router users, triggers an action on a specified endpoint - storing and retrieving data. This application is then used in Lagoon-core testing, Lagoon image testing, and sample projects, and it couldn’t be simpler.

Testing Redis

Testing Storage

Testing Mongo

Testing Solr

Chris has also been working on the Lagoon CLI, refactoring the codebase, adding support for Organizations, and aligning the CLI better with the API. He’s also done some work on the Lagoon UI, adding CRUD functionality for project and environment variables, and allowing for new environments to be created in the UI.

Adding an Environment in the Lagoon UI

Managing Organizations in the Lagoon CLI

Managing Environment and Project Variables in the Lagoon UI

As you can see, the Lagoon team has been busy! It was a great way to wrap up the talks at LagoonCon.

Wrapping Up

After the talks, we had some time for networking, and a lot of attendees were really excited and already brainstorming new projects with Lagoon.

This comment from Jeffrey McGuire (Known as jam in the Drupal Community) really summed up the night:

I’ve been in open source since 2005, and I've never seen a presentation like this, and you’re walking the walk that you’re already talking. I’ve never seen a company/project say “we do this well and this well but we’ve got this problem and we’re working on this” and this whole - it’s so open, and it proves that what [Bryan] said, is really true, and it’s possible, so thank you.

Thanks so much to everyone who attended - we hope you had a great time, and we hope to see you both in the Lagoon Discord and at the next LagoonCon!