Laravel-MediaWiki Mutant

Hello and welcome to the story of how Shroomok project was born.
I'm gonna recall my initial motives, decisions, mistakes, pain, and website transformations (which are still going on).

* This publication does not pretend to be a tutorial at all. Also it's a bit chaotic :)

Prologue

Since the immemorial, humans have struggled to share information with each other. Approaches to do this has been changed a lot: from nodding/touching/biting/screaming to using sensible voice constructions. But the real tectonic shift was The Writing!

Nowadays, thousands and thousands of people producing tons of content about their knowledge, ideas, and humble opinions on the internet. Sometimes, they receive feedback from audience as well :-)

To do so we either use platforms such as twitter/dev.to/reddit/etc, or some of us choose the Samurai way with autonomous tools (e.g. self-hosted WordPress)

How often do you worry about the details of what's happening under the hood of someone's blog? Let me guess... never? Running a blog is a bloody trivial task from the tech point of view and no one cares what solutions have been used to deliver publications from author to reader, right?

However, common content project can turn out to be a truly rabbit hole. How deep it's gonna be?

You never know...

Mushrooms

A few years ago I was blasted with the idea of growing mushrooms at home. There were a lot of guides, different subreddits, and communities (such as Shroomery.org) on the internet already. But for me it was still not an easy task to do.

Every grower has different context and background. Each grow report is unique! It consists of many variables such as: choosen strain, spore print quality, substrate, environment conditions, cultivation technique, ability to buy particular equipment, etc. When somebody gives it a try to write down own experience, they usually skip a bunch of details that seem obvious for themselves, but may be completely unclear to readers.

I decided to log every single bit of my growing process, and describe prerequisits in details (which is important). It could be useful at least for myself. If I failed, I'd be able to analyze my report and, possibly, recognize what to change for the next attempt.

I've been stuck with multiple stupid questions even before Cultivation was started. During the growing journey, number of them was growing faster than my mushrooms did. Each step had to be double-checked.

Fungi are gentle creatures. Especially, when one genus meets another: you want to grow Lion's Mane, but get Trichoderma contamination instead :-)

I had enough courage to decide that I was able to accumulate information, personal observations (maybe get some insights), and combine all useful stuff into the new comprehensive guide for newbies (as I was)

...but I never wrote instructions, articles, or even blog posts.

MVP?

You can always build a clumsy beast (what I did further) and fail on first step.

Instead, go simple and fast. Define the focus and move forward (c) Startup dogma that never gets old

First meaningful thing I could call "Shroomok" was the... Google Doc!

It was a good starting point with all functionality I needed for my personal diary. Cool! So, I have a tool for content creation, what's next? Nothing much left to do (hehe): to make more photos, to record videos, to write down each step accurately and thoroughly (with numbers; explaining why did I do exactly this in that way, and not another), and the most important -- to grow mushrooms :-)

Content was cultivating rapidly, although it was just raw material. It definitely was not ready for publication as is. However, some ideas on its look on the web were born before text editor job was done. Rough concept: some kind of tree (but mushroom) with fancy cap, and instruction steps all way down by the stem.

(this museum "masterpiece" :-) is still available here)

It was easy task to do (easier than grow mushrooms): I've found free html templates with similar structure, reworked one (or two) of them to way I needed, added mushroom cap, changed colors, and that's it!

CMS & Harvest

Cultivation process has been successfully finished! I've harvested multiple mushroom flushes, made spore prints, etc. Google Doc counts up to 25+ ready for publication articles. Time to go live! Where and how?

First idea was to convert Google Docs to static html, to link pages between, inject Google Analytics, and upload it on hosting. It doesn't smell like overengineering, right? I said that articles were ready for publication, but in real, I improve them on regular basis. Even first articles are getting updates still.

Of course, I could transform Google Doc into html, upload results to server on each edit, but clever people invented more suitable tools for such purpose -- Content Management Systems!

"Don't waste your time -- take WordPress" -- this is the first thought appeared in my head. Honestly, at that moment, I already had a pretty annoying experience with this powerful CMS. Most likely, it caused by my laziness. I just didn't know how to cook it proper. Struggling with issues (especially malware) on WP projects was imprinted in my memory heavily. Personal preconception against WordPress was the reason why I kept looking for another software.

(Looking back, I think WordPress .MD-based blog framework (like Astro in JS world) would be one of the best solutions at that moment)

Meanwhile, I was wondering about the limits: how big my project could possibly grow. I wanted to start from just another blog, and then build something significant around!

It's hard to think about scalability when you have almost nothing. However, I definitely felt that Shroomok should become a Community with user generated content. Humans are power!

Inspiration came from PsychonautWiki project which is a niche encyclopedia based on MediaWiki software. Everybody can put effort into the content.

You are damn right: it's a Wikipedia from world of substances!

Concept of Wiki for Mushroom Cultivators seemed to me ambitious. That's exactly what I like :-)

MediaWiki

MediaWiki is not a regular content management system. It was created as a collaborative software based on conception when everyone can write own articles and edit pages created by others.

However, I didn't (and still don't) want to wake up in the morning and see changes on my own guide from internet users. Maybe, I'm wrong. Maybe, I must give people opportunity to add their thoughts, to fix something in my publications?

MediaWiki provides functionality to make content changes visible only after approve by moderator. Edit actions from newcomers are required to pass admin (my) review. Sounds good? But it's a whole new process that I have had to learn.

I decided to carry on fast and simple: to cut off main Wiki feature (with related headache and risks), and configure it as one-person blog. Collaboration functionality kept closed until better times...

MediaWiki (MW) dressed in gorgeous responsive Citizen skin with dark-mode support was good enough for Shroomok's first public appearance!

Articles moved from Google Doc to MW's database. It seemed everything was looking good, except one thing: lack of navigation. No worries! Mushroom-stem template was waiting for that moment. I added links from each level of stem to related MW pages. This template became an entrypoint of the website.

On the last days of 2020 I allowed googlebot to index pages. Shroomok was born!

Sidework

Very important part of content projects is... a content, obviously.

However, you can create 101% useful & engaging content and get zero audience as a result. This is not a reward I'm striving to :-)

Most of 2021 I was trying to spread the word about my new project: YouTube channel, Reddit, Discord, etc. Along with that, content improvements was never stopped.

MediaWiki customization

That's why I had no chance to invest much energy in work on technical part of Shroomok website, except of minor changes such as: embeded YouTube videos, comment system for anonymous users, infinite scroll, adding whole navigation into MW's sidebar, etc. Even those little tasks were not so easy to implement due to specific constraints of MediaWiki.

MediaWiki has its advantages: simple pages creation, editing history, powerful work with images, great visual editor. The feature I like very much: if your admin MW account has been hacked, the most dangerous possible scenario is to get unpleasant content changes (which can be reverted) and that's it!

And yet, sometimes I went crazy while tried to invent workarounds for implementation of functionality that is provided by WordPress from the box.

For example, MediaWiki doesn't allow you to set title, page url and h1 separately. By default, MW pages do not contain meta description tag as well.

A lot of ideas were postponed, some features got stuck on half way just because it could break MediaWiki.

Also, I wanted to redevelop Shroomok's UI/UX from scratch, but this rabbit hole seemed to me even deeper.

Lack of control over application led to frustration :-(

Feeling of wrong choice kept coming to my head...

Tailwind

Once upon a time, during another Twitter-feed-doomscrolling session, my eyes caught a post of one guy who shared a list of technologies he used for quick start of new projects. I guess it was a MERN stack + one name, which I've seen before, but never investigated... Tailwind!

My procrastination was interrupted by curiosity. I've opened Tailwind website and it blowed my mind!

I love attractive UI. I ain't so good at this craft, so great design makes me jealous sometimes :-)

In general, Tailwind is a bundle of predefined CSS classes -- many people can say. Everything you need is already provided by pure CSS. I cannot argue with that, but, what Tailwind gave to me is a fantastic documentation with great examples, which makes me able to develop interfaces I always wanted.

Tailwind inspired me to rework Shroomok UI completely!

Shroomok 2.0

Just another fancy skin for MediaWiki wouldn't solve any problem. It was my aesthetic desire.

I've started to think what exactly hurts me, what I want from Shroomok in the future, and about shortcuts to achieve these plans (including new UI).

First of all, I got hunger for 'normal' creative work: when you focus on final results (not on implementation details and workarounds), when you have a choise between bad & good solutions and hence you can find the balance. I missed fun.

The only possible way seemed to me was decoupling from MediaWiki.

But! The "lets-throw-out-MediaWiki-and-rewrite-everything-from-scratch" way didn't attract me at all. I knew that this brutal approach could lead to nothing but depression caused by amount of endless work. Moreover, I would like to keep some MediaWiki features I've mentioned before.

Manifesto (roughly):

• Shroomok application should be easy extendable/maintainable. I want to ship features and design changes quickly!
• first renovated app version must be released as soon as possible.
• make migration from old Shroomok to the new one with least efforts. Keep content and same URL structure (redirects are pure evil for Google crawler)
• be still able to use MediaWiki visual editor, file uploads, text search
• possibility to rollback to previous (MediaWiki-only) version. Just in case
• Tailwind! :-)

It's been necessary to migrate pages one-by-one. This approach allows to catch bugs, to investigate user behaviour changes in Google Analytics, to confirm that job done well.

I must close MediaWiki to the public gradually, and leave it as protected admin panel with editor functionality.

How to sort this out? It's better to overthink in the beginning than to pay overhead later.

Laravel

Of course, I could create a simple wrapper using PHP, HTML and JS to fetch needed content from MediaWiki and display it the way I want.

Hmmm, what smells? It smells like exponentially growing complexity in the future :-)

I was not ready to develop application in single index.php file. Shroomok needed more extensive approach.

So I took a framework I was a bit familiar with -- Laravel!

Everything for bootstrapping is in place: routing, active record, MVC, caching, queues, session provider... and nice Tailwind support.

Okay, here we go

I've created a new Laravel project using composer command.
composer create-project laravel/laravel shroomok-laravel

Beasts settled by the next door:
/var/www/shroomok/en/wiki/
/var/www/shroomok-laravel/


But they still were not aware of each other.

Let's start with a simple -- a new home page. There are no dynamic components, just a static html, styles, and images.

Fortunately, Tailwind installation for Laravel project is pretty much straightforward.

I've added custom color palette to Tailwind.config.json:

Inspiration was coming directly from Tailwind website so Shroomok started to look slightly the same, after all why not to reuse UX which already working well? :-)

During the first evening draft template has been ready.

Data migration

Development in my life always starts chaotically: in some cases I clearly see how data should be structured and stored, in others -- how should the resulting app should perceived by users.

This project is all around the data and relations between entities. So it's time to create proper models and tables in MySQL.

Articles. I've added all things I missed in MediaWiki: seo_title, h1, meta_desc, thumb, meta_image, etc. But, MediaWiki on other hand implements cool stuff which I'd like to carry on with, for example, it uses H1-H6 headings in the article in order to build 'sections' which is used as a "Table of Contents" for end-user.

Images. Yes, in order to replicate MediaWiki structure we should to store images separately. Each image is kind of Article -- it has slug, title, description. Also, MediaWiki stores relations between each image and articles.

Categories. Nothing to say, just categories for articles :-)

SidebarNavigation. As I said, MediaWiki is lacking of ability to automatically create a huge structured navigation from existing publications. That's understandable, because its purpose is to serve thousands and thousands of articles, how would you put them into the menu? I managed to do so with 2 approaches: separate static html page I mentioned before, and custom extension via Hooks, but it was not flexible and kind of messy.

I decided to migrate articles on the fly. Laravel app worked as a caching layer, and MediaWiki was a data store:

• User requests URL, Nginx now routes to laravel project
• Laravel bootstrap and routing happen, then it goes to corresponding controller. Remember that urls structure is a mirror of MediaWiki urls (i.e. '/en/wiki/')
• Controller tries to fetch Article from own MySQL. If article is missing, then it switches to MediaWikiService which handles communication with MedaWiki private site (I've put it on another host) via API. Some data preprocessing happen (image extraction, 'sections' mapping), eventually it returns newly created Article model and corresponding Image models. We can save them into DB for further use.
• Controller happily responds with fetched data injected into nice blade templates seasoned with Tailwind styling. Each following request to the same route with the same path will not do a trip to MediaWiki API, unless there is a need to refresh article contents. For such purpose I've added a condition inside Controller which is looking for query param '?refresh=1'

At that moment, Laravel's role was a pure front-end backed by MediaWiki for data editing, image uploads and so on.

Search

Even search was outsourced to MediaWikiApi. A few hours to implement nice mapper and that's it!

Image processing

MediaWiki does image compression and resizing jobs very well, it pushes the limits when it comes to html markup for images (includes srcset, alt, titles). However, I'd like to present list of articles as a nice grid, with rectangle images cropped to 256x256. Also i was willing to experiment with image compression in terms of SEO optimization. In order to do so another data model was introduced -- CompressedImages with foreign key to original Image. CompressedImages model consists of bunch of variants of thumbnails made from original Images.
Nothing fantastic: ImageMagick and many weeks of experiments. Unfortunately, these experiments didn't outcome with better ranking in search engines. Wasted time? No. Conclusion? If something works, don't touch it. But still, I use cropped images as a card previews in many sections of Shroomok.com

Shroomok 3.0. Separation from MediaWiki. Independent Laravel with Filament

In a life of everybody should be the moment to separate from parents. Laravel's Shroomok reached that stage in 12 months or so. Bottle-neck was the absence of nice admin dashboard focused on text-content edits. I mean, of course, there were solutions, but I didn't feel right about them until the King was found -- Filament!

I can't express my respect enough to Ryan Scherler who initially started this diamond project and Dan Harrin who joined the project shortly after the launch. In my opinion, such projects and community are the main power of Laravel.

In a few weeks, after some tuning of TipTap editor, creating bunch of Resources and Pages (Filament terminology), Shroomok was able to exit relationships with MediaWiki, woo-hoo!

...almost! Shroomok was in need of a good search functionality which should be able to handle typos.

I don't really like to read documentation (but if I did, I'd constantly spending a few hours reading docs, instead of struggling for many days in order to reinvent the wheel). I was terrified by possibility to write custom bridge between Laravel and ElasticSearch or Sphinx. What I'm saying is that Laravel includes many-many batteries and serves fantastic documentation on them which makes life easy! There I found a Scout.

Scout is a Laravel package with which you can plug into existing app and take advantage of Algolia, Meilisearch or other search engines you are not squeamish to use. I've picked Meilisearch -- easy installation, very gentle from resources perspective. I still use it and still satisfied.

Shroomok 3.0 or Fail?

Much time was spent just to come to the stage where Shroomok was initially started. Main MediaWiki functionality finally reproduced, and now I can bring to life any crazy ideas that come to my mind. So, journey just started and there are many things ahead!

P.S: This post was laying in the drafts for so long that it's almost covered with dust. Grass was greener, Twitter was not bought by Elon... many cool features were implemented since: images migration to S3, integration with Discord (oauth, bot for saving photos), gallery with nested tagged images, strains vocabulary / Mushroom Genetics Tree, Quizzes, Forum, experiments with AMP, internal tooling, etc. I'll try to add some stories in the following publications. Stay tuned!

P.P.S: If you want to self-host a blog -- just user CloudFlare pages and some fancy JS framework for that. ChatGPT will help you to set is up -- and everything for free!