Jason Brown
Posted on
I built a large production static site for a publisher (100K articles) [without GatsbyJS], Ask Me Anything!
For further actions, you may consider blocking this person and/or reporting abuse
Top comments (10)
When we started the project it was before Next.js had reached any level of maturity.
We built a pipeline using gulp and metalsmith, which takes partials and markdown, renders HTML and then inlined CSS and JS which had been minified in a previous stage.
Webpack would have been a better alternative had we understood more about the complexity we'd face using gulp for concurrency. The more tasks we needed, the longer the build would run.
We could use partials and mustache to extend templates.
For publishing or re-building based on changes, we would run a jenkins task within our CI and that would re-build all 100K article, updating the header for example.
How long do the re-builds take for all those articles?
How difficult would it be to move from a gulp based setup to webpack?
A full site build would take 30-40min, which isn't just articles but also RSS feeds, pagination for categories and landing pages.
We didn't see a need to move to webpack, we also couldn't do incremental builds with webpack due to how it chunks JS.
Any changes would be a roll forward type thing, so you could see older content over time look or act differently to newer content. Meant we were also able to identify when breaking changes took place and it wouldn't affect the whole site.
Do you have your own admin panel for the authors, or do they submit using markdown files/docs in e-mails? If you have an admin panel, how do you handle previews and edits... meaning do you store a copy of the content in database and then generate the static files using database as source of truth?
Or do you just generate the static files directly once the user has submitted the content.
We built a CMS which using a WYSIWYG editor, marketing and design teams could also create pages and update banners.
SEO teams could improve and update meta-data for pages.
Changes from this system go into an event bus SQS (AWS) and that is then pushed into a build pipeline in jenkins.
What features would you like to add to the site, that you struggle to do well with it being static?
When we built it, SSR react was not a "thing" so we had to mount the React app above the HTML templates, this led to a lot of duplication. So we avoided adding React components unless it provided some form of meaningful interaction with the user: comments, real-time updates etc.
We didn't really struggle with features because it was static, we just had to find better ways to prevent duplication or placeholder elements that would be overridden.
Markup consistency is something I'd add.
If you were to redo this site today, knowing what you know now about your build process, what would you do differently? Would you use an existing SSG, or make the build pipeline yourself?
One problem I see with existing SSG is incremental builds, particularly when it comes to JS. So build performance would need to be really good, we feel we still have that with what we have, we had issues scaling resources to process more content, so I would probably change the language of the pipeline to something more suited towards concurrency and better resource management.
What we didn't know was that we'd have 100K articles. So scaling became a linear problem. More resources = more articles generated.
Scaling memory is painful for nodeJS.
With everything being React nowadays, definitely using a tool or framework that supports SSR for React.
We also suffer a lot from chained processes, so moving towards a more event driven pipeline would allow us to reduce build times and increase efficiency on small tasks.
Knowing what I do now, I would still write our own pipeline, because there isn't an existing SSG that cuts it for what we need. I would just improve the way we did some of the things we did.