DEV Community

Jason Brown
Jason Brown

Posted on

I built a large production static site for a publisher (100K articles) [without GatsbyJS], Ask Me Anything!

Oldest comments (10)

Collapse
 
ctrl_alt_aldr profile image
Christian | 🧑🏼‍💻 • Edited

Did you use an alternative static site gen? (Hugo, Next.js?, etc) and if not how did you approach splitting up and managing the code?

I'm a newbie to this field but without a static gen (or using something like PHP) I can't imagine how you'd wrangle implementing a header to each page for example, and changing it manually across each of the 100k articles opposed to once ^

Collapse
 
jjaybrown profile image
Jason Brown

When we started the project it was before Next.js had reached any level of maturity.

We built a pipeline using gulp and metalsmith, which takes partials and markdown, renders HTML and then inlined CSS and JS which had been minified in a previous stage.

Webpack would have been a better alternative had we understood more about the complexity we'd face using gulp for concurrency. The more tasks we needed, the longer the build would run.

We could use partials and mustache to extend templates.

For publishing or re-building based on changes, we would run a jenkins task within our CI and that would re-build all 100K article, updating the header for example.

Collapse
 
bradtaniguchi profile image
Brad

How long do the re-builds take for all those articles?

How difficult would it be to move from a gulp based setup to webpack?

Thread Thread
 
jjaybrown profile image
Jason Brown

A full site build would take 30-40min, which isn't just articles but also RSS feeds, pagination for categories and landing pages.

We didn't see a need to move to webpack, we also couldn't do incremental builds with webpack due to how it chunks JS.

Any changes would be a roll forward type thing, so you could see older content over time look or act differently to newer content. Meant we were also able to identify when breaking changes took place and it wouldn't affect the whole site.

Collapse
 
cjbrooks12 profile image
Casey Brooks

If you were to redo this site today, knowing what you know now about your build process, what would you do differently? Would you use an existing SSG, or make the build pipeline yourself?

Collapse
 
jjaybrown profile image
Jason Brown

One problem I see with existing SSG is incremental builds, particularly when it comes to JS. So build performance would need to be really good, we feel we still have that with what we have, we had issues scaling resources to process more content, so I would probably change the language of the pipeline to something more suited towards concurrency and better resource management.

What we didn't know was that we'd have 100K articles. So scaling became a linear problem. More resources = more articles generated.

Scaling memory is painful for nodeJS.

With everything being React nowadays, definitely using a tool or framework that supports SSR for React.

We also suffer a lot from chained processes, so moving towards a more event driven pipeline would allow us to reduce build times and increase efficiency on small tasks.

Knowing what I do now, I would still write our own pipeline, because there isn't an existing SSG that cuts it for what we need. I would just improve the way we did some of the things we did.

Collapse
 
cjbrooks12 profile image
Casey Brooks

What features would you like to add to the site, that you struggle to do well with it being static?

Collapse
 
jjaybrown profile image
Jason Brown • Edited

When we built it, SSR react was not a "thing" so we had to mount the React app above the HTML templates, this led to a lot of duplication. So we avoided adding React components unless it provided some form of meaningful interaction with the user: comments, real-time updates etc.

We didn't really struggle with features because it was static, we just had to find better ways to prevent duplication or placeholder elements that would be overridden.

Markup consistency is something I'd add.

Collapse
 
sant0shg profile image
Santosh G • Edited

Do you have your own admin panel for the authors, or do they submit using markdown files/docs in e-mails? If you have an admin panel, how do you handle previews and edits... meaning do you store a copy of the content in database and then generate the static files using database as source of truth?
Or do you just generate the static files directly once the user has submitted the content.

Collapse
 
jjaybrown profile image
Jason Brown

We built a CMS which using a WYSIWYG editor, marketing and design teams could also create pages and update banners.

SEO teams could improve and update meta-data for pages.

Changes from this system go into an event bus SQS (AWS) and that is then pushed into a build pipeline in jenkins.