DEV Community: Rahul Nagare

Securing WordPress

Rahul Nagare — Mon, 27 Jan 2020 16:18:29 +0000

WordPress gets a bad rap when it comes to security. Because WordPress powers over 35% of the web, it is an attractive target for hackers, crackers, and script-kiddies alike. Review the web server logs of any WordPress site, and you’ll feel worse than after reading YouTube comments.

When you run a high profile or high traffic WordPress site, it’s understandable that the hackers take an interest in it. But why do we see the same hack attempts on new WordPress sites running on a default theme with no content?

More often than not, hackers are interested in the underlying server resources more than the site itself. If they can hack enough sites, those can be used to create an army of bots. Since most of these hack attempts are automated, it is cheaper and easier to try and hack every WordPress site rather than classifying active vs. dev sites.

Think of it like the junk mail you receive for extended car warranty or insurance offers. It’s cheaper for the sender to bulk mail these letters to every household. Even if 0.1% of people respond, it still makes the scam profitable. Hackers use the same law of large numbers when trying to hack WordPress sites.

Does this mean that WordPress is the wrong choice for enterprise sites?

No.

Since most of the hack attempts are automated, we can block them with security measures that are essential for any content management system.

WordPress hacks commonly fall in these categories:

Brute force logins
Plugin/theme exploits
SQL Injections

Brute force login attempts try to gain access to your site by trying commonly used username and password combinations. In the past, installing a plugin to block visitor IP after X failed attempts used to be sufficient. These days hackers rotate the IPs between login attempts to maximize chances of a successful login. We can stop these attacks effectively by switching to a whitelist based approach. By permitting admin access to a small list of approved IP addresses effectively blocks everyone else from accessing your site. If you use WooCommerce on your site, limiting access to a specified list of IPs for a certain number of critical user roles helps.

A byproduct of implementing a whitelist is that it helps find any plugins that rely on admin-ajax.php functionality. Replacing these plugins with better alternatives helps protect your site against the next type of hack.

Plugin and theme exploits are one of the reasons WordPress is considered insecure. With these, hackers try to use recent 0-day exploits to gain admin access to WordPress or install malware at scale. If you implement a whitelist to access wp-admin, some of these exploits are blocked automatically. For the rest, you can tweak your WAF to prevent direct access to .php files and block non-standard query strings.

Finally, we look at SQL injections. These are one of the oldest exploits and have a permanent place on the OWASP top 10 application vulnerabilities. To effectively protect WordPress against SQL injections, we need to go beyond WAF rules. Making sure that your plugins and themes are sanitizing inputs is essential to a secure site. Routine code reviews and instilling a culture of security-first coding helps keep your mission-critical site safe against these exploits.

While it’s true that no security measures are 100% effective, the ones listed here help you block most of the automated hacks. With a smaller attack surface, your WebOps team can work more effectively to protect your site.

Scaling Webapps: Up vs Out

Rahul Nagare — Fri, 17 Jan 2020 15:47:07 +0000

As your web app starts to grow, there comes the point where current server resources are no longer sufficient. WordPress starts to slow down, there are occasional timeouts, and your marketing team has to limit their campaigns to avoid taking the site offline. At this point, you have a decision to make – scale-up or scale-out.

Scaling up means adding more resources to the existing web server. In most cases, this refers to resizing the VM instance to the next available size. Scaling up is also called verticle scaling.

Scaling up is easy, reasonably painless, and has minimal potential to break your site.

But

There is an upper limit on how much you can scale up. Although you can launch VMs with 128 GB memory and 32 CPUs these days, other bottlenecks show up way before you run out of memory. In other words, doubling the memory doesn’t double the capacity to serve visitors. The server starts to run out of available ports, sockets, and in case of containers, IP addresses as the traffic grows. Scaling up also doesn’t protect you against single-point-of-failure.

On the plus side, scaling up keeps the deployment and ops workflows the same as before and doesn’t introduce any changes.

Scaling out means splitting the traffic across several web servers. Scaling out is also called horizontal scaling.

Scaling out is complex and requires extensive work to get started and to keep the site running smoothly. On the plus side, scaling out provides high-availability and significantly more capacity to serve visitors when done right.

When you consider scaling out your infrastructure, you need a plan for the following areas that aren’t necessary with a single web server:

Replicated file system to share code and media across web servers.

Standalone database server accessible over the network

SSL Termination and load balancing.

Log aggregation.

A deployment strategy with an optional blue-green deployment solution.

So, which scaling method is right for you?

If you expect the traffic to grow 2x this year, scaling up saves you from the complexity and overhead. All you need is a maintenance window where you can resize your server and adjust server configurations to utilize available resources effectively.

If you expect the traffic to grow 10x this year, investing in scaling out is more effective. It requires careful planning, testing, and migration from the current server to the new cluster. But when done right, scaling out helps you grow an order of magnitude more compared to scaling up.

If you don’t expect the traffic to change but need better availability for your site, consider this:

Daily automated server snapshots in 2 regions and low TTL on your DNS server.

If there is an extended outage, you can spin up a new VM in a different region using the daily snapshot, update the DNS, and get back online without the complexity of high-availability systems.

Scaling WordPress: The Autoscaling Fallacy

Rahul Nagare — Thu, 16 Jan 2020 22:36:19 +0000

Busy sites need autoscaling. That’s what everyone expects, technical and non-technical users alike. The rising popularity of cloud computing has taught us to expect unlimited capacity for all online services.

How does this apply to WordPress?

A quick search returns hundreds of results that equate AWS with automatic infinite capacity. While AWS is no silver bullet, some of these results do recommend a scalable architecture.

How does autoscaling actually work in the real world, though?

In theory, an adequately configured stack keeps track of various resources, such as CPU, Memory, PHP workers, etc. When the utilization for any of these resources exceeds a predefined limit for more than a few minutes, stack launches another VM. This new VM is then used to serve the additional traffic. Stack repeats this process as long as there is enough traffic to warrant additional resources.

Pretty cool, right?

While this process looks great on paper, it’s a little less magical in reality. When the stack launches a new VM to serve additional traffic, it can take 60 to 120 seconds for it to become ready and join the load-balancer. A good web ops team knows this and utilizes various caches and queues to make this latency invisible to the end-user. Without such caches and queues in place, visitors experience timeouts, slowdowns, or random errors while new VMs are being prepared.

Since autoscaling requires sustained heavy traffic to launch new VMs, it makes little sense to use it on smaller sites.

Time for a car analogy.

F1 cars are built for massive speed. But if you try diving an F1 car on slow speed, it doesn’t generate enough downforce to provide decent traction, the breaks aren’t warm enough to be useful, and the handling is tricky at best.

To get around some of these issues, the F1 crew pre-warms the tires, uses a starter motor to start the engine, and has the skills to handle any pit stops as quickly as possible.

When it comes to operating an autoscaling environment for WordPress, you do need similar systems in place. A process to warm up the load balancer, a way to spin up additional VMs, and a star crew that maintains the pit stops. There are just too many moving parts for autoscaling to “just work” automatically.

When you are shopping for an autoscaling solution, ask the providers on your shortlist these questions:

What’s your plan to minimize the latency while new VMs starting?
Will I be able to launch more VMs in advance if there is not enough traffic yet?
Will I be able to control the thresholds when autoscaling launches additional VMs?

And finally,

Will I be able to contact the crew that set this up directly when I’m expecting a traffic surge?

These questions will help you look past the marketing speak and save a lot of stress down the line.

Going Beyond Uptime Monitoring

Rahul Nagare — Tue, 14 Jan 2020 19:43:14 +0000

In 2020 it is common knowledge that slow websites poorly affect revenue, brand loyalty, and conversions. Do you know what impacts conversions even more?

Outages!

An unreachable website can’t generate new conversions, revenue, or brand loyalty.

There are a wide variety of services that help monitor WordPress sites from different locations in the world. Most of these services access your site every minute and generate an alert if the site is unreachable. While you can’t go wrong with any of these services, I prefer uptimerobot. The number of false positives is minimal with uptimerobot, plus they have a generous free tier.

Uptime monitoring, in combination with server alerts, is sufficient for most small to medium-sized sites. Afterall there aren’t many things that can go wrong on a single-server setup.

But what happens as your site grows and you move to a distributed setup?

Imagine scaling your site across 200 containers on Kubernetes. 198 of these containers are healthy. Remaining two containers, while active, are returning HTTP 200 responses with white pages to all visitors. In other words, most visitors and uptime monitoring services can access your site just fine, but a small number of visitors see blank pages.

How do you find out about the two faulty containers? You can’t wait for visitors to complain, only to have the Ops team respond, “It works on my machine!”.

This is where you need to go beyond uptime monitoring. In high-availability and distributed hosting environments, transaction monitoring becomes essential.

You can do this using a combination of real user monitoring (RUM) and APM insights using a service like New Relic.

RUM monitors and logs each visitor’s browser timing. RUM helps measure website response time across different browsers, regions, and devices. APM is integrated with PHP to analyze, profile, and logs each transaction. APM helps you measure the response time for uncached pages on your site and transactions such as signups, checkouts, and more.

Services like New Relic support setting up custom events for RUM and APM. For example, you can set up an alert if the response time is more than 3 seconds for any visitor or if the bytes transferred are below 1 Kb for logged-in users.

Utilizing RUM + APM based monitoring can help you catch edge cases and significantly minimize the troubleshooting time. Throw container monitoring in the mix, and you get excellent visibility into the high-availability environment that powers your site.

5 Myths about WordPress Backups

Rahul Nagare — Mon, 13 Jan 2020 21:35:42 +0000

Everyone agrees that backups are essential. Not only for websites but also for desktops, laptops, phones, and even git repositories. Yet, when it comes to protecting mission-critical WordPress sites, the backup solutions in place are often inadequate.

More often than not, these backup shortcomings are a result of myths and assumptions about the underlying infrastructure your site runs on. Here are several myths I come across when onboarding new clients.

Myth 1: My site uses EBS, which is replicated across multiple servers. I don’t need backups.

While it’s true that EBS volumes are replicated across multiple servers, they are not protected against any file changes or deletions done by the user. Failed WordPress updates, user error, or malware can alter files on your server. If you don’t have a backup solution in place, EBS can’t recover file changes or restore older versions on demand.

Myth 2: I store my media on S3. I don’t need backups.

Similar to EBS, storing WordPress uploads on S3 is also not an alternative to backups. Without backups, any files deleted from wp-admin even by mistake are lost forever. One caveat to this is if you enable versioning on your S3 bucket, it protects against file changes and deletions. Remember that this is not enabled by default. If your site offloads static assets to S3, make sure that versioning is enabled and working as expected.

Myth 3: AWS takes snapshots of my server. I don’t need backups.

While snapshots are better than not having any backups, restoring them is time-consuming and costly. Imagine your team asking for a copy of wp-config.php from last week. Backup archives can selectively restore this one file in seconds. Server snapshots, on the other hand, need to be restored entirely on a new volume before any files are accessible. If you restore server snapshots on a new instance, there is a risk of wp-cron altering the database with stale values. Snapshots also make local development difficult compared to backup archives.

Myth 4: I don’t need to backup dev sites.

While chances of data loss on development sites are smaller compared to production, these sites should still have a backup solution in place. Development site backups enable developers to rollback 3rd party libraries to previous versions when things go wrong. Backups also protect against dependencies that are no longer available on public repositories, software bugs that result in malformed artifacts, and more.

Myth 5: I don’t need to backup git repositories

Git repos are distributed by design. Most teams also sync with git hosting services such as Github or Gitlab. So does it make sense to backup git repositories? In a high-availability environment, git hooks are often used for various post-deployment tasks. These hooks are not present on developers’ local repos or Github. Having a backup of the git repository on the server protects these hooks in case of a disaster. Git backups are a lifesaver in the event of cloud and SaaS outages.

A good backup strategy contains the following three elements:

Frequency: Backup frequency should be adjusted depending on how frequently your site and database change.

Destination: Backups should be stored in at least 2 separate regions to protect against any cloud outages.

Verification: Backups should be tested periodically to ensure the integrity of the archives.

Once you have a backup policy in place, you can start looking into the performance impact and overhead of your strategy.

Scaling WordPress: Apache vs Nginx vs Apache+Nginx

Rahul Nagare — Fri, 10 Jan 2020 21:38:24 +0000

When it comes to scaling any PHP application, this is a common question. Apache or Nginx or Apache with Nginx? I think this is as important a question as tabs vs. spaces.

Apache gets much hate for not being scalable or reliable. This hate comes from the old days of poor defaults and mod_php. With mod_php, apache spawns a PHP parser even when static files are requested, and that results in higher memory usage per child process. In other words, you end up using ~200 MB memory when processing WordPress files and another ~200 MB memory to serve each image that goes along with the site.

Throw this configuration in a high traffic situation, and like clockwork, it crashes when the memory runs out.

In reality, apache httpd powers popular sites like apple.com and adobe.com. If you use php-fpm with httpd, it scales very well for WordPress, and you get to keep your .htaccess rules.

Nginx uses php-fpm by default with the fastcgi_pass mechanism. This gives it an edge in the benchmarks when compared against apache with mod_php.

If you compare Nginx and Apache side by side with both using php-fpm, the results are almost identical.

So which one is better?

Well, it comes down to the team that needs to support this site regularly. If the team is comfortable with httpd, go for it. If they are comfortable with nginx, use that instead.

Some people like to use nginx in front of httpd to serve static files and let apache handle php requests. While this provides the best of both worlds, I prefer keeping my stacks lean. Lean stacks make troubleshooting easier when things go wrong.

One argument I hear a lot against using only nginx is that you don’t get to use .htaccess, which may break WordPress plugins.

I manage ~15,000 wp sites, and for 99% of them, the following nginx config is enough to replace htaccess:

location / {
try_files $uri $uri/ /index.php?$args;
}

For anything else more complicated, offloading it to 3rd party services (WAFs, CDNs) scales better.

The bottom line is you can get identical performance with Apache or Nginx when using php-fpm. Go with the one you are comfortable with and look for optimizations on another layer. Unless there is a clear need for it, avoid apache+nginx and keep your stack lean.

P.S.: Apache httpd has come a long way from its mod_php days. You can even use Lua scripts with httpd just like Nginx and Resty!

Scaling WordPress: Using wp-cli for WebOps

Rahul Nagare — Thu, 09 Jan 2020 17:24:58 +0000

Maintaining a high traffic WordPress or WooCommerce site takes a lot of work. While automation takes care of tasks such as log rotation, daily backups, and monitoring, ongoing performance optimization still requires manual attention.

When you are working on a mission-critical WordPress site, you can’t install plugins for operational tasks without risking outages or performance drop. As you know, plugins can be a liability on high-traffic sites.

This is where wp-cli comes in handy. Wp-cli is the command line interface for WordPress. It uses the REST API and provides functionality that would otherwise require plugins or code snippets.

Wp-cli is quite useful in high-availability environments because you can use it with PHP-CLI without blocking any php workers. Another upside to using wp-cli instead of plugins is that php-cli has an unlimited execution time with no timeouts by default. This means that if you need to run a query on a large dataset, wp-cli can handle it without timing out halfway.

Here are a few examples where WebOps teams can utilize wp-cli:

Manage wp-cron:
Wp-cli provides better control over cron events. This means you can run specific cron events as needed instead of merely calling wp-cron.php and letting it timeout while clearing expired transients.

This command can execute all pending events without timing out after 60 seconds:
wp cron event run –due-now

Delete expired transients:
While expired transients should delete automatically and keep the wp_options table lean, sometimes it is essential to delete them more frequently. Poorly coded plugins generate transients that stick around for longer than necessary, and the following command can get rid of them:

wp transient delete –expired

If you need to delete all transients, this command helps:

wp transient delete –all

Regenerate Thumbnails:
Instead of adding a plugin to regenerate thumbnails (And slowing down wp-admin), following command can do it for you more effectively:

wp media regenerate

To selectively regenerate thumbnails you can pass the attachment id or instruct wp-cli to regenerate all missing thumbnails:

wp media regenerate <attachment-id> --only-missing

Database operations:
Wp-cli uses the db credentials specific in wp-config.php and provides an easy interface for routine database operations. This comes in handy when you are using RDS, Aurora or other remote MySQL solution where the hostname is complex, and you would rather not keep typing it for each operation.

DB Export:
wp db export <filename>

DB Import:
wp db import <filename>

DB Reset:
wp db reset

DB Repair:
wp db repair

DB Optimization:
wp db optimize

User Management:
WebOps teams need to create users with different roles from time to time to test wp-admin functionality. Wp-cli makes it easy, secure, and fast:

wp user create demouser wp@demo.com --role=editor

Once you are done with the user account, you can delete it with this:

wp user delete demouser

Cache Management:
Wp-cli can manipulate the object-cache without requiring any separate plugins.

To quickly purge the object-cache, this command helps:

wp cache flush

Core and Plugin updates:
Wp-cli can update the core and plugins from the command line:

wp core update
wp plugin update --all
wp plugin update wordpress-seo

The ability to run non-interactively, single .phar binary, and extensibility using modules makes wp-cli a great tool for deployment and ops automation.

Scaling WordPress: Latency vs Throughput

Rahul Nagare — Wed, 08 Jan 2020 17:35:56 +0000

When scaling WordPress sites, one of the first things people try is adding more hardware. Unless this is a proven bottleneck in the stack, this added hardware capacity is often left idle only to be used by various buffers and caches later.

So how do you do capacity planning for large WordPress sites? At Scale Dynamix, we like to think of it in terms of latency and throughput.

Roughly speaking, latency is how fast the site loads for one visitor. Throughput is how many visitors can access the same site at the same time. If latency is too high, adding more hardware makes little difference.

Imagine you are going on a road trip. To avoid traffic, you leave at 2 am and have all six lanes of the interstate to yourself.

But…

You are driving a Toyota Yaris! Would adding another lane on the freeway make you go faster? Of course not.

Similarly, if WordPress is slow for one visitor while most of the CPU is idle, adding more hardware won’t make much difference.

This is why the right starting point when scaling WordPress is latency reduction. Making sure the site loads as fast as it can for one visitor helps when you are ready to open the floodgates and start serving thousands of visitors per minute.

Having said this, I would choose an empty freeway and a Yaris over a congested road and a BMW any day.

Scaling WordPress: AWS is no silver bullet

Rahul Nagare — Tue, 07 Jan 2020 20:24:35 +0000

There is a common misconception that hosting WordPress on AWS makes it scalable automatically. Every online WordPress community I visit has a fair number of posts asking about solutions for a growing site. 10 out of 10 times, these posts have a suggestion that says, “If you need scalable infrastructure, AWS is the only option.”

Suggestions like this lead to a scenario we often see when onboarding new customers at Scale Dynamix. It goes something like this:

The site starts to grow and runs into scaling troubles. Developers suggest a CDN. When the site still has issues, management decides to move the site to AWS.

At this point, either the IT dept., or the agency managing WordPress launches a new EC2 instance. Since AWS recommends using Bitnami’s WordPress AMIs, most people go with that.

While these EC2 images are an excellent way to launch development environments, using them in production doesn’t provide any of the scalability or high-availability features that AWS offers. In other words, users trade their bare-metal servers and VPSs in favor of EC2 VMs and are no better off than before. And now, they also need to have a plan for backups, security, and staging sites (Most AMIs only support one WordPress install).

Here’s how I like to design a scalable and highly-available stack for WordPress using AWS:

Start with appropriately sized EC2 instances (At least two).
Use an application load balancer to route traffic across instances.
Use RDS / Aurora for database hosting. A separate database server facilitates horizontal scaling of web nodes.
Offload static assets to S3

With a stack like this, you can add more EC2 instances as the traffic increases and delete them as needed. Elastic Beanstalk can handle this automatically if the traffic is highly variable.

You can make this stack more reliable and cost-effective by using multiple availability zones for EC2 and using spot instances to handle traffic spikes.

Granted, this is more work, but it helps WordPress scale a lot further than using a pre-created image meant for prototype sites.