DEV Community: Madalin Ignisca

How to configure Apache MPM Prefork with mod_php for performance and stability

Madalin Ignisca — Sun, 26 Mar 2023 09:43:33 +0000

Why you should never use Apache with mod_php in the frontend? Because PHP can take your services to Out Of Memory when somebody really wants to take your website down. Read next about this.

First, I am not against Prefork. I find it actually a decent way of running PHP, but only if it is for handling PHP and no static files. When it is used in combination with another web server (Apache included) and proxying to it only PHP requests, can help to make the usage of many popular web applications even easier, as for example WordPress with some plugins will always set extra directives in .htaccess (consider W3 Total Cache for example).

But read the following to understand what you are facing leaving it with defaults and allowing it to be hit directly from the Web.

Next are the default values of how Prefork will be configured.

StartServers 5
Number of child server processes created at startup

MinSpareServers 5
Minimum number of idle child server processes
If only <= 5 concurent requests happen, max 5 processes will live, but if serving static files, you’ll see this up after just a few minutes, hours.

MaxSpareServers 100
Maximum number of idle child server processes
If you are trying to set the value equal to or lower than MinSpareServers, Apache HTTP Server will automatically adjust it to MinSpareServers + 1.
As concurent requests happen, this will become current idle number of processes, and all above 100 idling will be terminated. Why waste cpu time on handling this? I would set it to MaxRequestWorkers value.

MaxRequestWorkers 256
Maximum number of connections that will be processed simultaneously
Put this on a heavy Magento website and cry if you don’t have 64GB of RAM only for Apache + mod_php 😂

ListenBacklog 511
Maximum length of the queue of pending connections
Will be useful for queuing more requests if your requests finish really fast (<50ms), but consider going horizontal scalling if you do more than 2-4 concurent requests per real cpu core.

ServerLimit 256
Upper limit on configurable number of processes
Use this directive only if you need to set MaxRequestWorkers higher than 256 (default). Do not set the value of this directive any higher than what you might want to set MaxRequestWorkers to.
Human explanation: DON’T YOU EVER F**KING THINK TO SET THIS VALUE VERY HIGH!

What it means is if you are hit on 256 HTTP requests in PHP, you will have 256 processes trying to concurrently do work.

If you are on a BIG server, saying you have 32, 64 real cores, that means no VCPU as that is already 1 core with 2 threads fighting for resources, it might play nice, as long as you have the amount of RAM that your php request will need to be handled.

For example, if you have a Symfony app that on each request uses in average 40mb of ram, to be safe, you should have your server with 10 GB of ram only for Apache + PHP and some extra for the OS to behave, so actually over 12 is safe. This is considering that the server will do only Apache + mod_php and not extra databases or others.

If Apache + mod_php is a must, you should calculate your MaxRequestWorkers to the max it can be handled.

You must benchmark your application for memory usage first. Speed is not the main issue here, as you must calculate how much RAM will be used on max concurent requests.

From personal observation, for common popular PHP apps, I observed that WordPress for example averages around 40MB, when using just few plugins (contact form 7 with an extra storage plugin) and a theme like Divi.

I tweaked many WordPress websites to ditch plugins and move in the theme a lot of extras, like meta directives in the header, Google Analytics etc. directly as html/js/css things as necessary. This always helped getting rid of about 10MB in total from all the extra plugins per each request.

If you use WooCommerce, you should raise your expectations to 60MB if you avoid any useless plugin (again, try to build in theme templates all you really need, avoiding PHP as much as possible).

On an plugins abusive WordPress website, think towards 90-120MB per request and accept high costs for hosting it.

Don’t forget that the the Admin of WordPress in a few operations might hit you with 200-220MB per request, and for an WooCommerce shop, you should already consider separate instance for handling the admin.

If you are going to have a correctly only PHP handling Apache + mod_php, with values at the limit of your hardware resources dedicated to it, you will actually mimic the same that PHP-FPM does and keeps your app much more safe. PHP-FPM gives you the possibility to use Apache, with Event MPM, and set on both sides limits to protect PHP going OOM (Out Of Memory).

To be safe you always end up in using a Web Server to access static files and Proxy to the PHP server, either Apache mod_php or PHP-FPM. You decide which one.

An example for a very small budget WordPress website, on 2GB ram, 2 vcpu:

Use Nginx to deliver static files and proxy to Varnish (give 100m to it), which will proxy to Apache.
Set MaxRequestWorkers to 4 (keep it safe for performance of requests, use 2 per vcpu or 4 per real core cpu)
Set MaxSpareServers to 4
Set MinSpareServers to 3
Set StartServers to 4
ListenBacklog to max 16 if your HTTP response is > 1s. Adjust depending on how fast are your responses from PHP (not cache)
Set Mysql/MaridDB innodb_buffer_pool_size higher if you have loads of posts (10K+ articles or heavy commented blog) but don’t allocate more than 768MB as it will take over 70% of ram.
Move sessions to Memcached in php settings.
Use any plugin that will set correctly HTTP cache headers so Varnish will serve instant cachable pages. Best might be you making a small plugin that controls cache headers.
All the above might need the hand of an experienced system engineer, but will make any WordPress website incredebly performant at a 5-10$ monthly hosting fee for a cloud server. Might be a good deal paying somebody a few hundreds to save thousands over next couple of years on expensive hosting.

Conclusion
Always set MaxRequestWorkers to a value you will not go above physical memory available (consider 20% for OS, or 5% if running in a container, Docker or Kubernetes).

Consider setting MaxSpareServers close to MaxRequestWorkers.

Set ListenBacklog to a value that it is realistic to queue requests.

Consider StartServers to a higher value, as forking new processes is also a bit time consuming and hits the cpu.

A great benfit of using it, is when monitoring system understand Apache HTTPd, but has not idea about PHP-FPM.

What do I need to be able to develop a web application

Madalin Ignisca — Thu, 09 Mar 2023 11:00:00 +0000

ChatGPT answered in a way I think it is OK for beginners, but for me sounds a bit outdated.

To develop a web application, you will need:

Programming skills: You should have a strong understanding of programming concepts and be able to write code in programming languages commonly used for web development such as HTML, CSS, JavaScript, PHP, and Python.
Familiarity with web development frameworks: You should know how to use web development frameworks such as AngularJS, React, Vue.js, Spring MVC, Ruby on Rails and Django.
Knowledge of databases: You need to know how to create, manipulate, and query databases such as MySQL, PostgreSQL, Oracle or MongoDB, that can store and retrieve data for your web application.
Understanding of web hosting: You should have a basic understanding of web hosting and the various hosting services such as VPS, Cloud Hosting, and Shared Hosting.
Familiarity with version control: Knowing how to use Git, Mercurial, or SVN for version control will help you manage your code base and collaborate with other developers.
Web server knowledge: Understanding how web servers work and how you can configure them for your web application’s needs is essential.
Security: Knowing how to secure your web application, protect data, and prevent attacks is critical to web development.
Testing and debugging skills: You need to have good testing and debugging skills to ensure the application works as expected and is free of errors or issues.
Integrated Development Environment (IDE): You will need a development environment that helps you write and test your code. Popular choices are Eclipse, Visual Studio Code, and JetBrains.
Front-end frameworks and libraries: There are many front-end frameworks and libraries available that make it easy to create user interfaces. Some popular ones include Bootstrap, Foundation, Materialize, and Semantic-UI.
APIs and web services: If your application needs to interact with other web applications or services, you may need to use APIs or create your web services.
Agile development methodology: Agile is a popular software development methodology that emphasizes collaboration, flexibility, and incremental development. Knowing how to work in an agile environment can be helpful.
Project Management: You will need to manage your project, schedule tasks, collaborate with others, and track progress. Popular project management tools include Trello, Asana and Jira.
Continuous Integration/Continuous Deployment: You may want to automate the deployment of your application, which involves integrating code changes and automatically deploying them into a production environment. Some popular tools for CI/CD are Jenkins, Travis CI, and CircleCI.
Communication Tools: To work effectively with other developers, you may need to use communication tools like Slack or Microsoft Teams to share updates, code snippets, and collaborate in real-time.

What do you think? Twit your opinion: https://twitter.com/16nsk/

How to reset correct directories and files permissions recursive in linux

Madalin Ignisca — Wed, 01 Mar 2023 13:30:00 +0000

Here is what you need to do to reset the right way all files and directories in Linux. Works also on Macos and FreeBSD.

Open the terminal. Change directory to the directory containing all you want to fix permissions.

We will fix also the owner, optionally.

I am setting OTHERS to 0, but if some services need read access to your directories or files, use 755, respectively 644. I am just a bit more restrictive and set correct the groups depending on how they will be used.

Fix directories:

find . -type d -exec chmod 750 {} \;

Fix the files:find . -type f -exec chmod 640 {} \;

Fix owner:

chmod -R [replace with user]:[replace with group] .

If this are my local files, usually I replace user and group with my username.

If this are in a web project, on a web server, I set user for the user running the application and group for a group the web server belongs to. I normally ensure both belong to that group. Make note you must set your application platform to do an umask of 027. PHP it is in the ini settings ;-)

The minimal AWS IAM policy for using a bucket with an application

Madalin Ignisca — Wed, 01 Mar 2023 13:30:00 +0000

This is the minimal policy for an application to access only an AWS S3 bucket in which it would upload / download files and generate signed urls for public access.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:GetObjectAcl",
                "s3:PutObjectAcl",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name",
                "arn:aws:s3:::bucket-name/*"
            ]
        }
    ]
}

Create a IAM user. Attach the above policy with bucket-name replaced.

Enjoy and remember to ignore all people that suggest you attach a give all permissions policy. You don’t give your house keys to strangers, right?

How to: permanent private ip on Multipass on Windows with Hyper-V

Madalin Ignisca — Tue, 13 Sep 2022 08:22:08 +0000

If you use Multipass on Windows with Hyper-V, when running MicroK8S or possible other services, you will notice that on restart of the instance or of the host, MicroK8S will not work for you.

The issue is that the assignated IP by the default V-switch of Hyper-V that provides also internet connectivity to your instances will always assign a different ip address to the instance.

Services like Kubernetes, MicroK8S, K3S, expect that the main IP used will not change. Working with changing IP addresses is more complex, but doable.

For Hyper-V on Windows, there is an easy to use solution: add a secondary V-Swith of type internal. Private can be used, but there will be some additional complexity unnecesarry for using Multipass and MicroK8S.

So open Hyper-V manager and in the V-Swith section add a new switch called for example multipass. Keep the name simple, one word and lower case. If not, you will understand yourself later why I advise so.

Now, when you want to start a new Multipass instance, add the network param. Example:

multipass launch -n microk8s-3 -c 2 -m 4G -d 40G --network name=multipass,mode=manual

This will give us in the instance a secondary ETH1 network interface that waits for us to configure it.

Do not try to find a way to do dhcpd with the new v-switch, as 99% of cases you will have identical same situation like with main default V-Switch, random ip addresses.

Now, shell in the instance and add a prepare netplan file:

ip l --- note the MAC address of ETH1 (I copy it in notepad, to be able to copy paste it later)

sudo nano /etc/netplan/99-multipass.yaml

Paste the following template and change the MAC and the IP address including netmask to fit your requirements.

network:
    ethernets:
        eth1:
            dhcp4: false
            match:
                macaddress: 52:54:00:d4:44:41
            set-name: eth1
            addresses: [10.0.1.1/16]
    version: 2

The only important thing is that your machines use identical netmask, so the ip class is not important if it is different from groups of instances to others, as the added vswith is like a dump simple hardware network switch in this case.

I used 10.0.0.0/16 for the instances, as for some experiments with no tunnel networks for pods Microk8s does 10.1.0.0/16 and I am matching this setup in Hetzner with their private network and automatic routes creation using the hetzner ccm for kubernetes (this is another topic, but for me this way of using Hyper-V allowed me to replicate the Hetzner Cloud locally and experiment without expenses).
Any other projects outside of MicroK8S, I use other ip address classes.

Run:
sudo netplan apply

This will freeze the CLI. If u leave it like this a couple of minutes it will return to your CMD prompt, or if u are impatient, close current windows terminal tab, open another one and shell in the instance again. Test now with :
ip a and you should see ETH1 configured.

Enjoy!

Originally posted on https://www.madalin.me/devops/2022/250/multipass-permanent-ip-private-network-hyperv-windows.html

OpenEBS on MicroK8S on Hetzner

Madalin Ignisca — Tue, 13 Sep 2022 08:19:49 +0000

Last few months I experimented more and more with all OpenEBS solutions that fit small Kubernetes cluster, using MicroK8S and Hetzner Cloud for a real experience.

Initially I thought that Jiva is the best choice, as this backend is easy to maintain and ignoring cStor as long time maintanance seemed to be too complex for less experienced people.

So my discoverings made me realise that both cStor and Jiva have their good cases and a decision which one to use from my point of view is simple:

If you want only a small minimum 3 nodes cluster, with expected storage requirements, possible with no expansion in the future, Jiva is the main choice, as on future upgrades, any manual steps will not be needed.

Jiva Storage Classes are defined to use for the storage replicas other dynamic storage classes (defined in Jiva Policies), and stright forward is to use OpenEBS localpath.

On Hetzner on most cases I add each of the 3 nodes an expected block volume with estimated initial size and create the Normal Storage Class. I create also a Fast Storage Class that uses the nodes local storage, which Hetzner provides local NVMe drives with best possible IOPS in all the public cloud industry. Anyway, the attached block storage have higher IOPS than premium storage on any other public cloud, and can be used for almost all cases also.

The idea with attached volumes on Hetzner is that you can increase size on the fly. I manage myself the initial formating with EXT4 and resize in realtime is easy using parted and resize2fs. Usually I use Ansible and run this in parallel on all 3 nodes, but if you are going to try, do it manually a few times than automate, to be fully confident in what you learn.

For the future, as I do Helm updates to the OpenEBS deployment, with Jiva I don't have any changes to do, volumes being on localpaths in reality, files are natively written to EXT4 file system, it is the easyest mode possible from my point of view.

Now on Cstor, which on my first attemps, looked more complicated, and I could not find a real business case to use it, as Jiva is fitting my clients and projects almost in all cases. But one day after re-reading the Cstor documentation again and again, I saw it. I can have POOLS of attached block volumes and the out of the box setup of Cstore with OpenEBS Node Disk Manager made me realise that Cstor is the best choice for a project I started recently, and to which sold storage for applications is a promise to the consumer that is THICK allocated to them, and I don't do oversell (Jiva with localpath would be that by default as it allows overprovisioning by default).

Both Cstor and Jiva can allow over provisioning or not, depedning on how you tweak it, but in my business case, Cstor would provide "block storage" to Deployments or Stateful Apps that the "disks" can be formated with needed filesystems (some require XFS, some EXT4, some clients kindly asked to have it BTRFS). As choice of filesystem did not bother me, I ended up implementing Cstor as this made clients happy (you do look better when you are flexible and provide the features clients demand, if that doesn't complicates your work).

The last experiment went with the recent Mayastor engine, which I still want to experiment more, but the first real blocker to use it, is that it's minimal requirements only to run are very high, and in most projects using MicroK8S to deploy a Highly Available ecommerce shop for example, to have Mayastor already increases the hardware requirements visible in public cloud costs (yes, for many clients even 100 euros extra per month must be really well justified). As initial benchmarks on how Woocommerce or Magento runs in a MicroK8S environment on Hetzner, I did not see a viable justification for using Mayastor, as with minimal caching done well on both WordPress and Magento side, both load tests provided same results. In both cases, MySQL was deployed using Percona Operator which uses hostpath, so DB access speed was at native NVMe speed of the local disks, and assets were cached by Nginx with cache path set to also a temporary localpath mount. Keeping all uploaded images of products on Mayastor provided zero advantange, simply increased the price for larger cloud servers.

My conclusion is use Jiva for most cases, Cstor when you want to expand by adding disk nodes. Avoid Mayastor if you are not in the high pay niche.

If you want to find out how I do all the stuff described above, don't hesitate to buy my book WordPress on MicroK8s. At this moment I'm finishing the chapter on storage and soon to publish a new update describe both using Jiva and Cstor on MicroK8S in Hetzner cloud.

Originally posted on https://www.madalin.me/devops/2022/249/openebs-on-microk8s-in-hetzner.html

Let's encrypt certificates, HTTPS on MicroK8s

Madalin Ignisca — Sat, 20 Feb 2021 05:23:11 +0000

Took me two days, well, not full days, but two rounds of my 1h hour per day to work on the WordPress on MicroK8s book, to figure out how to get Let's encrypt working with default setup of MicroK8s and provision a real certificate for a real domain I use.

First, examples from cert-manager did not work out of the box, but were perfect to get started.

1st, MicroK8s by enabling dns and ingress addons, simply will give you a well configured Nginx ingress controller and a compatible expected internal dns for network communication between pods.

To install cert-manager simply running: kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-manager.yaml worked, and if you follow along, make sure to look at current latest tag and replace v1.2.0 with what is right now.

If you use RBAC on MicroK8s, at this moment I have not yet experimented as it is not in my scope for the moment, but when I will experiment with it, I will update on this.

Next, you need to setup Issuers, and for MicroK8s the example is slightly different from the official docs:

Make sure you will edit according to your needs the email,
privateKeySecretRef/name!!!

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: user@example.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: example-issuer-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: public

The difference is in the class, which needs to be public and not nginx. I guess that nginx is default on official installation of nginx ingress and MicroK8s is configuring it to public. I noticed that in the logs of the nginx ingress pod, the first lines. Also the other difference is in the kind which on MicroK8s only ClusterIssuer worked, and for the official Issuer example in documentation, simply errors that ingress is unknown for http01.
Not sure on the error, but other examples in the official documentation do use ClusterIssuer, so it might need some updates.

The above example will configure it for Let's encrypt staging so you can experiment without getting your domain/subdomain blocked while experimenting. To go for production, create one for production. You can and should have both staging and production Issuers.

Here the production equivalent you need to adapt and apply:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: user@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: example-issuer-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: public

For the Ingress definition, all you need to do is add the annotation like: cert-manager.io/cluster-issuer: letsencrypt-staging or cert-manager.io/cluster-issuer: letsencrypt-prod. Use staging to test change to prod if all is ok for you.

Here is an example for a basic service I use for healthcheck of my home server:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: healthcheck-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - healthcheck.home.madalin.me
    secretName: healthcheck-home-tls
  rules:
    - host: healthcheck.home.madalin.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: healthcheck
                port:
                  number: 8080

The service can be any web server that responds with HTTP 200 and any content you wish

Major benefit: it will self renew the cert with 15 days before expiration.

Also, you can use cert-manager to manage other types of certificates and has some Cloudflare integration which I want to explore next.