DEV Community: kakisoft

Docker - Host–Container File Sharing : Named Volumes vs Bind Mounts, Behavior Differences, and Key Pitfalls

kakisoft — Sun, 16 Nov 2025 15:24:19 +0000

When sharing files between a host and a Docker container, there are two main options: "named volumes" and "bind mounts".

They look similar in docker-compose.yml, but their behavior is very different — and that difference can cause serious headaches if you don’t understand it.

Let’s dig into what actually happens, using a real case I ran into.

(Note: I’ll skip the basics of Docker and Docker Compose setup here.)

< environment >

Infrastructure: AWS EC2
Architecture: 64-bit ARM
OS: Ubuntu Server 24.04
Docker: 28.1.1
Docker Compose: v2.35.1

The Two Ways to Share Files Between Host and Container

Docker provides two ways to share files between the host and a container:

Named Volumes
Bind Mounts

They look nearly identical in configuration, but behave differently when you start up your container.

Let’s look at both.

Named Volume

No "./" prefix:

volumes:
  - mediawiki_data:/var/www/html

The shared directory is managed internally by Docker.

In this example, it’s stored somewhere like:

/var/lib/docker/volumes/mediawiki_mediawiki_data

That path is far from your home directory — and only accessible by root.

Bind Mount

Has a "./" prefix:

volumes:
  - ./mediawiki_data:/var/www/html

This time, the directory is managed by the host, and the path is relative to where your Dockerfile or compose.yaml exists.

For example, if you run this under the "ubuntu" user’s home directory, it’ll look like:

/home/ubuntu/MediaWiki/mediawiki_data

At first glance, you might think "same thing, different path."

But in reality, there’s a critical difference in how Docker handles these at startup.

The Real-World Case

At work, we decided to manage our internal manuals using "MediaWiki" (the same engine used by Wikipedia).

We wanted to easily copy and back up uploaded files (images, PDFs, etc.) from the host side, so I configured the media directory to be shared between the host and the container.

But as the setup grew, I found myself constantly adding more directories to share in compose.yaml.

Eventually, I got tired of tweaking the file every time I needed another folder, and thought:

Why not just share the entire "/var/www/html" directory between host and container? Then I’ll never have to touch compose.yaml again.

Sounded reasonable. So I did that.

What Is MediaWiki?

It’s the open-source wiki software that powers Wikipedia.

You can check it here:

👉 MediaWiki

The details of MediaWiki itself aren’t important for this story — it’s just the app I was containerizing.

The First Compose File (Named Volume)

Here’s how my initial setup looked:

volumes:
  - mediawiki_data:/var/www/html

But I forgot the "./" prefix.

That small mistake made Docker create the shared volume under "/var/lib/docker/volumes/mediawiki_mediawiki_data" — not in my project folder.

That meant:

All uploaded files were buried deep inside Docker’s internal path
Only root could access them

So I fixed the "volumes:" section.

The Updated Compose File (Bind Mount)

volumes:
  - ./mediawiki_data:/var/www/html

Just added "./". Simple, right?

Since there was no existing "mediawiki_data" directory, I copied the files from the container manually:

sudo docker run --rm -v mediawiki_mediawiki_data:/data:ro -v "$PWD":/backup alpine sh -c 'cd /data && tar czf /backup/mediawiki_data_backup.tgz .'
mkdir -p ./mediawiki_data
sudo tar -xzf mediawiki_data_backup.tgz -C ./mediawiki_data

Everything worked fine.

I updated the repo with the corrected compose.yaml, and all looked good.

The Problem on Another Server

Later, I needed to replicate this setup on a new server.

Same compose.yaml, same image — but MediaWiki wouldn’t start.

Turns out, "/var/www/html" inside the container was completely empty.

Every file required for the app to run was gone.

What happened?

Root Cause

Here’s the relevant section again:

volumes:
  - ./mediawiki_data:/var/www/html

When using a "bind mount", Docker always treats the host’s files as the source of truth.

If the host folder exists (even if it’s empty), Docker will overwrite the container directory with it.

So on a clean machine, Docker:

Creates an empty "./mediawiki_data" folder
Then syncs it into the container
Effectively deleting everything inside "/var/www/html"

Brutal. But that’s the expected behavior.

What About Named Volumes?

volumes:
  - mediawiki_data:/var/www/html

With a "named volume", Docker manages the directory under "/var/lib/docker/volumes/...".

If it doesn’t exist yet, Docker automatically creates it and copies the container’s existing files into it.

So:

On the first run -> container files are treated as the source of truth
On subsequent runs -> host (volume) files take priority

Bind mounts, on the other hand, always trust the host side.

The Difference in Behavior

Type	First Run	Subsequent Runs	Risk
Bind Mount	Host overwrites container (empty folder = deletes container files)	Host remains the source	High (can delete app data)
Named Volume	Container copies files into the volume	Host (volume) becomes source	Moderate

So if you switch from named volume -> bind mount mid-project, you may end up deleting everything unintentionally.

Workaround

Sure, you could do this the long way:

Start the container with a named volume
Backup the files
Stop the container
Switch to bind mount
Restore the files

…but it’s messy, and managing those compose file changes across environments is painful.

Here’s a cleaner approach.

Step-by-Step Fix

Instead of starting from an empty directory, I pre-populate the host folder with the same files the container expects.

# Create target directory
mkdir ./mediawiki_data

# Copy initial files from the container
sudo docker run --rm -u $(id -u):$(id -g) \
  -v "$PWD/mediawiki_data":/out \
  mediawiki:1.43.1 \
  sh -c 'cp -a /var/www/html/. /out/'

What this does:

Creates a "mediawiki_data" folder on the host
Starts a short-lived container
Copies "/var/www/html" from the container into the host directory
Exits immediately

Now the host folder matches the container’s expected state — no accidental deletion, no surprises.

Summary

Docker supports two main ways to share files between host and container: "named volumes" and "bind mounts".
They look similar in YAML but behave very differently.
"Bind mounts" always trust the host side; an empty host folder will wipe the container’s directory.
"Named volumes" trust the container at first run, then switch to the host on subsequent runs.
The difference can cause subtle and destructive bugs if you’re not aware of it.

If your shared directory is supposed to start empty and only receive files from the container, this won’t matter.

But if the container ships with essential files — be careful.

Which One Should You Use?

Opinions differ.

The official Docker docs generally recommend "named volumes", but that’s not always the best idea.

In real projects, you often nuke your Docker environment when testing or debugging —

and "docker rm -v" or similar commands can easily wipe out those volumes (and all your data) in one go.

For production data or important files, I prefer bind mounts:

Paths are visible and understandable
You can clearly see what’s being deleted
You don’t depend on Docker’s internal volume management

Sure, Docker devs want to show that “Docker is safe and convenient,”
but as a sysadmin, I’d rather store my data where I can see it.

Docker - Host–Container File Sharing : Named Volumes vs Bind Mounts, Behavior Differences, and Key Pitfalls

kakisoft — Sun, 16 Nov 2025 15:24:19 +0000

Let’s dig into what actually happens, using a real case I ran into.

(Note: I’ll skip the basics of Docker and Docker Compose setup here.)

< environment >

Infrastructure: AWS EC2
Architecture: 64-bit ARM
OS: Ubuntu Server 24.04
Docker: 28.1.1
Docker Compose: v2.35.1

The Two Ways to Share Files Between Host and Container

Docker provides two ways to share files between the host and a container:

Named Volumes
Bind Mounts

They look nearly identical in configuration, but behave differently when you start up your container.

Let’s look at both.

Named Volume

No "./" prefix:

volumes:
  - mediawiki_data:/var/www/html

The shared directory is managed internally by Docker.

In this example, it’s stored somewhere like:

/var/lib/docker/volumes/mediawiki_mediawiki_data

That path is far from your home directory — and only accessible by root.

Bind Mount

Has a "./" prefix:

volumes:
  - ./mediawiki_data:/var/www/html

/home/ubuntu/MediaWiki/mediawiki_data

At first glance, you might think "same thing, different path."

But in reality, there’s a critical difference in how Docker handles these at startup.

The Real-World Case

But as the setup grew, I found myself constantly adding more directories to share in compose.yaml.

Eventually, I got tired of tweaking the file every time I needed another folder, and thought:

Why not just share the entire "/var/www/html" directory between host and container? Then I’ll never have to touch compose.yaml again.

Sounded reasonable. So I did that.

What Is MediaWiki?

It’s the open-source wiki software that powers Wikipedia.

You can check it here:

👉 MediaWiki

The details of MediaWiki itself aren’t important for this story — it’s just the app I was containerizing.

The First Compose File (Named Volume)

Here’s how my initial setup looked:

volumes:
  - mediawiki_data:/var/www/html

But I forgot the "./" prefix.

That small mistake made Docker create the shared volume under "/var/lib/docker/volumes/mediawiki_mediawiki_data" — not in my project folder.

That meant:

All uploaded files were buried deep inside Docker’s internal path
Only root could access them

So I fixed the "volumes:" section.

The Updated Compose File (Bind Mount)

volumes:
  - ./mediawiki_data:/var/www/html

Just added "./". Simple, right?

Since there was no existing "mediawiki_data" directory, I copied the files from the container manually:

sudo docker run --rm -v mediawiki_mediawiki_data:/data:ro -v "$PWD":/backup alpine sh -c 'cd /data && tar czf /backup/mediawiki_data_backup.tgz .'
mkdir -p ./mediawiki_data
sudo tar -xzf mediawiki_data_backup.tgz -C ./mediawiki_data

Everything worked fine.

I updated the repo with the corrected compose.yaml, and all looked good.

The Problem on Another Server

Later, I needed to replicate this setup on a new server.

Same compose.yaml, same image — but MediaWiki wouldn’t start.

Turns out, "/var/www/html" inside the container was completely empty.

Every file required for the app to run was gone.

What happened?

Root Cause

Here’s the relevant section again:

volumes:
  - ./mediawiki_data:/var/www/html

So on a clean machine, Docker:

Creates an empty "./mediawiki_data" folder
Then syncs it into the container
Effectively deleting everything inside "/var/www/html"

Brutal. But that’s the expected behavior.

What About Named Volumes?

volumes:
  - mediawiki_data:/var/www/html

So:

On the first run -> container files are treated as the source of truth
On subsequent runs -> host (volume) files take priority

Bind mounts, on the other hand, always trust the host side.

The Difference in Behavior

Type	First Run	Subsequent Runs	Risk
Bind Mount	Host overwrites container (empty folder = deletes container files)	Host remains the source	High (can delete app data)
Named Volume	Container copies files into the volume	Host (volume) becomes source	Moderate

So if you switch from named volume -> bind mount mid-project, you may end up deleting everything unintentionally.

Workaround

Sure, you could do this the long way:

Start the container with a named volume
Backup the files
Stop the container
Switch to bind mount
Restore the files

…but it’s messy, and managing those compose file changes across environments is painful.

Here’s a cleaner approach.

Step-by-Step Fix

Instead of starting from an empty directory, I pre-populate the host folder with the same files the container expects.

# Create target directory
mkdir ./mediawiki_data

# Copy initial files from the container
sudo docker run --rm -u $(id -u):$(id -g) \
  -v "$PWD/mediawiki_data":/out \
  mediawiki:1.43.1 \
  sh -c 'cp -a /var/www/html/. /out/'

What this does:

Creates a "mediawiki_data" folder on the host
Starts a short-lived container
Copies "/var/www/html" from the container into the host directory
Exits immediately

Now the host folder matches the container’s expected state — no accidental deletion, no surprises.

Summary

Docker supports two main ways to share files between host and container: "named volumes" and "bind mounts".
They look similar in YAML but behave very differently.
"Bind mounts" always trust the host side; an empty host folder will wipe the container’s directory.
"Named volumes" trust the container at first run, then switch to the host on subsequent runs.
The difference can cause subtle and destructive bugs if you’re not aware of it.

If your shared directory is supposed to start empty and only receive files from the container, this won’t matter.

But if the container ships with essential files — be careful.

Which One Should You Use?

Opinions differ.

The official Docker docs generally recommend "named volumes", but that’s not always the best idea.

In real projects, you often nuke your Docker environment when testing or debugging —

and "docker rm -v" or similar commands can easily wipe out those volumes (and all your data) in one go.

For production data or important files, I prefer bind mounts:

Paths are visible and understandable
You can clearly see what’s being deleted
You don’t depend on Docker’s internal volume management

Sure, Docker devs want to show that “Docker is safe and convenient,”
but as a sysadmin, I’d rather store my data where I can see it.

When I tried pair programming for infrastructure work, it turned out to be super effective!

kakisoft — Sun, 03 Nov 2024 02:24:27 +0000

In one project, I explained infrastructure task to assign other team member, while sharing my screen. The situation ends up similar to pair programming.

Surprisingly, I fount it is hugely effective. So, I thought "It is the most effective method for infrastructure tasks"
Then I will tell you how to do it.

What is pair programming?

Pair programming is a method that two engineers program using only one computer.
One acts as coder, and another acts as supporter. The role can be switched when it's needed.

The problem of pair programming (In my opinion)

Each engineer often has different develop environment. As times go by, we have more and more variety of choices to build our develop environment.

For example, hardware such as the number of the display, type of keyboard, and mouse.
And software such as editor, the configuration of shortcut key and shell.
There are uncountable numbers of differences.

It is impossible to prepare develop environment customized for you when do pair programming, because you have to share hardware and software with your partner.

I use VSCode, default shell without any customization, a few alias, a few snippets, some plugins.
If my partner needs Vim, or heavily customized shortcut keys for their performance, we lose productivity.

Fir of all, these days most of engineers work remotely, so it is not common to work at same place.

Considering these points, it is difficult to prepare not only to keep their performance but also incorporate this method.

How does pair programming works for infrastructure

Unlike normal programming, it is common for engineers to have completely different infrastructure environment.

Once they login to server, there is no difference of their useful command.
Also, it is rare to have a necessity to work with many displays.

So, you can make pair programming environment only by sharing engineer's display and using chat application.

It is easy to prepare online as well.

Benefits of pair programming for infrastructure development

I found these benefits.

Reduce obscuring of infrastructure works

Infrastructure works tend to be obscure, because it is common to see only one person for infrastructure work.
And the number of infrastructure engineers is often reduced due to budget.

Also, many issues are left out with excuse of "we will follow up".
Some cases, the project have many kludges to work around unnecessarily complicated infrastructure.
These information and knowledge are not officially documented and shared with team members.

These knowledge tends to be exposed when other engineer take over from resigning engineer.

But, you can find out inside of closed box in pair programming, because when you work for infrastructure with sharing display, the partner can ask "What is that for?".
Also, the team can make sure what should be documented after the task.

It is possible to take over without formal documents

Infrastructure engineers often have to do ineffective task because of unoptimized infrastructure.
And, documenting those takes away the time and effort.
(ex. Previous assigned engineer did inappropriate job, which is confusing and complicated. The deadline was close, so the team did last minutes job, thinking to redo it. But that last minutes job stayed for a long time.)

It is not rare to get stuck in a negative cycle like putting off documentation later because it requires a lot of effort, not being able to find other engineer to manage infrastructure, taking forever to educate other engineer because of how complicated to explain.

But, if you work infrastructure task using pair programming method, you can explain about complex issue when the partner ask you "Why do you use this operation?".

Many infrastructure task and documents are not reviewed unlike source code.
Pair programming can solve this problem.

It has the same effect as code review by pair programming. You can find more effective approach from your partner.

All infrastructure work can't be done by code even though IaC exists.
So, pair programming let you work productively.

Notice commend mistake

There are a lot of infrastructure task that you need to pay attention.

Some tasks are nerve wrecking because one small mistake on command often lead to critical problem.

But, when you write a command with shared screen, you can ask your partner to check.

If you made a mistake, your partner can warn you, and reduce your mistake.

Share useful commands

You will use many variety of commands at infrastructure works.

Some commands that you use on a daily basis make an impression to other engineers like "I didn't know that command." or "This is new way to use this command for me."

Also, you can share some information like "I should use this command when I want to check it"

Collaborate to solve the errors

If you face unknown error, you can divide the task to investigate.

For example, a engineer check the logs from the web server such as nginx, the other engineer check the logs from application such as Laravel.

You can get an advantage by using same server. Because it doesn't mean you both checking different local data.

Good for your mental health

Personally, it is the most effective point.

Since our infrastructure tasks have a significant impact on project, you get ill if you get unknown errors and you have no idea how to solve it.

For example, you might face a situation like,
"Shit... I just deployed as usual. But why am I getting this errors?? I have to solve this before the demo tomorrow morning."
"Are you serious? It is already 4:00 AM!?"

In this case, by researching with your partner, you can reduce your stress and get more effective.

I highly recommend this to engineer like you who have got stomachache by infrastructure task.

Our actual conversations

It is not easy to imagine these advantages by only reading. So, I wrote down our actual conversations.

I changed some phrases to make them easier to understand, but the general outline is almost same as actual one.

case 1．

Me: "Oh! I got an error in npm install. Why? Have you added some new libraries?"

My colleague: "Yes. I have added some of them. They might be the problem."

Me: "OK. I'll check the error message. I found it. Oh! The problem comes from Node.js v17."

My colleague: "Why do you use v17? The development environment uses v18."

Me: "Amazon Linux 2 is not supporting v18. So, I installed v17 unwillingly."

My colleague: "Really? Is it impossible to install v18?"

Me: "Yes. We need to prepare Amazon Linux 3 to use v18. At first I was using Amazon Linux 3. 3 is too deferent from 2. So, I changed 2."

My colleague: "Is is so different?"

Me: "Definitely. We can't use scripts and commands from Amazon Linux 2 in Amazon Linux 3. We had to relearn so many commands.
I can't understand why it is necessary to change drastically. I swear to god not to use Amazon Linux any more.
The deadline for demo environment was just around the corner. I didn't have enough time to relearn them. So I discarded 3. Then I tried to install v18, but I couldn't."

My colleague: "Then, we got this shit."

Me: "Oh, but v16 might work. Let's install nvm and change the version."

case 2．

Me: "Oh! Now, I got a different error in npm install! Why? What's wrong?"

My colleague: "I have no idea. I'll check it now."

Me: "Thank you. I'll check it too."

investigating -

My colleague: "We get this error when the Computer doesn't have enough memory, maybe this is the problem."

Me: "Really? micro is not enough. I remember we were using most of memory in previous project."

My colleague: "We can't think of other reason. Let's change the setting and try it again."

Me: "OK! I'll try."

My colleague: "By the way, have you change the EC2 setting? I am scared because I have never done it."

Me: "Don't worry. I have done it before. It was not complicated and quick. You can only stop the instance and change the setting."

working -

Me: "Oh. It's not working."

My colleague: "We got same error. Let's upgrade instance size more."

Me: "Sure. Let's try medium."

working -

Me: "Yeah! It's working now!"

My colleague: "We finally fix it."
f
Me: "I avoid unstable version of Node.js. I think it is best to use latest version in case of Laravel, but Node.js is not like that.
I will only use Node.js LTS version. Also, I will stop being stingy for memory and use medium for EC2 "

case 3．

Me: "Shit! I got seeder error!? What's wrong??"

My colleague: "It works in local."

Me: "I execute it after refreshing tables, so it is not possible existing data causing this. And it works in our local environment. What's wrong?? I have no idea!"

My colleague: "I'll check that."

researching -

Me: "I might find it! Look!"

My colleague: "Oh, you found it? ... what the fuck is this!?"

Me: "There are 2 seeders! One of the seeder is "Out*DoorUnitGroupSeeder", and other one is "Outd*oorUnitGroupSeeder".
The one has uppercase "D", other one has lower case "d"."

My colleague: "Why do these files exist in a same directory??"

Me: "I have no idea. Maybe, a developer pushed a file with wrong name. He realized his mistake and re-pushed it with correct name.
If this assumption is right, it is my bad, I should have checked better."

My colleague: "Which is correct?"

Me: "Upper case is right. It is enough with that."

My colleague: "What is going on.? There is only 1 file in local machine."

Me: "Maybe, we both use Windows. Widows can't distinguish upper case and lower case of file name.
Then, when we run git clone, Windows controls it.
So, we have only 1 file, and we can execute without errors."

My colleague: "I see.
....By the way, how to fix it? Windows only have 1 file, so we can't fix it."

Me: "I'll use Mac and push it. Give me a second.
We could edit the source code directory in EC2. But, it might cause annoying task at next deploy.
It is not difficult task, I'll do it."

case 4．

Me: "CORS error never disappear! What's wrong!?"

My colleague: "We have investigated everything we could come up with. Why?"

Me: "We have demo for the client tomorrow. Why we face such so many errors only today!"

My colleague: "What do we do with it?"

Me: OK! Let's disable CORS."

My colleague: "Are you sure?"

Me: "Yes. I am sure. No problem. I fix it immediately after demo. It is too late, we have to focus on making it work rather than investigating."

My colleague: "How do we disable CORS?"

Me: "We can only comment out this middleware. we gotta pull through."

My colleague: "Really? you edit the source code directly?"

Me: "We can't include this. I'll write down we must execute revert before deploy."

My colleague: "It is too late. We don't have much more options."

Me: "...OK! Maybe it will work."

My colleague: "Hey! I found an another error!"

case 5．

My colleague: "I can't log in! What's wrong!?"

Me: No idea. I found that user is registered.

My colleague: "Let me see... Oh! I found the error."

Me: It looks like some backend errors.

My colleague: "OK! Let's investigate together. I will check nginx logs. Please check Laravel logs."

Me: "Sure!"

investigating -

Me: "I got it! It is a permission problem. It couldn't write laravel.log."

My colleague: "Great. Please fix it."

Me: "OK! ...Yeah! I could log in!"

case 6．

Me: "Phew. We finally finished the deploy. The frontend app works with PM2, and it is stable now.
Hey! Look at what time we started talk. It has been 4 hours."

My colleague: "I'm exhausted."

Me: "Thank you for implementing PM2. I did't tell you, I needed to run this using "npm run dev"."

My colleague: "Are you kidding?"

Me: "No. While the demo app running, I restarted it manually when some errors happened. I had my console on side screen."

My colleague: "What were you doing? lol"

Me: "I didn't have other options. Complain to your frontend leader if you have any. I didn't have time to care about it.
He is always on unimportant tasks, and run away from critical issues. Every single time he found tiny error and insist that "it it serious problem".
I'm struggling to get his progress."

My colleague: "I have heard about it. I don't know the details because I joined the project recently."

Me: "He dump all frontend technical issues on me. Actually, it didn't even build until last week.
I told him that we have demo soon, but he let it cooked."

(It was a total chaos. I wrote why this happens in this article. To sum up, we had an engineer who might fake his CV. )
【システム開発】タチの悪いエージェントの地雷を踏んだ時、被害を最小限に抑える方法を考えてみた

Summery

It seems we are having extremely pointless conversations, but if you have worked as an infrastructure engineer, you have faced similar situation without letting other members know.
Right? I know you have.

Like that, I believe it is very effective to use pair programming in infrastructure jobs.
When I take over this kind of job, I will ask them "I want you to explain it with sharing screen, because of these reasons. "

The most effective Schema-Driven Development using OpenAPI for Logistic Engineer

kakisoft — Sat, 09 Dec 2023 16:15:09 +0000

In web application development, your team use OpenAPI document in many cases when you separate backend and frontend.

But, the API specification document based on OpenAPI is written in yaml or json format. Then it is a very stressful task.

Also, writing up the API document doesn't mean completing by nature. It is a very tiresome task to keep updating document as a project progress.
(Or rather, it is common not to update at all after it is first created because of maintenance costs. And there is a big difference between actual API behavior and API document.)

There are some services to help you write API specification and visualize it graphically, but there are no standard services
So, many developers struggle to decide what is the best approach for this project.

I had a hard time in my current project too.
After researching a lot of different services and applications, I finally found an effective service.

In fact, I introduced it to my clients and my friends, then I received their good replies.

Let me get straight to the point.
We decided to proceed as below.

Use the service called "Stoplight" that helps writing OpenAPI document based on GUI. We can import/export yaml files.
Use "Prism" as a mock server. We can use yaml file as it is and launch using only one command.

Here is the details to decide to use services above.

Development Environment

For frontend, We hired a software vendor that specializes in drawing complicated map.

Our team is in charge of backend.

Main function of backend is done, but API for the vendor has not been touched, neither API design.

In short, backend anf frontend has not been started yet.
The rough requirement is already specified. Our team will decide the details.

We will make new function from scratch, so there are no mock server neither infrastructure ready.

It is required to show a lot of information on the map.
But, we have no idea what data is necessary, what is the suited data format.
It means, we should collaborate with the vendor.

Therefore, we should't give tasks without specification document.

About the adapted services

1．Stoplight

https://stoplight.io/

It helps writing OpenAPI document based on GUI.

You would understand better with images rather than reading explanations.
*It is sample API document of pet store from OpenAPI.

For those familiar with API, you can intuitively understand it.

As you can see, you can edit endpoints, HTTP methods, query parameters, and response parameters based on GUI.

As an additional outstanding feature, you can use it as yaml file.

The generated yaml file can be executed with other OpenAPI plugins.
(This image is when using VSCope plugin "OpenAPI (Swagger) Editor")

It is possible to not only export yaml file but also import yaml file.

It is free to use by yourself, but you can also share with other developers by subscribing paid plan

2．Prism

https://stoplight.io/open-source/prism

In short, "The application that let user use yaml file as a mock server."

You can intuitively understand it with an image too.

As below, by specifying yaml file in command line and executing, the mock server will be launched. ( "-p" is port number)

This is the result of executing with curl command.

As above, you can easily launch mock server without preparing source code and execution environment.

It is incredibly fantastic application, and even more, you can install it very easily with "npm install" command.

The sample command is here.
(Please refer to the official document for details.)

// install
npm install -g @stoplight/prism-cli

// Start mock server from yaml file
prism mock swagger-petstore.yaml -p 4010

I install it globally in my local machine, because it doesn't clutter your environment and doesn't required strict node version.

Explanation of the process

Create API speciation in yaml file using Stoplight based on GUI.
Share it with the vendor in GitHub repository.

The API document generated by Stoplight is based on OpenAPI standard, so there is no strict requirement to use Stoplight as viewer.
The vendor can use their preferred applications.

About launching mock server, write README file that "Use the application 'Prism'. Here is how to install and execute."

Feedback from the client

I proposed these applications and this approach. The client repose was favorable and told me "It's a good idea! Let's introduce it to other members!".
So, I presented to other development team.

I got positive feedback too, we decided to use it for other projects.

It is not necessary to use Stoplight, but the feature of Prism where "let user use yaml file as a mock server" is truly remarkable.
It is worth considering its adaption.

We haven't started development with the vendor yet, so we will get the result later.
I will update the information.

Docker/PHP - What is the most prioritized timezone setting?

kakisoft — Sun, 09 Apr 2023 16:03:36 +0000

When you use Docker & PHP, you can set timezone to many places.

php.ini
example

[Date]
date.timezone = "Europe/Madrid"

Dockerfile
example

FROM php:8.1.12-fpm

ENV TZ=America/Los_Angeles

docker-compose.yml
example

  php:
    container_name: php
    build: ./containers/php
    volumes:
      - ./src:/var/www/src:cached
    working_dir: /var/www/src
    environment:
      TZ: Asia/Tokyo

It depends on project where the value is set.

I wondered what is the most prioritized timezone setting if I set all of places differently.

Here is a source code that I used to research.

<?php

$defaultTimezone = date_default_timezone_get();
var_dump( $defaultTimezone );

1. define 3 places, php.ini, Dockerfile and docker-compose.yml

Result.

string(13) "Europe/Madrid"

Dockerfile and docker-compose.yml definitions were ignored and php.ini definition was reflected.
It seems like php.ini definition is the most prioritized.

Next, remove php.ini definition and execute.

2. define 2 places Dockerfile and docker-compose.yml

Result.

string(3) "UTC"

Why? UTC appeared which is not defined anywhere.

So, I checked OS timezone setting.

$ timedatect
bash: timedatect: command not found

$ echo $TZ
Asia/Tokyo

$ cat /etc/timezone
Etc/UTC

Each command showed different result such as "Asia/Tokyo" and "Etc/UTC". It is a bit confusing.

After researching I found out that timezone setting will not be reflected to PHP setting only by defining environment valuable "TZ" in Dockerfile.
If you want to use OS timezone, you have to define in tzone.ini.

So, I edited Dockerfile.

ENV TZ=Asia/Tokyo
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN printf '[Date]\ndate.timezone="%s"\n', $TZ > /usr/local/etc/php/conf.d/tzone.ini

Then, the result of PHP code execution, timezone setting in Dockerfile is reflected.

string(19) "America/Los_Angeles"

/etc/timezone will be updated.

$ cat /etc/timezone
America/Los_Angeles

3. define 1 places docker-compose.yml

Result.

string(3) "UTC"

The result is same as I expected because tzone.ini is not set.

Just in case, check OS timezone setting.

$ timedatect
bash: timedatect: command not found

$ echo $TZ
Asia/Tokyo

$ cat /etc/timezone
Etc/UTC

It is not practical to define the setting in docker-compose.yml as section "2", so, if you need to set timezone, you should set in other places.

Sometimes I found timezone setting as "TZ: UTC" in docker-compose.yml, this setting may not be working, but actually default timezone setting is working.

However, you can reflect timezone setting of docker-compose.yml by stating as below in php.ini.

[Date]
date.timezone = ${TZ}

(note) It is necessary to set php.ini in the container.

Then, the timezone setting of docker-compose.yml is reflected.
Here is the result.

string(10) "Asia/Tokyo"

Summary

Timezone of Dockerfile reflects to only OS but not to PHP. It is necessary to set tzone.ini.
It is too much work, I personally recommend to set timezone in php.ini.
If you want timezone as variable, set docker-compose.yml and pass to php.ini.

When you face to such a situation "Where is the timezone set in this project?", I recommend check "php.ini" first, after that "docker-compose.yml" lastly "Dockerfile".

Extra

You can overwrite timezone using "date_default_timezone_set".

<?php

date_default_timezone_set('Australia/Sydney');

$defaultTimezone = date_default_timezone_get();
var_dump( $defaultTimezone );

Result.

string(16) "Australia/Sydney"

Docker/Alpine - Why you should avoid alpine linux

kakisoft — Wed, 05 Apr 2023 11:16:59 +0000

Have you ever heard OS called Alpine Linux?

It is very light weight, and not Debian or RedHat distribution.

The official website is here.
https://www.alpinelinux.org/

Because of this characteristic, it can be adapted docker container base image.

You can check out the container from Docker Hub, here.
https://hub.docker.com/_/alpine

I have adapted it in some projects at work and personal project.
We found too many disadvantages, I would like propose to reconsider adapting Alpine for actual product.

As far as I am concerned, the cases where you can take advantage of Alpine are very limited, and many projects will find more disadvantages rather than the benefits.

Here is the list of reasons why you should avoid alpine linux.

It uses unpopular package manager "apk"

It is common to use "yum" or "apt" as Linux package manager, or "Composer" or "pecl" as PHP package manager.

We often finish it after many struggles like reconfiguring package reference to install preferred version of libraries for the project, and adding other package managers such as "remi", "EPEL" and "pickle" over and over again.

It is important that "the problem can be solve easily" because you have to manage many package managers and set different configurations.

In particular, how easy we can find the solution is the key for development.

apk is not popular at all compared with yum and apt as package manager, it means there are few information on the internet.
So, it makes complicated to solve the problem.

There are some cases that you face difficult problems with yum, apk spur difficulty on to solve errors.

It doesn't matter for the quality of product how difficult to build development environment, it is the best way to simplify.

In addition, file extension of Android app is "apk" as well, so you will get more search result about Android app.
Then you will be struggling with filtering these result.

Some library versions are not updated

If you install some libraries using apk, there is high possibility that you can't install latest version(they may not be supported by apk).

In this case, you may need to get and compile source code.

This task will absorb your motivation.

This is totally waste of your time to have a headache for issue that is easily solved by other package manager.

It uses unpopular shell "ash"

Alpine adapts unpopular shell "ash", it makes difficult to create development environment.

By the way, I use default OS shell without any customization.(Well, I set some alias.)

I want to avoid to make it even more difficult to create development environment by adding some customization to the shell.

There was a period a shell called "fish" got attention, however, I completely ignore that trend and kept using default shell.
Because, I believe that "The number of disadvantages by customizing the shell > The number of advantages by customizing the shell" in most cases.

I do understand the advantages by customizing the shell, but sometimes it might make confusing issue when you create development environment.

It is okay if you use customized shell that you perfectly understand all parameters and behaviors. However, if you use the shell that other developer customized, you might face some extra unexpected issues.

Using unpopular shell means you will face similar issues.

In addition, "ash" has low google-ability such as "apk". So, when you face unexpected issues, you will be struggling.

It uses unpopular distribution.

First of all, if you use unpopular distribution, you often face unexpected tough issues.

This article is about some unpopular distributions which I gave a try.

Linux - The report of unpopular distributions

I finally decided to go with Ubuntu.

It is better to use popular distribution.

Creating Dockerfile tend to depend on each engineer's skill

It can be difficult to create Dockerfile because of reasons above.

It uses unpopular package manager and unpopular shell, and sometimes it is required stressful knowledge that you think it is a waste.
Technical debt increase when we use Alpine rather than Debian or RedHat.

The performance is low compared with other distributions.(it can be happened)

I found this interesting article.

軽量Dockerイメージに安易にAlpineを使うのはやめたほうがいいという話(You shouldn't use Alpine for lightweight Docker image without particular reason)

libcに一般的な互換性が不足しているからです。Ruby、Python、Node.jsなどでNativeモジュールをバンドルしているアプリケーションの場合、パフォーマンスの劣化や互換性の問題にぶち当たる場合があります。
(libc is lacking general compatibility. When application uses native module bundle such as Ruby, Python, and Node.js, you might face the problem of performance or compatibility.)

I have made a React development environment in Alpine Docker image, however, the command "npm run watch" and "watch-poll" are too heavy and slow, which makes inconvenient to develop.
(But, the problem might be caused by the way I build. Fortunately it was only my personal project, so I left it like that.)

I used the development environment which had node-slims(node:16-slims) base image, the performance was OK.

To calculate these performance precisely, I have to prepare two exactly same environment with Alpine and node-slims.
But I'm not going to do it. I wait for the engineers who recommend Alpine prove Alpine has high performance.

There will be chaotic incident research after production.

When you use AWS service such as Fargate or ECS, you can deploy the containers without installing some applications and libraries in EC2.

But, it is common to face some problems that occurs in production or staging environment, but not local one.
In this case you will want to login the container directly to find out.
(Sometimes, you find some article state "When you use container, there are no difference between local and production environment, so, you don't worry about difference of these". Truth be told, while local environment doesn't use AWS resources, production environment use it such as RDS, ElastiCache, SQS and SES. if any kind of error has been produced in the connection or configuration, your application will not work properly.)

But, when you use Fargate, you will not be able to login like EC2 SSH.
(Internally, Fargate is managed by AWS dominant EC2. So, it is highly restricted for user access.)

If you use the AWS service "ECS Exec", you can login the uploaded container in Fargate, but it is required to install the application called SSM Agent.

But, Alpine has not and will not support SSM Agent. So, you have to do in an unofficial way.

I found this article.
Alpine Linux に amazon-ssm-agent をインストールする(How to install amazon-ssm-agent in Alpine Linux)

amazon-ssm-agent は Alpine Linux をサポートする予定はないとのこと。
(Alpine has not and will not support amazon-ssm-agent.)

It is possible installed, but the author do in an unofficial way.

I couldn't install it this way.

It often happens to get different result from the author's one since we have different environment, so, we have to solve these issue depending on the error messages.
But, it is complicated to fix some unexpected problems, because it is not easy to find right information.

When the application is launched in AWS, it is very important to have access to container, there is no other way to implement output log and deploy each time.

If it takes more than 20 minutes to deploy an application, we are risking engineer's motivation.

If you use AWS Fargate, you have to avoid Alpine, because container login function is not supported officially.

It is recommended Debian image in Dockerfile best practice in official Docker documentation.

Here is the reference from official documentation.
http://docs.docker.jp/engine/userguide/eng-image/dockerfile_best-practices.html

In terms of base image, use the official repository for the best possible result.

We recommend Debian image. It is well maintained and compact despite full-featured distribution.(Only under 150MB)

I suggest to follow best practice unless there are specific reasons to use Alpine, because official documentation said.

Deployment time is not reduced as much as you expected.

Truth be told, deployment time is not reduced even if you use Alpine.

It is common to think it will take less to deploy because it is light weight.
But, there are many processes to complete deployment. There is only 1 step of container upload time which get benefit from weight saving.

Also, when you upload some containers to AWS, they don't upload each one of them, but upload like broadcast delivery. It seems like process in parallel, so it will not produce difference of processing time.

To be specific, it doesn't mean "Deploying 100 containers takes 100 times longer than deploying 1". However both take almost same time.

Therefore, the benefit from weight saving of container is only the amount when you compare each container.
(I asked AWS specialist.)

I would rather find some disadvantages which sacrifices performance instead of weight saving.

Start-up time of the container might be faster, however ,most of engineers would choose
"the container which takes 5 minutes to start up, but 1 second for 'npm run watch' command"
over
"the one which takes 1 minute to start up, but 1 minute for 'npm run watch' command"

based on these disadvantages

I ended up writing a lot of disadvantages of Alpine unexpectedly.

But, these problems can be avoided easily by not using Alpine.

When you should use Alpine?

In my opinion, development and maintenance cost will increase just by adapting Alpine.
It is too stressful and exhausting job if there isn't infrastructure team with specialists of containers and deployment.

Sometimes it is required stressful knowledge that you think it is a waste, so, it is a quite tough work for backend engineers to do at the same time.

I really recommend to reconsider adapting Alpine even if you have external infrastructure specialists.
In many case, these wasteful knowledge will not be taken over unless you notice and ask for these information.
to keep facing technical debt is bad for your health because it is difficult to solve these issues

Summary

One day, I talked about Docker and building environment with an engineer. We both had a tough time using Alpine, so, we hit it off.
Then I decided to write about it.

If I have a opportunity to join a technical selection and someone propose using Alpine for container base image, I will definitely propose "NOOOO!!!! Use Debian or Debian Slim. NOT Alpine".
The objective of article is that.

You might feel this is debateable, however if you really want to use Alpine, you have to explain advantages outweigh the disadvantages listed above.

Docker/DB - Isn't it better to set share directory between host and db container?

kakisoft — Sat, 04 Feb 2023 14:38:48 +0000

After development, I came up with an idea during application maintenance.

"Isn't it better to set share directory between host and db container?"

For example :

docker-compose.yml

  mysql:
    image: mysql:8.0.29
    container_name: mysql
    environment:
      MYSQL_DATABASE: myapp01
      MYSQL_USER: user
      MYSQL_PASSWORD: password
      MYSQL_ROOT_PASSWORD: password
    ports:
      - "3306:3306"
    volumes:
      - mysql-data:/var/lib/mysql
      - ./shared_db:/shared_db  # set share directory between host and db container

The reason is After lunching the service, make it easy to import dump file in local environment from production environment.

For example, there are many case you want to test out with same environment data as production, because the error could not be reproduced in local environment, but in the specific occasion.

In this case, you have to import dump file which has been exported from production environment or periodical batch.

But, sometimes this task can be very annoying and complicated.

These day, It is common to allow user to use many variety of OS such as Mac, Windows or Linux without specifying machine for development.
Also, the virtual machine technology like docker helps us achieve it.

In this case, to execute import command in the container(MySQL, PostgresSQL or other DB), it is necessary to store dump file somewhere you can refer from the container.
(There might be better approach that specify dump file in the host, I couldn't find it.)

If you share the file between host and container, you need to prepare some protocols or use shared storage that both can refer.

Other approach is executing import command in the host, but shell in the host will drive you crazy.

Especially, PowerShell is special in many ways. For example you can't use the character "<" because it is the reserved word.
So, you have find the way to avoid this problem.
(When I solve the problem, it is necessary to store the file in the container to refer. It was a waste of time.)

Other way is using DB client application such as MySQL Workbench. But, there are differences in each OS, also operation is different even in same application. Then it is annoying to share this information with your team.

From the beginning, even thought we are using virtualization software such as Docker which let us ignore host environment of each developers, it is not a efficient way to have different operations depending on the OS of each developers.

Also, if you successfully execute import command, after you will be struggle with error related to character code.

I wrote about research including failed example.

MySQL/Windows/Docker - How to import a dump file to MySQL container

The clear solution is "Set share directory between host and db container" which I suggested in the beginning.

This makes your explanation clearly and shorter when you need to teach developers how to import dump file in their local environment.
You can only tell them "Store the file in the shared directory between the DB container and the host, and execute the import command from inside the container."

I believe this is the best way. It doesn't get affected by shell differences or host OS.

In addition, no matter how much developers edit docker-compose.yml ,it affects only their local environment.
It can be resolved without any negative impact on the production or staging environment.

However, I have never seen anyone claiming something like mine, and if I put the above setting in docker-compose.yml, I would be asked, "What is this setting for?" ,
so I decided to post it on this blog because I wanted to explain how it is useful and how much problems I faced when I didn't include these settings.

MySQL/Windows/Docker - How to import a dump file to MySQL container

kakisoft — Thu, 02 Feb 2023 19:18:37 +0000

< environment >
OS : Windows
DB : MySQL
virtualization software ： Docker
shell ： PowerShell / git bash / command prompt

I tried import to MySQL container on Windows using dump file, it was a really tough job.

How to export dump file using PowerShell command (execute from outside of container)

When you get MySQL dump file from outside of container, the command is like this.

I have a MySQL container named "mysql".

docker-compose exec mysql mysqldump --user=root --password=password myapp01 > dump_file_20221119.sql

To execute, replace "--user" and "--password" in your environment.
And the scheme is "myapp01".

After executing this command, dump_file_20221119.sql will be created.
There is no problem so far.

How to import dump file

PowerShell (execute from outside of container) [failed]

I tried the most basic import command.

docker-compose exec mysql mysql --host=localhost --user=root --password=password myapp01 < dump_file_20221119.sql

The result is:

PS C:\kaki\work> docker-compose exec mysql mysql --host=localhost --user=root --password=password myapp01 < dump_file_20221119.sql
発生場所 行:1 文字:90
+ ... ql --host=localhost --user=root --password=password myapp01 < dump_fi ...
+                                                                 ~
演算子 '<' は、今後の使用のために予約されています。
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : RedirectionNotSupported

You can't use the character '<' in PowerShell.

I looked up how do I make the character escape, but there is no way to do that.

(Reference)
about Special Characters - PowerShell | Microsoft Learn

I tried "<" and "<", it didn't work.

PowerShell (execute from outside of container - other command) [failed]

When you execute import command, you can use '-e' option instead of the character '<'.

(Reference)
MySQL :: MySQL 5.6 Reference Manual :: 4.5.4 mysqldump — A Database Backup Program

(example)
shell> mysql -e "source /path-to-backup/backup-file.sql" db_name

So, I tried :

docker-compose exec mysql mysql --user=root --password=password -e "source dump_file_20221119.sql" myapp01

The result is:

ERROR at line 1: Failed to open file 'dump_file_20221119.sql'

I found out that the "-e" option only target files inside the container.

I tried store the dump file in the container and execute the command, it recognized the file.

Although, the error occurred.

ERROR: ASCII '\0' appeared in the statement, but this is not allowed unless option --binary-mode is enabled and mysql is run in non-interactive mode. Set --binary-mode to 1 if ASCII '\0' is expected. Query: '��m'.

The cause is that PowerShell default character code is CP932(SJIS).

(Note)
The writer is Japanese.

(Reference)
How to export dump file on Windows, import it to Ubuntu

I thought if "binary-mode" was set "1", it would work.
But, it didn't.

(Reference)
mysql Client Options

Execute command:

docker-compose exec mysql mysql --user=root --password=password -e "source dump_file_20221119.sql" myapp01 --binary-mode

The result:

ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'source dump_file_20221119.sql' at line 1

When I google the word "binary-mode" option, I found "--binary-mode=1" and "binary-mode 0".
But, I did't understand quite well. (I tried them, but I got errors)

I'm not sure why there are so many different statement. Anyway, I refer the one from official website.

But, if I solve the problem, I have to set the file in the container. So, it is not an efficient approach.

When I export using MySQL Workbench appended command, I could successfully import without mentioned errors.

The command is:

export using MySQL Workbench appended command

& "C:\Program Files\MySQL\MySQL Workbench 8.0 CE\mysqldump.exe" --result-file=dump_file_20221119.sql myapp01 --user=root --password=password --host=127.0.0.1 --port=3306 --set-gtid-purged=OFF --skip-lock-tables --skip-add-locks --skip-column-statistics

The shell is Powershell.
When you execute Powershell command that contains space, the error occurred.
So, you have to do other approach.

You need to put the dump file in MySQL container.

Execute PowerShell script [failed]

I thought it would work if I created PowerShell script file and executed.
So, I tried.

I created the file "dump-import.ps1".
The file contents is:

dump-import.ps1

Write-Host "dump importing..."

# docker-compose exec mysql bash
docker-compose exec mysql mysql --host=localhost --user=root --password=password myapp01 < dump_file_20221119.sql

Execute command:

PowerShell -ExecutionPolicy RemoteSigned .\dump-import.ps1

The result:

PS C:\kaki\work\tmp> PowerShell -ExecutionPolicy RemoteSigned .\dump-import.ps1
発生場所 C:\kaki\work\dump-import.ps1:4 文字:90
+ ... ql --host=localhost --user=root --password=password myapp01 < dump_fi ...
+                                                                 ~
演算子 '<' は、今後の使用のために予約されています。
    + CategoryInfo          : ParserError: (:) [], ParseException
    + FullyQualifiedErrorId : RedirectionNotSupported

I got exactly same error.

It works the command "docker-compose exec mysql bash" correctly.

use git bash [failed]

I gave up using PowerShell, so I experimented with git bash.

The result is:

$ docker-compose exec mysql mysql --host=localhost --user=root --password=password myapp01 < dump_file_20221119.sql
the input device is not a TTY.  If you are using mintty, try prefixing the command with 'winpty'

The error message says use "winpty".

Anyway, add "winpty" on the head of the command.

(Reference)
the input device is not a TTY. If you are using mintty, try prefixing the command with 'winpty' · Issue #2888 · vercel/hyper

So, re-execute.

$ winpty docker-compose exec mysql mysql --host=localhost --user=root --password=password myapp01 < dump_file_20221119.sql
stdin is not a tty

The error message says "stdin is not a tty".

I found the page after researching.

docker-compose on wsl: stdin is not a tty · Issue #166 · rprichard/winpty

It's a long standing issue with Docker.
The way it does terminal detection only works with cmd.exe and powershell.exe at the moment. Any third-party terminal breaks that detection and gives the message.

It seems impossible to add "winpty" on the head of docker command, it is necessary to use directly powershell or cmd. (If you use Windows)

As a side note, you can check out about "winpty" here.

[Git Bash] winpty コマンドについて調べてみた([Git Bash] About winpty)

"winpty" command is interface to interact between Windows console program and UNIX virtual terminal.

use command prompt [failed]

So, I used cmd.

C:\kaki\work>docker-compose exec mysql mysql --host=localhost --user=root --password=password myapp01 < dump_file_20221119.sql
the input device is not a TTY.  If you are using mintty, try prefixing the command with 'winpty'

Next, I re-execute the command with "winpty" on the head.

C:\kaki\work\ryuki-prd>winpty docker-compose exec mysql mysql --host=localhost --user=root --password=password myapp01 < dump_file_20221119.sql
'winpty' は、内部コマンドまたは外部コマンド、
操作可能なプログラムまたはバッチ ファイルとして認識されていません。

It didn't work.

execute the command inside container [success(But, inefficient)]

I tried to execute import command after entering the container.

It is necessary to set share directory between host and container.
The example is here :

docker-compose.yml

  mysql:
    image: mysql:5.7

# (omission)

    volumes:
      - db-data:/var/lib/mysql
      - ./shared_db:/shared_db  # set share directory between host and container

Store exported dump file in "shared_db" directory, after login mysql container.

And execute import command in "shared_db" directory inside of the container.

Drawback : Need to rebuild container. annoying.

Also, it is not suitable at all to edit docker-compose.yml and rebuild container only to import dump file.

use MySQL Workbench. [success]

After this long research, I recommend using MySQL Workbench to import dump file to MySQL container on Windows.

There is a annoying error about character encoding when I import the dump file with the command above.
So, it is better to use MySQL Workbench to export.

MySQL Workbench：export

1. Server -> Data Export

2．Start Export

Select the schema to export. ("myapp01" in the image.)

Select Export Self-Contained File.
(If you don't check the option, the dump files will be created for each tables, which create disaster in your file system.)

Set export file name.

After click "Start Export" button.

3. Data export complete

MySQL Workbench：import

1．Delete existing schema

delete the schema to import "myapp01".

2. Create schema

Recreate deleted schema.(myapp01)

Set the name, and click "Apply".

You can leave Algorithm and Lock Type as default.

Scheme will be created.

3. Server -> Data Import

4．Start Import

Select "Import from Self-Contained File", and set export file.

Set target scheme.

Then, click "Start Import".

5．Import Completed

observation

I hope Powershell accepts the character "<".

Amazing Documents Converter Application 'Docurain'

kakisoft — Sat, 07 Jan 2023 15:34:38 +0000

I would like to introduce the service document converter service 'Docurain' which I was impressed.

(Note)

I would like to introduce one impressive service in this slide.

It may seems like a sales presentation.

But, I just love this service. I'm not working for this company.

Have you, as engineers, ever had difficult time making some documents?

Today, I would introduce amazing documents converter application 'Docurain'

Estimate, invoice and purchase order ...

Some system is required to print these documents.

It is not easy to create these functions.

Every single document has totally different and unique format.

Also, we have to choose an appropriate library among many others.

We face many challenges to make some documents download functions.

But, there is a simple solution.

It is "Docurain".

If you use this service, you can be liberated from the annoying document creation tasks.

There are only 2 steps to create a function for document creation using Docurain.

Firstly, create the document template.

It is incredibly simple.

Only tool that you need is Excel.

You have full control to create a complex template.

There is no need of complex operation.

Lastly, call the API.

Specify the template, and set the parameters you want to print, and call.

Of course, you can use with any computer languages.

There is no need to install some specific libraries.

And, there are no initial costs, monthly costs, or support costs.

It costs only 5 yen per document.

So, let me show you how to create documents using Docurain.

First, prepare your Excel template.

Like this, fill with specific code.

And, upload this file.

Next, call the API.

Choose the template, and set the parameter as json object.

Click execute button, document will be downloaded.

You can create such document.

Needless to say, you can call the API from source code.

As you can see, you can easily create document function.

Why don't you make your work easy?

If you want to get more information, please visit official website.

https://docurain.jp/

We are looking forward to hearing from the companies that find document creation frustrating.

!!!Caution!!!

Let me repeat that, I'm not from Docurain company neither this slide is production advertizement.

It is truly truly for this event for IT engineers.

Actually, I talked about this in the presentation at engineer's event.

The presentation material is here.
https://kakisoft.github.io/slides/amazing-documents-converter-application-docurain-en/export/index.html

And, the event website is here.
https://try-english-lt.connpass.com/event/267720/

Note: This service hasn't been translated in English yet.

Laravel - PhpRedis is recommended by Redis library, but aren't there a lot of situations where it's better to use Predis?

kakisoft — Sun, 23 Oct 2022 14:51:57 +0000

Environment
MySQL version : 5.7
Laravel version : 8.16.1
PHP version : 7.4.7

The library when you use Redis with Laravel

There are 2 PHP libraries when you use Redis in Laravel.

By default, composer.lock contains "predis".

composer.lock

    "packages": [
        {
            "require-dev": {
                "predis/predis": "^1.1.2",
                "symfony/cache": "^5.1.4"
            },

config\database.php

    'redis' => [
        'client' => env('REDIS_CLIENT', 'phpredis'),

"predis" is not used which is contained in composer.lock.(By default config)

It is necessary to install "phpredis" separately, which is contained in database.php.

cf.
phpredis couldn't be installed by composer.

What is this contradiction?

I wondered about it and looked it up.
It seems like predis was supposed to be removed from the default, but it has been reconsider.
However, they would like to suggest to use phpredis...

It seems that they are still in confusion.(maybe)

Which should we adapt?

There seems to be a lot of debate about which one to use, and here's what I found.

PHP向けRedisクライアントのpredisを使うのは止めた方がいいです(You should not use predis, the Redis client for PHP)

predisはPHP向けのRedisクライアントです。
predis is a Redis client for PHP.

リポジトリを見て察しのいい方は気づいたと思うのですが、2年ぐらいメンテ（コミットすら）されていません。
If you have a good guess and check their repository, you'll notice that it hasn't been updated (even committed) for about two years.

すでにPHP7.3周りで問題が出てます。これが「どういった事情でメンテされていないのか？」「されないのではなく、できないのか？」などは誰もわからないと思いますし、もしかしたら将来的に再度メンテされる可能性はありえます。
There is already a problem from PHP7.3. I don't think anyone knows "Why is this not being updated?"

I looked up current situation, and as of September 2022, development has resumed and is active.
https://github.com/predis/predis/pulse

Therefore, there is no concern due to development stoppage.

Laravelのドキュメントにも書いてありますが、phpredis使えばいいと思います。
It is also stated in the Laravel documentation, but I think you should use phpredis.

こちらはcomposerでインストールできない（PHP拡張モジュールなので）のでpeclでインストールします。
This can not be installed with composer (because it is a PHP extension module), so install it with pecl.

Nooooooo!!!!!
Are you serious? We're not in the Stone Age now!

As expected, I thought that it was fixed now, so I looked at the official PhpRedis installation guide on the official website.
But, there is a message without a dream or hope to use pecl.

https://github.com/phpredis/phpredis/blob/develop/INSTALL.markdown

I found also this.
How to install phpredis using composer?

I need to install phpredis using composer on windows server running IIS. What is the package name for this?

You cannot.
Composer is for pure-PHP library not for C extension (pickle will do it later)

You need to compile it! See more information here:

Apparently it is impossible to install with composer, and the possibility of being able to do it in the future seems to be close to zero.

Just in case, I checked other articles as well.
Laravel 8のRedisクライアントのパフォーマンスの比較(Laravel 8 Redis client performance comparison)

It seems that PhpRedis was better in performance comparison.

上記のコマンドをそれぞれ50回実行し、平均（±標準偏差）を算出しました。
I executed each command 50 times, and calculate average and standard deviation.

PhpRedis … 1.317 ± 0.03 (秒) *secs

Predis … 1.509 ± 0.026 (秒)*secs

Predisを使用した場合の処理時間はPhpRedisの場合の約1割増しと、明らかに差がでました

Laravel 8 でPhpRedisとPredisを比較しました。PhpRedisの方が普通に速いので、導入できる環境であれば、Laravelの推奨に従ってPhpRedisで良さそうです。
I compared PhpRedis and Predis in Laravel 8. PhpRedis is usually faster, so if it is an environment that can be introduced, PhpRedis seems to be good according to Laravel's recommendation.

But I hate not being able to manage it with composer.

PhpRedis vs Predis: Comparison on real production data

PhpRedis is faster about x6 times. Using igbinary serializer reduces stored data size about 3x times.

If Redis installed on separate machines, reducing network traffic is a very significant speedup.

PhpRedis is also praised here.

Here is Laravel official statement.

Redis 8.x Laravel

Before using Redis with Laravel, we encourage you to install and use the phpredis PHP extension via PECL.

Many developers highly recommended phpredis.

How to install PhpRedis

install command

I have tried Debian and Alpine distributions.

pecl install redis

If you get an error as below, try the following commands before.

/application # pecl install redis
downloading redis-5.3.4.tgz ...
Starting to download redis-5.3.4.tgz (268,154 bytes)
........................................................done: 268,154 bytes
29 source files, building
running: phpize
Configuring for:
PHP Api Version:         20190902
Zend Module Api No:      20190902
Zend Extension Api No:   320190902
Cannot find autoconf. Please check your autoconf installation and the
$PHP_AUTOCONF environment variable. Then, rerun this script.

ERROR: `phpize' failed

Command to run when the above error occurs

apk add --no-cache \
        $PHPIZE_DEPS \
        openssl-dev

When writing in Dockerfile

FROM php:7.4.11-fpm

# omission

RUN git clone https://github.com/phpredis/phpredis.git /usr/src/php/ext/redis
RUN docker-php-ext-install redis

Installation check

You can check whether PhpRedis is installed correctly with the following command.

php -m | grep redis

If you see the following message, PhpRedis has been successfully installed.

redis

Redis container configuration

docker-compose.yml

version: '3'
services:

# omission

  redis:
    image: "redis:6.0"
    container_name: redis
    restart: always
    ports:
      - "6379:6379"
    volumes:
      - "./data/redis:/data"

Laravel configuration

.env

CACHE_DRIVER=redis

REDIS_HOST=redis
REDIS_PASSWORD=null
REDIS_PORT=6379

config\database.php

    'redis' => [

        'client' => env('REDIS_CLIENT', 'phpredis'),

Before running the app, clear the cache with the following command.

php artisan config:clear

Sample sources

I have written the following code to try it.

routes\web.php

namespace App\Http\Controllers;

use Illuminate\Support\Facades\Route;
use Illuminate\Support\Facades\Cache;

//============================================================================================
//                                       Cache
//============================================================================================
// http://localhost:8000/cache/put
Route::get('/cache/put', function () {
    echo "put";

    // store value in cache
    Cache::put('key01', 'value01');
});

// http://localhost:8000/cache/remember
Route::get('/cache/remember', function () {
    echo "remember";

    // store value permanently
    $cache = \Cache::remember('key02', 10, function(){
        return "value02_remember";
    });

    dump($cache);
});

// http://localhost:8000/cache/get
Route::get('/cache/get', function () {
    echo "get";

    // get one value from cache
    $cache01 = Cache::get('key01');
    $cache02 = Cache::get('key02');

    dump($cache01);
    dump($cache02);
});

// http://localhost:8000/cache/forget
Route::get('/cache/forget', function () {
    echo "forget";

    // remove value from cache
    Cache::forget('key01');
    Cache::forget('key02');
});

I was able to successfully get Redis to work.

caution

After installing with command pecl install redis, the following message was displayed.

You should add "extension=redis.so" to

However, it worked without adding it.
I have no idea about the details, but I guess Laravel take care of it and read it on its own.

As I thought, I also found this.

Laravel Redisのライブラリをインストールしたらエラーが発生した(An error occurred when installing laravel redis library)

The content says,
"I got an error, but I fixed it by commenting out php.ini "extension="redis.so". 』

So, when you use Laravel, this setting may not be necessary.

However, there seems to be a case where an error occurs when uploading to another environment.
AWSにLaravelをデプロイしたらエラーが出たときの対応方法(How to solve an error when deploying Laravel to AWS)

It seems that extension=redis.so is added to php.ini to make it work in EC2.

It seems better to keep in mind that there are cases where such an error occurs, just in case "It does not work in the production environment!"

Maybe, using Amazon Linux(RedHat distribution) is one of the causes.
I tried it later and it worked just fine on Debian.

How to add "extension=redis.so" to php.ini

First, check the path of php.ini.
Below is the command.

php -i | grep php.ini

Execution result example

/application $ php -i | grep php.ini
Configuration File (php.ini) Path => /usr/local/etc/php
Loaded Configuration File => /usr/local/etc/php/php.ini

In the above, the saved path was "/usr/local/etc/php/php.ini", so the additional path is as follows.

echo "extension=redis.so" >> /usr/local/etc/php/php.ini

After adding it, check the contents of php.ini.

Then reboot.

case of Predis

I also tried it.

It can be installed with composer, so it's much easier than PhpRedis.

For more details, read the official website.
https://readouble.com/laravel/8.x/ja/redis.html

Execution command is here.

composer require predis/predis

If .env and config are set correctly, I think it works without any problem.

Laravel configuration

config\database.php

    'redis' => [

        'client' => env('REDIS_CLIENT', 'predis'),

Other settings and the code used in the experiment are the same as above.

Since there is no difference due to distribution, you will not encounter unexpected issue, and you will have peace in your mind.

perspective of library selection

Same engineers say that "PhpRedis is fast! PhpRedis is the best! You should definitely use PhpRedis!"
But, it cannot be managed by a package manager, and since it is necessary to modify parts close to hardware, there are many things to take account.

At the end of time, the difficulty of the building an environment may be complicated, it increases the possibility of unexpected errors during regular work.
In my opinion, then Predis is a good choice.

I found an article talking about the concern which Predis development had been suspended, but now development has resumed.
It can be managed by composer and has many advantages.

PhpRedis has better performance, but depending on system, there are cases where access to the cache does not happen so frequently.
If you don't need such a strict access speed performance, I think one of the selection criteria is to choose a library that can reduce management costs.

By the way, I am not agree with such an idea.
"Faster and lighter is always right! (Even if the application does not require high front-end performance. It should be implemented even if you have technical debt!)

Since you can't manage it with composer, you need to recreate the Docker image.
And also, If you are planning to use it in virtual server such as EC2, it will be necessary to prepare an installation command for each distribution, and the number of complicated works will increase.

summary

When using Redis in Laravel, you will need to adopt one of two types of libraries. "Predis" or "PhpRedis"
Previously, Predis was commonly used, but now PhpRedis is recommended. (even in official website)
PhpRedis cannot be managed by composer and must be installed by pecl.
Therefore, if you are using containers, you need to rebuild the image.
You may need to add "extension=redis.so" to php.ini, but it may work without it depend on the environment.

If you think that "Why am I having such a hard time just using the cache driver...?The front end is not important in this system",
I recommend consider adopting Predis.

Extra

One day, I got a "500 error" no matter what I did.
So I looked in laravel.log and found such a message.

local.ERROR: Please make sure the PHP Redis extension is installed and enabled. 
{"exception":"[object] (LogicException(code: 0): Please make sure the PHP Redis extension is installed and enabled. 
at /var/www/html/my-laravel-app/vendor/laravel/framework/src/Illuminate/Redis/Connectors/PhpRedisConnector.php:77)

Why? I didn't edit about redis configuration?
I wondered and looked into it, and it was caused by an error in reading the environment setting file due to editing .env.(The Parts that have nothing to do with Redis)

It's quite assertive that redis error is prioritized from all other errors and come to front...

Laravel, MySQL - column_type is set to "int (11)" even though the size of int was specified.

kakisoft — Tue, 24 May 2022 21:55:58 +0000

Environment
MySQL version : 5.7
Laravel version : 8.16.1
PHP version : 7.4.7

When adding table columns in the migration file in Laravel, int size can be specified like this.

Schema::table('projects', function (Blueprint $table) {
    $table->integer('category_code')->length(3);
});

However, looking at the actual generated MySQL schema, column_type is "int (11)".
What's going on? Did I make a mistake?

So, I looked it up, and found something like this.

https://stackoverflow.com/questions/25772759/schema-builder-length-of-an-integer

If you're using MySQL, you can't specify the length of an integer column.
You can only choose between one of the available integer types, described at http://dev.mysql.com/doc/refman/5.1/en/integer-types.html

In short, apparently "if you are using MySQL, you cannot specify the length".

This means that for some of the MySQL types, the size is already predetermined, and it cannot be changed as you like.
The table is here.

Type	Storage (Bytes)	Minimum Value Signed	Maximum Value Signed	Maximum Value Unsigned
TINYINT	1	-128	127	255
SMALLINT	2	-32768	32767	65535
MEDIUMINT	3	-8388608	8388607	16777215
INT	4	-2147483648	2147483647	4294967295
BIGINT	8	-263	263-1	264-1

Conclusion

"If you use int in MySQL, it will always be" int (11) ", but that's just the way MySQL works, so please bear that in mind.

So what about tinyint?
When I looked it up, I found something like this.

Database: Migrations - Laravel - The PHP Framework For Web Artisans

The following column types can be modified: bigInteger, binary, boolean, date, dateTime, dateTimeTz, decimal, integer, json, longText, mediumText, smallInteger, string, text, time, unsignedBigInteger, unsignedInteger, unsignedSmallInteger, and uuid. To modify a timestamp column type a Doctrine type must be registered.

In short, it says
"Cannot change from int to tinyint"
(The opposite is possible)

If you can't do the opposite, it means that the rollback doesn't work properly, so you should avoid it.

In fact, I also suffered from unnecessary errors.

So, if you use int or bigint, it is OK to default length.
(Rather, there is no other way.)

Note

If you want to change from int to tinyint, it is possible to do so without Laravel syntax, but using alter table.
For example:

DB::statement("alter table projects modify category_code tinyint;");

PHP, Docker - How to enable PCNTL(Process Control Extensions)

kakisoft — Tue, 03 May 2022 22:55:05 +0000

Environment
Laravel version : 8.16.1
PHP version : 7.4.7

Sometimes PCNTL(Process Control Extensions) is required to use specific features in Laravel.

(PHP: PCNTL - Manual)
https://www.php.net/manual/en/book.pcntl.php

For example, if you want to specify the job execution timeout period in Laravel, this function must be enabled.

The following configuration and source code shows that timeout period has been set.
(You can set the job to terminate as an error after X seconds.)

There are two ways.
one is to specify it from the artisan command when executing the job, and the other is to write it in the source code.

When using the artisan command, it is as follows.

php artisan queue:listen --timeout=30

When writing in source code, it is as follows.

namespace App\Jobs;

class ProcessPodcast implements ShouldQueue
{
    /**
     * The number of second that the job can execute until timeout period.
     *
     * @var int
     */
    public $timeout = 120;
}

As an aside, when these are used both, the source code value takes priority.

It means, in the case of the above example, it will time out in 120 seconds.

But, to use this feature, PCNTL(Process Control Extensions) must be enabled as described above, otherwise the timeout period will be 60 seconds by default.

For more information, please refer to the link.
https://laravel.com/docs/8.x/queues#timeout

How do I enable that feature? when you look into that, you will find "Please compile PHP".
It will discourage you.
https://www.php.net/manual/en/pcntl.installation.php

PHP Official

Process Control support in PHP is not enabled by default. You have to compile the CGI or CLI version of PHP with --enable-pcntl configuration option when compiling PHP to enable Process Control support.

Stack Overflow
How to enable pcntl in php ( while using a framework like Symfony2 )
How to enable PCNTL on Ubuntu server 16.04 - Stack Overflow

It's a daunting task for Docker developers.

However, it is possible to enable PCNTL(process control function) by editing the Dockerfile without recompiling PHP.

To be specific, add the following syntax.

Docker image was designed to use the official php-fpm distribution.

RUN docker-php-ext-configure pcntl --enable-pcntl \
  && docker-php-ext-install \
    pcntl

Example of editing the Dockerfile.

FROM php:7.4.11-fpm

# install composer
RUN cd /usr/bin && curl -s http://getcomposer.org/installer | php && ln -s /usr/bin/composer.phar /usr/bin/composer
RUN apt-get update \
&& apt-get install -y \
git \
zip \
unzip \
vim

RUN apt-get update \
    && apt-get install -y libpq-dev \
    && docker-php-ext-install pdo_mysql pdo_pgsql

RUN docker-php-ext-configure pcntl --enable-pcntl \
  && docker-php-ext-install \
    pcntl

WORKDIR /var/www/html

If you are using Docker-composer, rebuild it with a command such as "docker-compose up -d --build".

To check if pcntl is enabled use the following command after login to the container.

php -i | grep pcntl

If you see the following message, pcntl is enabled.

pcntl support => enabled

See below for php command options.
PHP: Options - Manual