The server environment is dangerous to run applications with the root account, making it easy for the shell to become a chicken. So teams with a little bit of awareness build a low-level user to run JAVA programs.
Low Clearance, a bit unlike his own son, especially in difficult times of resource constraints can be seen.
Phenomenon
The problem is that it happened on a machine in a public test environment, and the formal environment is not replicated. This server deployed dozens of services, and the deployment account recently switched from root to Xjbot.
After running it for a while, the server frequently had problems. First, a large number of connections are in the CLOSE state, which was once thought to be a passive shutdown problem. But it's not.
NETSTAT-ANTP | GREP CLOSE | Awk'{ print $7}'| sort | UNIQ-C
Oddly, logging in using the root account or other accounts works just fine. However, when switching to an xjbot account, the following error is reported:
Sudo su-xjjbot
Bash: Fork : no child processes
Bash: Fork : no child processes
Bash: Fork : no child processes
Bash: Fork : no child processes
Bash: Fork Resource temporarily unavailable
This is a system-level error message. In this case, the JVM will also have the corresponding error, but I am afraid you have no chance to see (you can use other system users to see oh) .
CAN NOT CREATE GC thread. Out of system resources
Java. (phone rings) Lang. OUTOFMEMORYERROR: unable to create new native thread
Why
The cause is the lack of resources, specifically the process of resources.
A Linux thread is actually a process, so Java is also, specifically, called "light weight process (LWP) " -- Light-weight process.
LWP shares all (or most) of the logical address space and system resources with other processes. A process can create multiple lwps so that they share most of the resources; LWP has its own process identifier and has parent-child relationships with other processes; . . Lwp Is kernel-managed and scheduled as a normal process
Use The following command to see how many process resources are being used by a user
Ps-eLf | GREP XJJBOT (UID) | wc-l
Use The following command to see how many threads each process has started
PS-O NLWP, Pid, lwp, args-u xjjbot (UID) | sort-n
Fix It
According to the Linux everything is a file rule, the first thing that comes to mind is modifying the ULIMIT parameters, but it's not, because it's already big enough. If you think about Elasticsearch, you need to configure something called NPROC when you install it, and that's probably where the problem is that you don't have enough process resources.
Related Profiles:
/ etc / Security / limits. CONF
There are also minor differences between kernel versions. Like what
/ etc / Security / limits. D / *
Will, at some point, overwrite limits. CONF configuration. So if the configuration doesn't work, be sure to check.
For these reasons, you can use limits. All the configurations in D are commented out and unified in limits. CONF configuration.
Here's the original configuration
- Soft NPROC 4096
Root soft NPROC unlimited
Change 4096 to a larger number, or simply change it to unlimited.
Configuration of ElasticSearch system parameters
Now that we mention es, let's take a look at what system configurations need to be changed for an es installation. These experiences are shared and can be extrapolated.
https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html
Disable swap
Swap is a performance killer, so the ES can't take it anymore. Just turn it off.
Sudo SWAPOFF-A
This parameter can also be added to the configuration file, and the JVM locks the memory so that it can not be swapped with the swap partition.
Bootstrap. MEMORY: True
Virtual memory
Es Uses mmapfs to map some data, but the default system parameters are too small for it and need to be modified.
SYSCTL-W VM. Max
To take effect permanently, you need to modify / etc / SYSCTL. CONF
File handle
ULIMIT
There is a limit to the number of File descriptor open on Linux. If your application is dealing with many small files at the same time, you need to configure this parameter.
Sudo Su
ULIMIT-N 65536
Su elasticsearch
/ etc / Security / limits. CONF
Ok, here's the file we just changed. To make the above configuration permanent, you need to change this file.
65536
Number of threads
As we mentioned above, I can think of it quickly because I installed es-. -
Therefore, do not open a large number of random threads, in addition to increase scheduling time, but also easy to top the ceiling of the system.
Under von Neumann's framework, don't they all work the same way?
With the same fate, struggling but unable to escape.
Top comments (0)