DEV Community: Lemon

Apache Doris be common problem positioning and processing

Lemon — Fri, 24 Feb 2023 09:58:58 +0000

In the process of using doris, there will be BE(backend) down,But many users don’t know how to locate why BE is down, and don’t know how to deal with this situation, This article mainly introduces how to deal with BE down.

Among the users we are currently in contact with, there are two main reasons for BE down: 1. OOM KILLER killed the BE process , 2.BE Core Dump.

If we find that be is down, we first need to open the log/be.out file,If there is log information in it, then this is the second case,Just look at the BE Core Dump processing method,If there is no log information in it, it is likely to be OOM.

1、The first and most common case：OOM

For this OOM situation, we can check the system log through dmesg -T to confirm whether the be process is KILLed because of memory OOM.

# dmesg -T | less

You can see that the corresponding doris_be has an Out of memory error log, and RSS is the actual memory occupied by the BE process when it was KILL.Pay attention to this, check whether the time below is consistent with the time of BE down

#For example
[Fri Sep 2 22:58:19 2022] Out of memory: Killed process 1332190 (doris_be) total-vm:21484964304kB, anon-rss:91048156kB, file-rss:1484kB, shmem-rss:0kB, UID:0 pgtables:225764kB oom_score_adj:0
[Fri Sep 2 22:58:22 2022] oom_reaper: reaped process 1332190 (doris_be), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

If this is the reason, it is because BE did not protect itself. This mission is realized by memtracker, that is, to control the memory of the query, so that the overall process memory of BE does not exceed the mem_limit configured by the user (the default is 80%), so that it does not trigger The OOM KILLER of the system kills BE.

If the user encounters this problem in the short term, they can appropriately add an item in be’s conf/be.conf to explicitly reduce the maximum memory usage to reduce the trigger probability, and rely on Memtracker for thorough protection in the long run.

#Configure this item in be.conf of be and restart BE to take effect
mem_limit=60%

2、BE Core Dump

In this case, be.out will have corresponding stack information.

For versions before 1.1.5, there is no record of query_id in be.out.This is a bit complicated, I will specifically introduce how to locate and process in a later article.

In versions 1.1.5 and later, we printed the SQL query_id of be core in the stack information, so that we can use this query_id to find the corresponding SQL and the corresponding SQL in the fe.audit.log of fe Create a table statement, contact the community developers in slack for fast reading and recurrence positioning processing.

Specific steps are as follows：

step 1：

Find the TID (Thread ID) that generated the coredump in be.out

** SIGSEGV (@0x0) received by PID 31685 (TID 0x7f921416c700))

step 2：

Convert hexadecimal thread id to decimal

[linux-terminal] printf "%d\n" 0x7f921416c700
140265378989824

step 3：

grep decimal thread id in be.INFO then get query_id

[linux-terminal] grep 140265378989824 be.INFO
I1019 16:57:26.899314 260721 fragment_mgr.cpp:441] _exec_actual(): query_id=xxxxx-xxxx fragment_instance_id=xxxx-xxxxxx thread id by pthread_self()= 140265378989824

step 4：

Go to grep query_id in the fe.audit.log to find the corresponding sql

At this point you get the sql that causes be down

At this time, you need to ask your question in the github or slack community, and put the corresponding table creation statement, sql, stack information in be.out, etc.I will see your information in time in the community and give you a reply

If you want to further locate the reason, for example, it may be because the use of a function caused be down, you can restart be, modify your sql to verify your idea.

Finally,welcome to the slack community to discuss issues together: https://doris.apache.org/ welcome to give doris a star： https://github.com/apache/doris

How to use docker to compile Apache Doris

Lemon — Fri, 24 Feb 2023 09:55:23 +0000

There are more and more new features developed based on the doris master branch：https://github.com/apache/doris however, these new features have not been released in time to follow the version release,But for users, these new features are an urgent need to use.

As a developer in the Apache Doris community,I offer two possible suggestions,1: Ask questions in the slack community or private message me(email: liqianfeng@selectdb.com), the community will directly provide compiled binary packages for user testing. 2: Pull the code and compile it yourself. So this article teaches you how to use docker to compile doris.

Step 1 ：

Pull the doris code from the ～ directory of your server

[lqf@host ~]$ cd ~
[lqf@host ~]$ git clone https://github.com/apache/doris.git

Notice：Other directories can also be used here,

but the corresponding doris directory should be modified when starting docker

Step 2 ：Download the Docker image

[lqf@host ~]$ docker pull apache/doris:build-env-ldb-toolchain-latest

Step 3：Set Maven warehouse mount and output file directory mount and start docker

docker run -it -v ${your_home_directory}/.m2:/root/.m2 -v ${your_home_directory}/doris:/root/doris apache/doris:build-env-ldb-toolchain-latest
# For example
#my home directory
[lqf@host ~]$ pwd
/mnt/disk1/lqf
[lqf@host ~]$ docker run -it -v /mnt/disk1/lqf/.m2:/root/.m2 -v /mnt/disk1/lqf/doris:/root/doris apache/doris:build-env-ldb-toolchain-latest

Step 4：

After the step 3 is successfully executed, you should have entered docker

Now you need to pay attention,You need to confirm whether your docker has network by ping IP ：

[root@host ~]# ping www.google.com
PING www.google.com (172.217.27.36) 56(84) bytes of data.
64 bytes from sin11s03-in-f36.1e100.net (172.217.27.36): icmp_seq=1 ttl=117 time=1.42 ms
64 bytes from sin11s03-in-f36.1e100.net (172.217.27.36): icmp_seq=2 ttl=117 time=1.40 ms
^C
--- www.google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.404/1.412/1.420/0.008 ms
、
[root@host ~]# ping registry.npmjs.org
PING registry.npmjs.org (104.16.22.35) 56(84) bytes of data.
64 bytes from 104.16.22.35 (104.16.22.35): icmp_seq=1 ttl=56 time=1.67 ms
64 bytes from 104.16.22.35 (104.16.22.35): icmp_seq=2 ttl=56 time=1.53 ms

if there is not：

[root@docker ~]# ping www.google.com
ping: www.google.com: Name or service not known
[root@docker ~]# ping registry.npmjs.org
ping: registry.npmjs.org: Name or service not known

First you need to have a network on the host machine,And start docker in host mode.

# Add parameters --network host
# For example
[root@host ~]$ docker run  -it --network host -v /mnt/disk1/lqf/.m2:/root/.m2 -v /mnt/disk1/lqf/doris:/root/doris apache/doris:build-env-ldb-toolchain-latest

then switch JDK version(Choose one)

# switch to JDK 8：
[root@docker ~]$ alternatives --set java java-1.8.0-openjdk.x86_64
[root@docker ~]$ alternatives --set javac java-1.8.0-openjdk.x86_64
[root@docker ~]$ export JAVA_HOME=/usr/lib/jvm/java-1.8.0

# switch to JDK 11：
[root@docker ~]$ alternatives --set java java-11-openjdk.x86_64
[root@docker ~]$ alternatives --set javac java-11-openjdk.x86_64
[root@docker ~]$ export JAVA_HOME=/usr/lib/jvm/java-11

Step 5：start compiling
There will be a doris directory in docker

[root@docker ~]# cd doris/
[root@docker ~]# sh build.sh

During the compilation process, errors may be reported due to insufficient memory and other reasons.At this time, you only need to execute the command again “sh build.sh”

After the compilation is successful :

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  13:16 min
[INFO] Finished at: 2023-02-23T13:23:32Z
[INFO] ------------------------------------------------------------------------
Install auditloader...
Build Auditloader Finished
***************************************
Successfully build Doris
***************************************

Now you need to enter the doris directory of your host machine,There will be an output directory here.

[lqf@host ~]$ cd doris/
[lqf@host doris]$ cd output/
[lqf@host output]$ ll
total 20
drwxr-xr-x  6 root root 4096 Feb 23 21:10 apache_hdfs_broker
drwxr-xr-x  2 root root 4096 Feb 23 21:23 audit_loader
drwxr-xr-x 10 root root 4096 Feb 23 21:01 be
drwxr-xr-x 10 root root 4096 Feb 23 21:01 fe
drwxr-xr-x  4 root root 4096 Feb 23 21:01 udf

Now congratulations on your successful compilation, you can happily use doris.

Finally, if you get unresolvable errors during compilation,You need to ask your questions in the slack community, the slack entry can be found on the Apache Doris official website:https://doris.apache.org/ , Or you can also private message me directly: liqianfeng@selectdb.com