Doris 踩坑系列

内容 隐藏
1 1.docker-compose重启
2 2.修改配置后进行重启
3 3.使用过程中用delete语句报错

背景：

Doris数据库无法写入数据，有张大表dwh_data_v2导致内存不足，手动进行重启Doris服务

根据官方docker-compose部署之后，重启FE BE之后启动异常 ，官方Docker Compose 脚本

1.docker-compose重启

docker-compose 重启之后 FE 一直尝试启动 无果

#报错日志信息
wait catalog to be ready. FE type: UNKNOWN. is ready: false

查询相关资料，发现无法启动原因为：

因为有多网卡的存在，或因为安装过 docker 等环境导致的虚拟网卡的存在，同一个主机可能存在多个不同的 ip。当前 Doris 并不能自动识别可用 IP。所以当遇到部署主机上有多个 IP 时，必须通过 priority_networks 配置项来强制指定正确的 IP。

上次启动时候，绑定的ip是 172.16.8.23，这次启动识别ip为 docker0 网卡地址，当前FE节点type为UNKNOWN，无法启动。

解决方法：

#docker-compose文件挂载fe的配置进行修改
version: "3"
services:
  fe:
    ......
    volumes:
     - /opt/apache-doris/fe/conf/:/opt/apache-doris/fe/conf/
     ......
  be:
    ......
     - /opt/apache-doris/be/conf/:/opt/apache-doris/be/conf/
     - /opt/apache-doris/be/bin/:/usr/local/bin/
    ......

修改FE的配置文件

vi fe/conf/fe.conf

# 最后一行配置(ip按照自己的来配置)：
priority_networks=172.16.8.0/24

同理BE的配置文件也需要进行修改

改FE的配置文件

vi be/conf/be.conf

# 最后一行配置(ip按照自己的来配置)：
priority_networks=172.16.8.0/24

2.修改配置后进行重启

发现be就无法注册到fe中了

doris-fe  | 2023-04-27 07:52:56,232 WARN (heartbeat mgr|34) [HeartbeatMgr.runAfterCatalogReady():139] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 10003, beHost: 172.30.80.3, bePort: 0, httpPort: 0, brpcPort: 0
doris-fe  | 2023-04-27 07:53:01,247 WARN (heartbeat-mgr-pool-4|220) [HeartbeatMgr$BackendHeartbeatHandler.call():268] backend heartbeat got exception
doris-fe  | org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
doris-fe  |     at org.apache.thrift.transport.TSocket.open(TSocket.java:226) ~[libthrift-0.13.0.jar:0.13.0]
doris-fe  |     at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:143) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:126) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:62) ~[commons-pool2-2.2.jar:2.2]
doris-fe  |     at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1012) ~[commons-pool2-2.2.jar:2.2]
doris-fe  |     at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) ~[commons-pool2-2.2.jar:2.2]
doris-fe  |     at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277) ~[commons-pool2-2.2.jar:2.2]
doris-fe  |     at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:203) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
doris-fe  |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
doris-fe  |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
doris-fe  |     at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
doris-fe  | Caused by: java.net.ConnectException: Connection refused (Connection refused)
doris-fe  |     at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_342]
doris-fe  |     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_342]
doris-fe  |     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_342]
doris-fe  |     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_342]
doris-fe  |     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_342]
doris-fe  |     at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_342]
doris-fe  |     at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ~[libthrift-0.13.0.jar:0.13.0]
doris-fe  |     ... 13 more
doris-fe  | 2023-04-27 07:53:01,247 WARN (heartbeat mgr|34) [HeartbeatMgr.runAfterCatalogReady():139] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 10003, beHost: 172.30.80.3, bePort: 0, httpPort: 0, brpcPort: 0

BE报错

doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: masterFe = 172.30.80.2
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: be_addr = 172.30.80.3:9050
doris-be  | 2023-04-27T07:30:46+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: feIpArray = 172.30.80.2
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: feEditLogPortArray = 9010
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: masterFe = 172.30.80.2:9010
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: be_addr = 172.30.80.3:9050
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: priority_networks = 172.30.80.0/24
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:31:07+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:31:07+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:31:07+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:31:27+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:31:27+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:31:27+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:31:47+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:31:47+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:31:47+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:32:08+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:32:08+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:32:08+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:32:28+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:32:28+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:32:28+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:32:49+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:32:49+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:32:49+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:33:09+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:33:09+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:33:09+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:33:29+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:33:29+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:33:29+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:33:50+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:33:50+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:33:50+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:34:10+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:34:10+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:34:10+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:34:31+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:34:31+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:34:31+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:34:51+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:34:51+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:34:51+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:35:11+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:35:11+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:35:11+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:35:32+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:35:32+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:35:32+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:35:52+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:36:13+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:36:33+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:36:53+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:37:14+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:37:34+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:37:55+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:38:15+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:38:36+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:38:56+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:39:16+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:39:37+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:39:57+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:40:18+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:40:38+00:00 [Note] [Entrypoint]: BE is not register. retr

解决方法

发现是官方的一个bug，有大佬回复：

You can’t change password in this Docker IMAGE. Because in BE initiation process, its will execute show frontends using mysql-client. Therefore, if you change the password, the initiation bash script couldn’t execute sql. So an error occurred and BE couldn’t startup normally.

在be容器里面, /usr/local/bin/中的entry_point.sh和init_be.sh脚本中, 使用mysql命令的部分, 加上-p[新密码]

至此重新启动服务之后，服务正常启动

[root@cnbjten7prodoris /opt/doris-compose]# docker-compose ps
NAME                 IMAGE                                COMMAND                 SERVICE   CREATED             STATUS             PORTS
doris-compose-be-1   apache/doris:2.0.0_alpha-be-x86_64   "bash entry_point.sh"   be        About an hour ago   Up About an hour
doris-compose-fe-1   apache/doris:2.0.0_alpha-fe-x86_64   "bash init_fe.sh"       fe        About an hour ago   Up About an hour

3.使用过程中用delete语句报错

ERROR 1105 (HY000): errCode = 2, detailMessage = failed to execute delete. transaction id 17113, timeout(ms) 30000, unfinished replicas [BackendId=TabletId]: 10003=29184, 10003=29186, 10003=29172, 10003=29188, 10003=29174

查询相关案例怀疑是目录权限的问题，进行宿主机挂载的配置文件目录进行授权

chmod 755 -R be/conf
chmod 755 -R fe/conf

重启服务后，进行尝试执行delet语句

MySQL [data_analysis]> delete from dwd_salary_info_115549 where owner_id = '115549' and business_id in ('sheet_178863241231998976');
Query OK, 0 rows affected (0.11 sec)
{'label':'delete_cc1dac54-588b-42f6-9ce4-ba9cb390873b', 'status':'VISIBLE', 'txnId':'18003'}

AD：《熊孩子博客》熊孩子QQ:443060965

赞0
如无特殊说明，文章均为本站原创，转载请注明出处
  • 转载请注明来源：Doris 踩坑系列
  • 本文永久链接地址：https://www.xionghaier.cn/archives/1351.html

该文章由 发布

这货来去如风，什么鬼都没留下！！！

相关推荐

发表我的评论

Hi，请填写昵称和邮箱！

取消评论
代码 贴图 加粗 链接 删除线 签到