三节点完全分布式Hadoop高可用集群部署

大数据 Hadoop

1 集群规划

1.1 集群资源规划

主机名	IP地址	资源	操作系统
hadoop01	192.168.2.11	2C4G	OpenEuler 2403
hadoop02	192.168.2.12	2C4G	OpenEuler 2403
hadoop03	192.168.2.13	2C4G	OpenEuler 2403

1.2 节点服务规划

节点	hadoop01	hadoop02	hadoop03
NameNode	√	√	×
DataNode	√	√	√
ResourceManager	×	√	√
NodeManager	√	√	√
JournalNode	√	√	√
ZooKeeper	√	√	√
ZKFC	√	√	×
JobHistoryServer	×	×	√

1.3 各组件版本

组件	版本
Hadoop	v3.3.6
ZooKeeper	v3.8.4
Java	v1.8

2 环境配置

除非特别说明，否则三个节点都要执行！

2.1 关闭防火墙

systemctl disable firewalld --now

2.2 关闭 SELinux

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

2.3 修改主机名

# 192.168.2.11执行
hostnamectl set-hostname hadoop01

# 192.168.2.12执行
hostnamectl set-hostname hadoop02

# 192.168.2.13执行
hostnamectl set-hostname hadoop03

2.4 添加域名映射

cat >> /etc/hosts << EOF
192.168.2.11 hadoop01
192.168.2.12 hadoop02
192.168.2.13 hadoop03
EOF

2.5 配置时间同步

2.5.1 配置服务端

在 hadoop01 节点执行：

vim /etc/chrony.conf

修改以下内容：

server ntp.aliyun.com iburst #（有外网/时钟服务器的情况下）
allow 192.168.2.0/24
local stratum 10
leapsectz right/UTC

2.5.2 配置客户端

在 hadoop02、hadoop03 节点执行：

vim /etc/chrony.conf

修改以下内容：

server hadoop01 iburst
leapsectz right/UTC

2.5.3 重启验证

systemctl start chronyd && sudo systemctl enable chronyd

hadoop02、hadoop03 节点验证：

chronyc sources -v

正常输出内容为：^* hadoop01

2.6 创建用户并配置免密

2.6.1 创建用户和用户组

创建 hadoop 用户：

# 创建用户组
groupadd hadoop

# 创建 hadoop 用户
useradd -g hadoop -s /bin/bash -m -d /home/hadoop hadoop

# 设置用户密码
passwd hadoop

2.6.2 配置免密登录

设置 root 用户免密登录，用于方便传输文件。

# 切换root用户
su - root

# 生成密钥
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

# 分别分发公钥到其它节点，包括自身
ssh-copy-id root@hadoop01
ssh-copy-id root@hadoop02
ssh-copy-id root@hadoop03

设置 hadoop 用户免密登录，用于脚本远程启动服务。

# 切换hadoop用户
su - hadoop

# 所有节点生成密钥
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

# 三节点分别分发公钥到所有节点（包括自己）
ssh-copy-id hadoop@hadoop01
ssh-copy-id hadoop@hadoop02
ssh-copy-id hadoop@hadoop03

# 验证免密登录其它节点
ssh hadoop@hadoop02
ssh hadoop@hadoop03

2.7 安装Java环境

# 切换root用户
su - root

# 解压Java8安装包
tar zxf jdk-8u361-linux-x64.tar.gz -C /opt/jdk1.8.0_361

3 部署Hadoop HA集群

3.1 安装ZooKeeper

三个节点都要安装：

# 切换root用户
su - root

# 解压安装包
tar zxf apache-zookeeper-3.8.4-bin.tar.gz -C /opt/

# 创建软连接
cd /opt && ln -s apache-zookeeper-3.8.4-bin /opt/zookeeper

修改/添加 zoo.cfg 配置：

cd /opt/zookeeper/conf && cp zoo_sample.cfg zoo.cfg && vim zoo.cfg

内容如下：

clientPort=2181
autopurge.snapRetainCount=5
autopurge.purgeInterval=24
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
#事务日志存放目录
dataLogDir=/data/zookeeper/logs/txlog

# 增加如下配置
server.1=hadoop01:2888:3888
server.2=hadoop02:2888:3888
server.3=hadoop03:2888:3888

修改日志文件路径（可选）：

vim /opt/zookeeper/logback.xml

# 修改日志输出目录（默认是zookeeper安装目录下的logs）
<property name="zookeeper.log.dir" value="/data/zookeeper/logs" />

创建目录：

mkdir -p /data/hadoop/zookeeper/data
mkdir -p /data/hadoop/zookeeper/logs/txlog

设置 myid：

# hadoop01主机执行
echo 1 > /data/zookeeper/data/myid

# hadoop02主机执行
echo 2 > /data/zookeeper/data/myid

# hadoop03主机执行
echo 3 > /data/zookeeper/data/myid

# 修改目录所属
chown -R hadoop:hadoop /data/zookeeper
chown -R hadoop:hadoop /opt/apache-zookeeper-3.8.4-bin

3.2 安装Hadoop

三个节点都要安装：

# 切换root用户
su - root

# 解压安装包
tar zxf hadoop-3.3.6.tar.gz -C /opt/

cd /opt && ln -s hadoop-3.3.6 /opt/hadoop

3.2.1 修改 core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!-- 指定 HDFS 的逻辑集群名称 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop-ha</value>
    </property>
    <!-- 指定 hadoop 运行时产生文件的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/hadoop/tmp</value>
    </property>
    <!-- 指定 zkfc 要连接的 zkServer 地址 -->
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>
    </property>
    <!--指定Web页面使用的用户 -->
    <property>
        <name>hadoop.http.staticuser.user</name>
        <value>hadoop</value>
    </property>
    <property>
        <name>hadoop.http.cross-origin.enabled</name>
        <value>true</value>
    </property>
</configuration>

3.2.2 修改 hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!-- 完全分布式集群名称 -->
    <property>
        <name>dfs.nameservices</name>
        <value>hadoop-ha</value>
    </property>
    <!-- 集群中 NameNode 节点都有哪些 -->
    <property>
        <name>dfs.ha.namenodes.hadoop-ha</name>
        <value>nn1,nn2</value>
    </property>
    <!-- NameNode 的 RPC 通信地址 -->
    <property>
        <name>dfs.namenode.rpc-address.hadoop-ha.nn1</name>
        <value>hadoop01:8020</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.hadoop-ha.nn2</name>
        <value>hadoop02:8020</value>
    </property>
    <!-- NameNode 的 http 通信地址 -->
    <property>
        <name>dfs.namenode.http-address.hadoop-ha.nn1</name>
        <value>hadoop01:9870</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.hadoop-ha.nn2</name>
        <value>hadoop02:9870</value>
    </property>
    <!-- JournalNode 共享编辑日志地址 -->
    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://hadoop01:8485;hadoop02:8485;hadoop03:8485/hadoop-ha</value>
    </property>
    <!-- 客户端故障转移代理 -->
    <property>
        <name>dfs.client.failover.proxy.provider.hadoop-ha</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <!-- 配置故障转移隔离方法，即同一时刻只能有一台服务器对外响应 -->
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>
            sshfence
        </value>
    </property>
    <!-- 使用隔离机制时需要ssh秘钥登录 -->
    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/hadoop/.ssh/id_rsa</value>
    </property>
    <!-- 启用nn故障自动转移 -->
    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>
    <!-- 指定HDFS副本数量 -->
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <!-- NameNode 数据存储目录 -->
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/data/hadoop/nn</value>
    </property>
    <!-- DataNode 数据存储目录 -->
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/data/hadoop/dn</value>
    </property>
    <!-- JournalNode 日志存储目录 -->
    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/data/hadoop/jn</value>
    </property>
    <!-- 块大小（128MB） -->
    <property>
        <name>dfs.blocksize</name>
        <value>134217728</value>
    </property>
    <!--开启 Web UI -->
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

3.2.3 修改 mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!-- 指定MapReduce程序运行在Yarn上 -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <!-- 历史服务器端地址 -->
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop03:10020</value>
    </property>
    <!-- 历史服务器web端地址 -->
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop03:19888</value>
    </property>
    <!-- 已完成作业的日志存放目录（HDFS） -->
    <property>
        <name>mapreduce.jobhistory.done-dir</name>
        <value>/jobhistory/done</value>
        <description>已完成作业的日志永久存储路径</description>
    </property>
    <!-- 正在运行作业的临时日志目录（HDFS） -->
    <property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>/jobhistory/done_intermediate</value>
        <description>作业运行期间的临时日志路径，完成后会迁移到done-dir</description>
    </property>
    <!-- 日志清理策略（可选） -->
    <property>
        <name>mapreduce.jobhistory.cleaner.enable</name>
        <value>true</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.cleaner.interval-ms</name>
        <value>86400000</value>
        <description>清理线程运行间隔，单位毫秒（默认1天）</description>
    </property>
    <!--日志保留时间（14d） -->
    <property>
        <name>mapreduce.jobhistory.max-age-ms</name>
        <value>1209600000</value>
        <description>作业历史记录保留时间，单位毫秒（默认7天）</description>
    </property>
</configuration>

3.2.4 修改 yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<configuration>
    <!-- 启用 resourcemanager ha -->
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>
    <!-- 声明两台 resourcemanager 的地址 -->
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yarn-cluster</value>
    </property>
    <!--指定 resourcemanager 的逻辑列表 -->
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>
    <!-- ========== rm1 的配置 ========== -->
    <!-- 指定 rm1 的主机名 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>hadoop02</value>
    </property>
    <!-- 指定 rm1 的 web 端地址（默认8088） -->
    <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>hadoop02:8088</value>
    </property>
    <!-- 指定 AM 向 rm1 申请资源的地址 -->
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm1</name>
        <value>hadoop02:18030</value>
    </property>
    <!-- 指定供 NM 连接的地址（默认8031） -->
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
        <value>hadoop02:8031</value>
    </property>
    <!-- 指定 rm1 的内部通信地址（默认8032） -->
    <property>
        <name>yarn.resourcemanager.address.rm1</name>
        <value>hadoop02:8032</value>
    </property>
    <!-- RM 管理命令端口（默认8033） -->
    <property>
        <name>yarn.resourcemanager.admin.address.rm1</name>
        <value>hadoop:8033</value>
    </property>
    <!-- ========== rm2 的配置 ========== -->
    <!-- 指定 rm2 的主机名 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>hadoop03</value>
    </property>
    <!-- 指定 rm1 的 web 端地址（默认8088） -->
    <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>hadoop03:8088</value>
    </property>
    <!-- 指定 AM 向 rm1 申请资源的地址（默认8030） -->
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm2</name>
        <value>hadoop03:8030</value>
    </property>
    <!-- 指定供 NM 连接的地址（默认8031） -->
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
        <value>hadoop03:8031</value>
    </property>
    <!-- 指定 rm1 的内部通信地址（默认8032） -->
    <property>
        <name>yarn.resourcemanager.address.rm2</name>
        <value>hadoop03:8032</value>
    </property>
    <!-- RM 管理命令端口（默认8033） -->
    <property>
        <name>yarn.resourcemanager.admin.address.rm2</name>
        <value>hadoop03:8033</value>
    </property>
    <!-- 容器本地化服务端口（默认8040） -->
    <property>
        <name>yarn.nodemanager.localizer.address</name>
        <value>0.0.0.0:8040</value>
    </property>
    <!-- 指定 zookeeper 集群的地址 -->
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>
    </property>
    <!-- 启用自动恢复 -->
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>
    <!-- 指定 resourcemanager 的状态信息存储在 zookeeper 集群 -->
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>
    <!-- 环境变量的继承 -->
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
    <!-- NodeManager 辅助服务 -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
        <value>org.apache.spark.network.yarn.YarnShuffleService</value>
    </property>
    <!-- =========资源限制========= -->
    <!-- NodeManager可用于运行容器的总内存（MB） -->
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
    </property>
    <!-- NodeManager可用于运行容器的总 vcores 数 -->
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>1</value>
    </property>
    <!-- 每个容器允许的最小内存（MB） -->
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>512</value>
    </property>
    <!-- 每个容器允许的最小vcores -->
    <property>
        <name>yarn.scheduler.minimum-allocation-vcores</name>
        <value>1</value>
    </property>
    <!-- 每个容器允许的最大内存（MB） -->
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>2048</value>
    </property>
    <!-- 每个容器允许的最大vcores -->
    <property>
        <name>yarn.scheduler.maximum-allocation-vcores</name>
        <value>1</value>
    </property>
    <!--是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>true</value>
    </property>
    <!--是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <!-- ========== NodeManager 本地工作目录 ========== -->
    <!-- Spark Shuffle、临时文件、Container 本地化数据 -->
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/data/hadoop/yarn/local</value>
    </property>
    <!-- ========== Container 日志目录 ========== -->
    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/data/hadoop/logs/yarn/container-logs</value>
    </property>
    <!-- 作业完成后临时文件清理 -->
    <property>
        <name>yarn.nodemanager.delete.debug-delay-sec</name>
        <value>0</value> <!-- 立即清理 -->
    </property>
    <!--日志聚集功能 -->
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <!-- 设置日志聚集服务器地址 -->
    <property>
        <name>yarn.log.server.url</name>
        <value>http://hadoop03:19888/jobhistory/logs</value>
    </property>
    <!--日志保留时间设置14天 -->
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>1209600</value>
    </property>
    <!-- 日志聚合检查间隔（秒） -->
    <property>
        <name>yarn.log-aggregation.retain-check-interval-seconds</name>
        <value>3600</value>
    </property>
</configuration>

3.2.5 修改 workers

hadoop01
hadoop02
hadoop03

3.2.6 修改 hadoop-env.sh

export JAVA_HOME=/opt/jdk1.8.0_361
export HADOOP_HOME=/opt/hadoop
export HADOOP_HEAPSIZE_MAX=2048

# 统一日志根目录（默认是 $HADOOP_HOME/logs）
export HADOOP_LOG_DIR=/data/hadoop/logs

# 统一 PID 文件目录
export HADOOP_PID_DIR=/data/hadoop/pid

# 各守护进程堆内存与日志目录。
# 注意：值末尾追加 $XXX_OPTS，用于保留 hadoop-env.sh 原有的安全/审计日志配置
export HDFS_NAMENODE_OPTS="-Xmx1g -Xms1g -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -Dhadoop.log.dir=${HADOOP_LOG_DIR}/nn -Xloggc:${HADOOP_LOG_DIR}/nn/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M $HDFS_NAMENODE_OPTS"
export HDFS_DATANODE_OPTS="-Xmx1g -Xms1g -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -Dhadoop.log.dir=${HADOOP_LOG_DIR}/dn -Xloggc:${HADOOP_LOG_DIR}/dn/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M $HDFS_DATANODE_OPTS"
export HDFS_JOURNALNODE_OPTS="-Xmx256m -Xms256m -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -Dhadoop.log.dir=${HADOOP_LOG_DIR}/jn -Xloggc:${HADOOP_LOG_DIR}/jn/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M $HDFS_JOURNALNODE_OPTS"
export HDFS_ZKFC_OPTS="-Xmx256m -Xms256m -XX:+UseG1GC -XX:MaxGCPauseMillis=400 -Dhadoop.log.dir=${HADOOP_LOG_DIR}/zkfc $HDFS_ZKFC_OPTS"
export YARN_RESOURCEMANAGER_OPTS="-Dhadoop.log.dir=${HADOOP_LOG_DIR}/rm $YARN_RESOURCEMANAGER_OPTS"
export YARN_NODEMANAGER_OPTS="-Dhadoop.log.dir=${HADOOP_LOG_DIR}/nm $YARN_NODEMANAGER_OPTS"
export YARN_PROXYSERVER_OPTS="-Dhadoop.log.dir=${HADOOP_LOG_DIR}/proxyserver $YARN_PROXYSERVER_OPTS"
export MAPRED_HISTORYSERVER_OPTS="-Dhadoop.log.dir=${HADOOP_LOG_DIR}/jobhistory $MAPRED_HISTORYSERVER_OPTS"

# 客户端/守护进程公共 JVM 参数
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"

说明：
堆内存必须根据服务器物理内存规格进行配置；HADOOP_HEAPSIZE_MAX 仅对没有单独指定 -Xmx 的守护进程生效。

3.3 创建目录并配置环境变量

创建Hadoop、Spark数据目录并修改目录所属：

mkdir -p /data/hadoop/{tmp,nn,dn,jn,logs}
mkdir -p /data/hadoop/logs/{nn,dn,jn,rm,nm,proxyserver,jobhistory}
mkdir -p /data/hadoop/logs/yarn/container-logs
mkdir -p /data/hadoop/yarn/local
mkdir -p /data/hadoop/pid
chown -R hadoop:hadoop /data/hadoop
chown -R hadoop:hadoop /opt/hadoop-3.3.6

配置环境变量：

cat > /etc/profile.d/hadoop_env.sh << EOF
# JAVA_HOME
export JAVA_HOME=/opt/jdk1.8.0_361

# zookeeper
export ZK_HOME=/opt/zookeeper

# HADOOP_HOME
export HADOOP_HOME=/opt/hadoop

export PATH=$JAVA_HOME/bin:$ZK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
EOF

#使环境变量生效
source /etc/profile.d/hadoop_env.sh

3.4 启动Hadoop HA集群

3.4.1 三个节点启动zookeeper

su - hadoop
zkServer.sh start

# 验证节点状态
zkServer.sh status

3.4.2 三个节点启动journalnode

hdfs --daemon start journalnode

3.4.3 在hadoop01执行初始化namenode并启动

注意：仅配置完毕第一次启动时执行

hdfs namenode -format

启动 namenode：

hdfs --daemon start namenode

3.4.4 在hadoop02同步元数据

hdfs namenode -bootstrapStandby

3.4.5 在hadoop01初始化zkfc

hdfs zkfc -formatZK

3.4.6 在hadoop01启动HDFS

先停止 hadoop01 的 namenode 进程再使用脚本一键启动 hdfs 集群：

hdfs --daemon stop namenode
start-dfs.sh

3.4.7 在hadoop02启动YARN

start-yarn.sh

3.4.8 在hadoop03启动job历史服务器

HDFS 创建工作历史目录：

hdfs dfs -mkdir -p /jobhistory/done
hdfs dfs -mkdir -p /jobhistory/done_intermediate

# 设置权限（统一用 hadoop 用户）
hdfs dfs -chown -R hadoop:hadoop /jobhistory
hdfs dfs -chmod -R 755 /jobhistory/done
hdfs dfs -chmod -R 1777 /jobhistory/done_intermediate

# 验证
hdfs dfs -ls /

# hadoop03启动
mapred --daemon start historyserver

3.4.9 查看组件状态

查看 namenode 状态：

hdfs haadmin -getAllServiceState

浏览器访问 namenode：

查看 ResourceManager 状态：

yarn rmadmin -getServiceState rm1
yarn rmadmin -getServiceState rm2

浏览器访问 resourcemanager：

访问这两个地址会跳转到另一个地址是正常的，因为它们是主备，访问备用节点会跳转到主节点

查看历史服务器：

http://192.168.2.13:19888/jobhistory

3.5 配置开机自启

3.5.1 停止Hadoop服务

1、停止 JobHistoryServer【hadoop03 执行】

mapred --daemon stop historyserver

2、停止 yarn【hadoop01 执行】

stop-yarn.sh

3、停止 hdfs【hadoop01 执行】

stop-dfs.sh

4、停止 zookeeper【三节点执行】

zkServer.sh stop

3.5.2 配置systemd服务

在 hadoop01、hadoop02、hadoop03 节点创建 zookeeper.service：

cat > /etc/systemd/system/zookeeper.service << EOF
[Unit]
Description=Apache ZooKeeper Server
Documentation=https://zookeeper.apache.org/
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
User=hadoop
Group=hadoop
Environment="JAVA_HOME=/opt/jdk"
PIDFile=/data/hadoop/zookeeper/data/zookeeper_server.pid
Environment="ZOO_LOG_DIR=/data/zookeeper/logs"
ExecStart=/opt/zookeeper/bin/zkServer.sh start
ExecStop=/opt/zookeeper/bin/zkServer.sh stop
ExecReload=/opt/zookeeper/bin/zkServer.sh restart
Restart=on-failure
RestartSec=30
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target
EOF

在 hadoop01、hadoop02 节点创建 namenode.service：

cat > /etc/systemd/system/namenode.service << EOF
[Unit]
Description=Apache Hadoop NameNode Service
Documentation=https://hadoop.apache.org/
After=network-online.target zookeeper.service journalnode.service
Wants=network-online.target zookeeper.service journalnode.service

[Service]
Type=forking
User=hadoop
Group=hadoop
Environment="JAVA_HOME=/opt/jdk1.8.0_361"
WorkingDirectory=/opt/hadoop
PIDFile=/data/hadoop/pid/hadoop-hadoop-namenode.pid
ExecStart=/opt/hadoop/bin/hdfs --daemon start namenode
ExecStop=/opt/hadoop/bin/hdfs --daemon stop namenode
Restart=on-failure
RestartSec=30
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target
EOF

在 hadoop01、hadoop02 节点创建 zkfc.service：

cat > /etc/systemd/system/zkfc.service << EOF
[Unit]
Description=Apache Hadoop ZK Failover Controller Service
Documentation=https://hadoop.apache.org/
After=network-online.target zookeeper.service journalnode.service
Wants=network-online.target zookeeper.service journalnode.service

[Service]
Type=forking
User=hadoop
Group=hadoop
Environment="JAVA_HOME=/opt/jdk1.8.0_361"
WorkingDirectory=/opt/hadoop
PIDFile=/data/hadoop/pid/hadoop-hadoop-zkfc.pid
ExecStart=/opt/hadoop/bin/hdfs --daemon start zkfc
ExecStop=/opt/hadoop/bin/hdfs --daemon stop zkfc
Restart=on-failure
RestartSec=30
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target
EOF

在三个节点创建 journalnode.service：

cat > /etc/systemd/system/journalnode.service << EOF
[Unit]
Description=Apache Hadoop JournalNode Service
Documentation=https://hadoop.apache.org/
After=network-online.target zookeeper.service
Wants=network-online.target zookeeper.service

[Service]
Type=forking
User=hadoop
Group=hadoop
Environment="JAVA_HOME=/opt/jdk1.8.0_361"
WorkingDirectory=/opt/hadoop
PIDFile=/data/hadoop/pid/hadoop-hadoop-journalnode.pid
ExecStart=/opt/hadoop/bin/hdfs --daemon start journalnode
ExecStop=/opt/hadoop/bin/hdfs --daemon stop journalnode
Restart=on-failure
RestartSec=30
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target
EOF

在三个节点创建 datanode.service：

cat > /etc/systemd/system/datanode.service << EOF
[Unit]
Description=Apache Hadoop DataNode Service
Documentation=https://hadoop.apache.org/
After=network-online.target namenode.service
Wants=network-online.target namenode.service

[Service]
Type=forking
User=hadoop
Group=hadoop
Environment="JAVA_HOME=/opt/jdk1.8.0_361"
WorkingDirectory=/opt/hadoop
PIDFile=/data/hadoop/pid/hadoop-hadoop-datanode.pid
ExecStart=/opt/hadoop/bin/hdfs --daemon start datanode
ExecStop=/opt/hadoop/bin/hdfs --daemon stop datanode
Restart=on-failure
RestartSec=30
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target
EOF

在 hadoop02、hadoop03 节点创建 resourcemanager.service：

cat > /etc/systemd/system/resourcemanager.service << EOF
[Unit]
Description=Apache Hadoop ResourceManager Service
Documentation=https://hadoop.apache.org/
After=network-online.target zookeeper.service
Wants=network-online.target zookeeper.service

[Service]
Type=forking
User=hadoop
Group=hadoop
Environment="JAVA_HOME=/opt/jdk1.8.0_361"
WorkingDirectory=/opt/hadoop
PIDFile=/data/hadoop/pid/hadoop-hadoop-resourcemanager.pid
ExecStart=/opt/hadoop/bin/yarn --daemon start resourcemanager
ExecStop=/opt/hadoop/bin/yarn --daemon stop resourcemanager
Restart=on-failure
RestartSec=30
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target
EOF

在三个节点创建 nodemanager.service：

cat > /etc/systemd/system/nodemanager.service << EOF
[Unit]
Description=Apache Hadoop NodeManager Service
Documentation=https://hadoop.apache.org/
After=network-online.target resourcemanager.service
Wants=network-online.target resourcemanager.service

[Service]
Type=forking
User=hadoop
Group=hadoop
Environment="JAVA_HOME=/opt/jdk1.8.0_361"
WorkingDirectory=/opt/hadoop
PIDFile=/data/hadoop/pid/hadoop-hadoop-nodemanager.pid
ExecStart=/opt/hadoop/bin/yarn --daemon start nodemanager
ExecStop=/opt/hadoop/bin/yarn --daemon stop nodemanager
Restart=on-failure
RestartSec=30
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target
EOF

在 hadoop03 节点创建 jobhistoryserver.service：

cat > /etc/systemd/system/jobhistoryserver.service << EOF
[Unit]
Description=Apache Hadoop JobHistoryServer Service
Documentation=https://hadoop.apache.org/
After=network-online.target resourcemanager.service
Wants=network-online.target resourcemanager.service

[Service]
Type=forking
User=hadoop
Group=hadoop
Environment="JAVA_HOME=/opt/jdk1.8.0_361"
WorkingDirectory=/opt/hadoop
PIDFile=/data/hadoop/pid/hadoop-hadoop-historyserver.pid
ExecStart=/opt/hadoop/bin/mapred --daemon start historyserver
ExecStop=/opt/hadoop/bin/mapred --daemon stop historyserver
Restart=on-failure
RestartSec=30
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target
EOF

配置开机自启：

# 重新加载配置文件
systemctl daemon-reload

# 配置开机自启
systemctl enable namenode zkfc journalnode datanode resourcemanager nodemanager jobhistoryserver

按顺序启动 Hadoop 集群：

# 启动zookeeper【hadoop01、hadoop02、hadoop03】
systemctl start zookeeper

# 启动journalnode【hadoop01、hadoop02、hadoop03】
systemctl start journalnode

# 启动namenode【hadoop01、hadoop02】
systemctl start namenode

# 启动zkfc【hadoop01、hadoop02】
systemctl start zkfc

# 启动datanode【hadoop01、hadoop02、hadoop03】
systemctl start datanode

# 启动resourcemanager【hadoop02、hadoop03】
systemctl start resourcemanager

# 启动nodemanager【hadoop01、hadoop02、hadoop03】
systemctl start nodemanager

# 启动jobhistoryserver【hadoop03】
systemctl start jobhistoryserver

3.6 验证集群高可用性

3.6.1 验证HDFS高可用

假设当前 hadoop01 节点的 namenode 为 active 状态，hadoop02 节点的 namenode 为 standby 状态。

停止 hadoop01 节点的 namenode 服务：

systemctl stop namenode

浏览器访问 hadoop02 的 namenode 服务：

http://192.168.2.12:9870

查看页面状态是否由 standby 变为 active。

3.6.2 验证YARN高可用

假设当前 hadoop02 节点的 resourcemanager 为 active 状态，hadoop03 节点的 resourcemanager 为 standby 状态。

停止 hadoop02 节点的 resourcemanager 服务：

systemctl stop resourcemanager

浏览器访问 hadoop03 的 resourcemanager 服务：

http://192.168.2.13:8088

结果验证：

如果页面能直接访问，说明 hadoop03 已切换为 active，否则页面会跳转到 hadoop02:8088

或者执行命令查看主备状态：

yarn rmadmin -getAllServiceState

如果觉得文章对你有用，请随意赞赏

Hadoop

三节点完全分布式Hadoop高可用集群部署

https://www.zyifei.top/archives/c610e4c6-c98e-4415-a113-6fbb090d2f85

作者

朝阳

发布于

2026年08月07日 21时04分

更新于

2026年09月18日 23时08分

许可协议

CC BY 4.0

三节点完全分布式Hadoop高可用集群部署

1 集群规划

1.1 集群资源规划

1.2 节点服务规划

1.3 各组件版本

2 环境配置

2.1 关闭防火墙

2.2 关闭 SELinux

2.3 修改主机名

2.4 添加域名映射

2.5 配置时间同步

2.5.1 配置服务端

2.5.2 配置客户端

2.5.3 重启验证

2.6 创建用户并配置免密

2.6.1 创建用户和用户组

2.6.2 配置免密登录

2.7 安装Java环境

3 部署Hadoop HA集群

3.1 安装ZooKeeper

3.2 安装Hadoop

3.2.1 修改 core-site.xml

3.2.2 修改 hdfs-site.xml

3.2.3 修改 mapred-site.xml

3.2.4 修改 yarn-site.xml

3.2.5 修改 workers

3.2.6 修改 hadoop-env.sh

3.3 创建目录并配置环境变量

3.4 启动Hadoop HA集群

3.4.1 三个节点启动zookeeper

3.4.2 三个节点启动journalnode

3.4.3 在hadoop01执行初始化namenode并启动

3.4.4 在hadoop02同步元数据

3.4.5 在hadoop01初始化zkfc

3.4.6 在hadoop01启动HDFS

3.4.7 在hadoop02启动YARN

3.4.8 在hadoop03启动job历史服务器

3.4.9 查看组件状态

3.5 配置开机自启

3.5.1 停止Hadoop服务

3.5.2 配置systemd服务

3.6 验证集群高可用性

3.6.1 验证HDFS高可用

3.6.2 验证YARN高可用

作者

发布于

更新于

许可协议

评论