hadoop集群环境部署,linux环境下
墨初 知识笔记 104阅读
Linux环境下Redis 集群部署 1.单机Redis部署2.Redis 集群配置2.1 创建redis集群安装目录2.2 将redis单机部署目录下的redis.confi文件复制到每个目录下2.3 修改每个文件夹下的redis.conf2.4 修改完六个配置内容后开始启动2.5 启动完后查看进程2.6 建集群
1.单机Redis部署
Linux下redis安装并设置开机自启
2.Redis 集群配置具体是参考这篇文章由于在搭建过程中仍遇到很多问题所以在其基础上补充
Linux下redis集群搭建与部署
2.1 创建redis集群安装目录 cd /home/xh/hadoop/
mkdir myredis
cd myredis
mkdir 7000 7001 7002 7003 7004 7005
cp /home/xh/hadoop/redis-3.2.5/redis.conf /home/xh/hadoop/myredis/7000
cp /home/xh/hadoop/redis-3.2.5/redis.conf /home/xh/hadoop/myredis/7001
cp /home/xh/hadoop/redis-3.2.5/redis.conf /home/xh/hadoop/myredis/7002
cp /home/xh/hadoop/redis-3.2.5/redis.conf /home/xh/hadoop/myredis/7003
cp /home/xh/hadoop/redis-3.2.5/redis.conf /home/xh/hadoop/myredis/7004
cp /home/xh/hadoop/redis-3.2.5/redis.conf /home/xh/hadoop/myredis/7005
#例如
vim myredis/7000/redis.conf
#修改如下
#端口号
port 7000
#后台启动
daemonize yes
#开启集群
cluster-enabled yes
#集群节点配置文件注意需要跟端口号一样
cluster-config-file nodes-7000.conf
#数据文件存放位置(注意此处的路径需要自己创建好)
dir /usr/local/redis/redis-cluster/7000/data/
#集群连接超时时间
cluster-node-timeout 5000
#进程pid的文件位置 注意需要跟端口号一样
pidfile /var/run/redis-7000.pid
#开启aof
appendonly yes
#aof文件路径
appendfilename “appendonly-7000.aof”
#rdb文件路径
dbfilename dump-7000.rdb
注意需要补充如下设置否则java连接报错
#保护模式设为no
protected-mode no
#绑定ip
bind 192.168.240.128
cd /usr/local/redis
#这里闲一个个启动麻烦的话可以自己配置一个配置文件启动
bin/redis-server myredis/7000/redis.conf
bin/redis-server myredis/7001/redis.conf
bin/redis-server myredis/7002/redis.conf
bin/redis-server myredis/7003/redis.conf
bin/redis-server myredis/7004/redis.conf
bin/redis-server myredis/7005/redis.conf
[roothadoop myredis]# ps -ef|grep redis
#显示有六个则是启动成功
root 63262 1 0 11:08 ? 00:00:00 redis-server 192.168.240.128:7000 [cluster]
root 63264 1 0 11:08 ? 00:00:00 redis-server 192.168.240.128:7001 [cluster]
root 63266 1 0 11:08 ? 00:00:00 redis-server 192.168.240.128:7002 [cluster]
root 63270 1 0 11:08 ? 00:00:00 redis-server 192.168.240.128:7003 [cluster]
root 63276 1 0 11:08 ? 00:00:00 redis-server 192.168.240.128:7004 [cluster]
root 63278 1 0 11:08 ? 00:00:00 redis-server 192.168.240.128:7005 [cluster]
root 63287 62960 0 11:08 pts/2 00:00:00 grep redis
到这里只是启动了六个单进程的redis开始创建集群先安装好ruby
yum install ruby rubygems -y
使用gem要先镜像一下
#这里需要镜像一下
gem sources --add --remove
#确保镜像成功
[roothadoop myredis]# gem sources -l
*** CURRENT SOURCES ***
然后执行连接ruby-redis
[roothadoop myredis]# gem install redis
上面步骤有可能报错查了资料发现是版本太低
解决办法是 先安装rvm再把ruby版本提升
ERROR: Error installing redis:
redis requires Ruby version > 2.3.0.
ruby版本提升若没有报错不需要进行此步骤
#安装curl
sudo yum install curl
#安装rvm
curl -L get.rvm.io | bash -s stable
(如出现错误参考:
source /usr/local/rvm/scripts/rvm
#查看rvm库中已知的ruby版本
rvm list known
#安装一个ruby版本
rvm install 2.4.0
#使用一个ruby版本
rvm use 2.4.0
#卸载一个已知版本
rvm remove 2.3.0
#查看版本
ruby --version
#再安装redis就可以了
gem install redis
创建集群
#进入src下面
cd /home/xh/hadoop/redis-3.2.5/src
# 创建集群[roothadoop src]# ./redis-trib.rb create --replicas 1 192.168.240.128:7000 192.168.240.128:7001 192.168.240.128:7002 192.168.240.128:7003 192.168.240.128:7004 192.168.240.128:7005>>> Creating cluster>>> Performing hash slots allocation on 6 nodes...Using 3 masters:192.168.240.128:7000192.168.240.128:7001192.168.240.128:7002Adding replica 192.168.240.128:7003 to 192.168.240.128:7000Adding replica 192.168.240.128:7004 to 192.168.240.128:7001Adding replica 192.168.240.128:7005 to 192.168.240.128:7002M: b553b5ed57b5d152dc1819c1818e61eba77b867f 192.168.240.128:7000 slots:0-5460 (5461 slots) masterM: d3bb8e329319f90d42f3a6163d0dcb2059cbeb47 192.168.240.128:7001 slots:5461-10922 (5462 slots) masterM: 40a44e92c9644e0366abb3aa2f18222b7d255b93 192.168.240.128:7002 slots:10923-16383 (5461 slots) masterS: 330be48eaa9bd676720738f0efe8007b2421944e 192.168.240.128:7003 replicates b553b5ed57b5d152dc1819c1818e61eba77b867fS: ba1195fe77df429079e34cb6c296878b1d0c7dcb 192.168.240.128:7004 replicates d3bb8e329319f90d42f3a6163d0dcb2059cbeb47S: 674176635075d82369ebba73060eadf8e4e1d701 192.168.240.128:7005 replicates 40a44e92c9644e0366abb3aa2f18222b7d255b93Can I set the above configuration? (type yes to accept): yes>>> Nodes configuration updated>>> Assign a different config epoch to each node>>> Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join.....>>> Performing Cluster Check (using node 192.168.240.128:7000)M: b553b5ed57b5d152dc1819c1818e61eba77b867f 192.168.240.128:7000 slots:0-5460 (5461 slots) master 1 additional replica(s)M: d3bb8e329319f90d42f3a6163d0dcb2059cbeb47 192.168.240.128:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s)S: ba1195fe77df429079e34cb6c296878b1d0c7dcb 192.168.240.128:7004 slots: (0 slots) slave replicates d3bb8e329319f90d42f3a6163d0dcb2059cbeb47M: 40a44e92c9644e0366abb3aa2f18222b7d255b93 192.168.240.128:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s)S: 330be48eaa9bd676720738f0efe8007b2421944e 192.168.240.128:7003 slots: (0 slots) slave replicates b553b5ed57b5d152dc1819c1818e61eba77b867fS: 674176635075d82369ebba73060eadf8e4e1d701 192.168.240.128:7005 slots: (0 slots) slave replicates 40a44e92c9644e0366abb3aa2f18222b7d255b93[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.到这里集群就安装好了测试一下
[xhhadoop myredis]$ redis-cli -c -h 192.168.240.128 -p 7000192.168.240.128:7000> set name frank-> Redirected to slot [5798] located at 192.168.240.128:7001OK192.168.240.128:7001> get namefrank在启动时会发现一个个启动太麻烦这里配置一下启动关闭文件
#进入安装redis的路径cd /home/xh/hadoop/myredis #编写关闭文件vi stop-all.sh #加入下面内容编辑完之后按esc键输入 :wq 退出保存#这里得找到自己配置启动得地方我得是在bin下面redis-cli -p 7000 -h 192.168.240.128 shutdownredis-cli -p 7001 -h 192.168.240.128 shutdownredis-cli -p 7002 -h 192.168.240.128 shutdownredis-cli -p 7003 -h 192.168.240.128 shutdownredis-cli -p 7004 -h 192.168.240.128 shutdownredis-cli -p 7005 -h 192.168.240.128 shutdown #赋值权限chmod ux stop-all.sh #编写启动文件vi start-all.sh #加入下面内容编辑完之后按esc键输入 :wq 退出保存#这里得找到自己配置启动得地方我得是在bin下面redis-server myredis/7000/redis.confredis-server myredis/7001/redis.confredis-server myredis/7002/redis.confredis-server myredis/7003/redis.confredis-server myredis/7004/redis.confredis-server myredis/7005/redis.conf #赋值权限chmod ux start-all.sh其中在java连接时出现问题通过参考如下文章解决
1.本地虚拟机搭建的Redis集群Jedis可以连接JedisCluster连接不上
2.Redis错误[ERR] Sorry, can‘t connect to node 192.168.10.3:6379
3.彻底解决[ERR] Node is not empty. Either the node already knows other nodes