欄目導(dǎo)航

新聞資訊

新聞資訊

建私有YUM倉庫

私有yum倉庫環(huán)境
系統(tǒng)版本：centos7.4 IP：192.168.1.47	#最好能上公網(wǎng)

私有yum倉庫服務(wù)端配置

第一：創(chuàng)建使用yum倉庫存放路徑
mkdir /home/mirros/siyou -p
第二：安裝必要軟件(yumdownloader這個命令在yum-utils工具里，用來下載軟件包，但不安裝)
yum install createrepo yum-utils -y
第三：上傳自己打包好的rpm包到/home/mirros/siyou
cd /home/mirros/siyou
lrzsz
#或則進入到該目錄，將軟件下載到此目錄，但是不安裝
yumdownloader openssl-*
第四：初始化repodata索引文件(每新放入一個rpm都需要更新索引，不讓識別不到新加的)
createrepo -pdo /home/mirrors/siyou/ /home/mirrors/siyou/
#下面是添加rpm包后的更新命令
createrepo --update /home/mirrors/siyou/
第五：提供http服務(wù)
cd /home/mirros/
python -m SimpleHTTPServer 80

客戶端配置

在使用私有源時，軟件包與base其他源一樣時，建議先禁用其他源。

cd /etc/yum.repos.d
vim Centos-siyou.repo
[siyou]
name=siyou
baseurl=http://192.168.1.47/siyou
enable=1 #1表示啟用，沒有此參數(shù)也表示啟用
gpgcheck=0
#重建緩存，并安裝軟件
yum clean all && yum makecache
yum install openssl-*

鏡像站環(huán)境

系統(tǒng)版本：centos7.4 IP：192.168.1.47	#要能上公網(wǎng)
centos及epel官方源，中國站（中科大鏡像站）http://mirrors.ustc.edu.cn
centos日常使用的軟件源介紹（base，extras，updates）
注意：中科大的rsync，每個ip同時只能同步兩條

內(nèi)網(wǎng)鏡像服務(wù)器配置

第一：安裝軟件
yum install rsync
第二：同步軟件源
#同步centos7的base源
mkdir /home/mirrors/centos/7 -p
rsync -av rsync://mirrors.ustc.edu.cn/centos/7/os /home/mirrors/centos/7
#同步centos7的extras源
	rsync -av rsync://mirrors.ustc.edu.cn/centos/7/extras /home/mirrors/centos/7
#同步centos7的updates源
rsync -av rsync://mirrors.ustc.edu.cn/centos/7/updates /home/mirrors/centos/7
+++++++++++++++++++++++++++++
或則省去前面三步，直接同步centos7的官方源
rsync -av rsync://mirrors.ustc.edu.cn/centos/7 /home/mirrors/centos
#簡單同步一個分支的所有信息
rsync -av rsync://mirrors.ustc.edu.cn/centos/7.4.1708 /home/mirrors/centos/
第三：添加epel源同步（epel源是擴展的，不是centos官方維護的，但是包含很多軟件）
mkdir /home/mirrors/epel/7 -p
rsync -av rsync://mirrors.ustc.edu.cn/epel/7/ /home/mirrors/epel/7
第四：要長期使用需要將上面同步命令，寫入定期執(zhí)行文件，定期同步更新
第五：啟動http
cd /home/mirrors
python -m SimpleHTTPServer 80

客戶端配置

第一：備份/etc/yum.repo/目錄下的官方源，并創(chuàng)建一個自己的源，如下
vim /etc/yum.repo/ccc.repo
[base]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
baseurl=http://192.168.1.47/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
#released updates 
[updates]
name=CentOS-$releasever - Updates
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
baseurl=http://192.168.1.47/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
baseurl=http://192.168.1.47/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
第二：客戶端怎么安裝epel源
#在安裝extras源后，默認是自帶并啟用的
yum install epel* -y
#編輯/etc/yum.repo/epel.repo文件
vim /etc/yum.repo/epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://192.168.1.47/epel/7/$basearch
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://192.168.1.47/epel/7/$basearch/debug
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://192.168.1.47/epel/7/SRPMS
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

客戶端使用

1 概述

本文檔描述CENTOS7.2操作系統(tǒng)部署CDH企業(yè)版的過程。Cloudera企業(yè)級數(shù)據(jù)中心的安裝主要分為4個步驟：

1.集群服務(wù)器配置，包括安裝操作系統(tǒng)、關(guān)閉防火墻、同步服務(wù)器時鐘等；

2.外部數(shù)據(jù)庫安裝

3.安裝Cloudera管理器；

4.安裝CDH集群；

5. 集群完整性檢查，包括HDFS文件系統(tǒng)、MapReduce、Hive等是否可以正常運行。

這篇文檔將著重介紹Cloudera管理器與CDH的安裝，并基于以下假設(shè)：

1. 操作系統(tǒng)版本：CENTOS7.2

2. MariaDB數(shù)據(jù)庫版本為10.2.1

3. CM版本：CDH 5.10.0

4. CDH版本：CDH 5.10.0

5.采用ec2-user對集群進行部署

6.您已經(jīng)下載CDH和CM的安裝包

2 前期準備

2.1 hostname及hosts配置

集群中各個節(jié)點之間能互相通信使用靜態(tài)IP地址。IP地址和主機名通過/etc/hosts配置，主機名/etc/hostname進行配置。

以cm節(jié)點（172.31.2.159）為例：

hostname配置

/etc/hostname文件如下：

ip-172-31-2-159

或者你可以通過命令修改立即生效

[ec2-user@ip-172-31-2-159 ~]$ sudo hostnamectl set-hostname ip-172-31-2-159

注意：這里修改hostname跟REDHAT6的區(qū)別

hosts配置

/etc/hosts文件如下:

172.31.2.159 ip-172-31-2-159

172.31.12.108 ip-172-31-12-108

172.31.5.236 ip-172-31-5-236

172.31.7.96 ip-172-31-7-96

以上兩步操作，在集群中其它節(jié)點做相應(yīng)配置。

2.2 禁用SELinux

在所有節(jié)點執(zhí)行sudo setenforce 0 命令，此處使用批處理shell執(zhí)行：

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo setenforce 0"

集群所有節(jié)點修改/etc/selinux/config文件如下：

SELINUX=disabled

SELINUXTYPE=targeted

2.3 關(guān)閉防火墻

集群所有節(jié)點執(zhí)行 sudo systemctl stop命令，此處通過shell批量執(zhí)行命令如下：

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo systemctl stop firewalld"

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo systemctl disable firewalld"

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo systemctl status firewalld"

2.4 集群時鐘同步

在CentOS7.2的操作系統(tǒng)上，已經(jīng)默認的安裝了chrony，配置chrony時鐘同步，將cm（172.31.2.159）服務(wù)作為本地chrony服務(wù)器，其它3臺服務(wù)器與其保持同步，配置片段：

172.31.2.159配置與自己同步

[ec2-user@ ip-172-31-2-159 ~]$ sudo vim /etc/chrony.conf

server ip-172-31-2-159 iburst

#keyfile=/etc/chrony.keys

集群其它節(jié)點：在注釋下增加如下配置

[ec2-user@ip-172-31-12-108 ~]$ sudo vim /etc/chrony.conf

# Use public servers from the pool.ntp.org project.

# Please consider joining the pool (http://www.pool.ntp.org/join.html).

server ip-172-31-2-159 iburst

#keyfile=/etc/chrony.keys

重啟所有機器的chrony服務(wù)

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "sudo systemctl restart chronyd"

驗證始終同步，在所有節(jié)點執(zhí)行chronycsources命令，如下使用腳本批量執(zhí)行

[ec2-user@ip-172-31-2-159 ~]$ sh ssh_do_all.sh node.list "chronyc sources"

2.5 配置操作系統(tǒng)repo

掛載操作系統(tǒng)iso文件

[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /media/DVD1

[ec2-user@ip-172-31-2-159 ~]$ sudo mount -o loop

CentOS-7-x86_64-DVD-1611.iso /media/DVD1/

配置操作系統(tǒng)repo

[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/local_os.repo

[local_iso]

name=CentOS-$releasever - Media

baseurl=file:///media/DVD1

gpgcheck=0

enabled=1

[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist

2.6 安裝http服務(wù)

安裝httpd服務(wù)

[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install httpd

啟動或停止httpd服務(wù)

[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl start httpd

[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl stop httpd

安裝完httpd后，重新制作操作系統(tǒng)repo，換成http的方式方便其它服務(wù)器也可以訪問

[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /var/www/html/iso

[ec2-user@ip-172-31-2-159 ~]$ sudo scp -r /media/DVD1/* /var/www/html/iso/

[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/os.repo

[osrepo]

name=os_repo

baseurl=http://172.31.2.159/iso/

enabled=true

gpgcheck=false

[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist

2.7 安裝MariaDB

由于centos7默認使用的是5.5.52版本的MariaDB，此處使用的10.2.1版本（http://yum.mariadb.org/10.2.1/centos7-amd64/rpms/），在官網(wǎng)下載rpm安裝包：

MariaDB-10.2.1-centos7-x86_64-client.rpm

MariaDB-10.2.1-centos7-x86_64-common.rpm

MariaDB-10.2.1-centos7-x86_64-compat.rpm

MariaDB-10.2.1-centos7-x86_64-server.rpm

將包下載到本地，放在同一目錄，執(zhí)行createrepo命令生成rpm元數(shù)據(jù)。

此處使用apache2，將上述mariadb10.2.1目錄移動到/var/www/html目錄下, 使得用戶可以通過HTTP訪問這些rpm包。

[ec2-user@ip-172-31-2-159 ~]$ sudo mv mariadb10.2.1 /var/www/html/

安裝MariaDB依賴

[ec2-user@ip-172-31-2-159 ~]$ yum install libaio perl perl-DBI perl-Module-Pluggable perl-Pod-Escapes perl-Pod-Simple perl-libs perl-version

制作本地repo

[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/mariadb.repo

[mariadb]

name=MariaDB

baseurl=http://172.31.2.159/ mariadb10.2.1

enable=true

gpgcheck=false

[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist

安裝MariaDB

[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install MariaDB-server MariaDB-client

啟動并配置MariaDB

[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl start mariadb

[ec2-user@ip-172-31-2-159 ~]$ sudo /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB

SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current

password for the root user. If you've just installed MariaDB, and

you haven't set the root password yet, the password will be blank,

so you should just press enter here.

Enter current password for root (enter for none):

OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB

root user without the proper authorisation.

Set root password? [Y/n] Y

New password:

Re-enter new password:

Password updated successfully!

Reloading privilege tables..

... Success!

By default, a MariaDB installation has an anonymous user, allowing anyone

to log into MariaDB without having to have a user account created for

them. This is intended only for testing, and to make the installation

go a bit smoother. You should remove them before moving into a

production environment.

Remove anonymous users? [Y/n] Y

... Success!

Normally, root should only be allowed to connect from 'localhost'. This

ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n

... skipping.

By default, MariaDB comes with a database named 'test' that anyone can

access. This is also intended only for testing, and should be removed

before moving into a production environment.

Remove test database and access to it? [Y/n] Y

- Dropping test database...

... Success!

- Removing privileges on test database...

... Success!

Reloading the privilege tables will ensure that all changes made so far

will take effect immediately.

Reload privilege tables now? [Y/n] Y

... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDB

installation should now be secure.

Thanks for using MariaDB!

建立CM和Hive需要的表

[ec2-user@ip-172-31-2-159 ~]$ mysql -uroot -p

Enter password:

Welcome to the MariaDB monitor. Commands end with ; or \g.

Your MariaDB connection id is 9

Server version: 10.2.1-MariaDB MariaDB Server

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

create database metastore default character set utf8;

CREATE USER 'hive'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';

FLUSH PRIVILEGES;

create database cm default character set utf8;

CREATE USER 'cm'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';

FLUSH PRIVILEGES;

create database am default character set utf8;

CREATE USER 'am'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON am. * TO 'am'@'%';

FLUSH PRIVILEGES;

create database rm default character set utf8;

CREATE USER 'rm'@'%' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';

FLUSH PRIVILEGES;

安裝jdbc驅(qū)動

[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir -p /usr/share/java/

[ec2-user@ip-172-31-2-159 ~]$ sudo mv mysql-connector-java-5.1.37.jar /usr/share/java/

[ec2-user@ip-172-31-2-159 java]$ cd /usr/share/java

[ec2-user@ip-172-31-2-159 java]$ sudo ln -s mysql-connector-java-5.1.37.jar mysql-connector-java.jar

[ec2-user@ip-172-31-2-159 java]$ ll

total 964

-rw-r--r--. 1 root root 985600 Oct 6 2015 mysql-connector-java-5.1.37.jar

lrwxrwxrwx. 1 root root 31 Mar 29 14:37 mysql-connector-java.jar -> mysql-connector-java-5.1.37.jar

3 Cloudera Manager安裝

3.1 配置本地repo源

將Cloudera Manager安裝需要的7個rpm包下載到本地，放在同一目錄，執(zhí)行createrepo命令生成rpm元數(shù)據(jù)。

[ec2-user@ip-172-31-2-159 cm]$ ls

cloudera-manager-agent-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

cloudera-manager-daemons-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

cloudera-manager-server-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

cloudera-manager-server-db-2-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

enterprise-debuginfo-5.10.0-1.cm5100.p0.85.el7.x86_64.rpm

jdk-6u31-linux-amd64.rpm

oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

[ec2-user@ip-172-31-2-159 cm]$ sudo createrepo .

Spawning worker 0 with 1 pkgs

Spawning worker 1 with 1 pkgs

Spawning worker 2 with 1 pkgs

Spawning worker 3 with 1 pkgs

Spawning worker 4 with 1 pkgs

Spawning worker 5 with 1 pkgs

Spawning worker 6 with 1 pkgs

Spawning worker 7 with 0 pkgs

Workers Finished

Saving Primary metadata

Saving file lists metadata

Saving other metadata

Generating sqlite DBs

Sqlite DBs complete

配置Web服務(wù)器

此處使用apache2，將上述cdh5.10.0/cm5.10.0目錄移動到/var/www/html目錄下, 使得用戶可以通過HTTP訪問這些rpm包。

[ec2-user@ip-172-31-2-159 ~]$ sudo mv cdh5.10.0/ cm5.10.0/ /var/www/html/

制作ClouderaManager的repo源

[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/cm.repo

[cmrepo]

name=cm_repo

baseurl=http://172.31.2.159/cm5.10.0.0

enable=true

gpgcheck=false

[ec2-user@ip-172-31-2-159 yum.repos.d]$ sudo yum repolist

驗證安裝JDK

[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install oracle-j2sdk1.7-1.7.0+update67-1

3.2 安裝Cloudera Manager Server

通過yum安裝ClouderaManager Server

[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install cloudera-manager-server

初始化數(shù)據(jù)庫

[ec2-user@ip-172-31-2-159 ~]$ sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql cm cm password

JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

Verifying that we can write to /etc/cloudera-scm-server

Creating SCM configuration file in /etc/cloudera-scm-server

Executing: /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/

oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.

[ main] DbCommandExecutor INFO Successfully connected to database.

All done, your SCM database is configured correctly!

啟動ClouderaManager Server

[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl start cloudera-scm-server

檢查端口是否監(jiān)聽

[ec2-user@ip-172-31-2-159 ~]$ sudo netstat -lnpt | grep 7180

tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 6890/java

通過http://172.31.2.159:7180/cmf/login訪問CM

4 CDH安裝

4.1 CDH集群安裝向?qū)?/strong>

admin/admin登錄到CM
同意license協(xié)議，點擊繼續(xù)

3.選擇60試用，點擊繼續(xù)

4.點擊“繼續(xù)”

5.輸入主機ip或者名稱，點擊搜索找到主機后點擊繼續(xù)

6.點擊“繼續(xù)”

7.使用parcel選擇，點擊“更多選項”,點擊“-”刪除其它所有地址，輸入

http://172.31.2.159/cm5.10.0/點擊“保存更改”

8.選擇自定義存儲庫，輸入cm的http地址

9.點擊“繼續(xù)”，進入下一步安裝jdk

10.點擊“繼續(xù)”，進入下一步，默認多用戶模式

11.點擊“繼續(xù)”，進入下一步配置ssh賬號密碼

12.點擊“繼續(xù)”，進入下一步，安裝Cloudera Manager相關(guān)到各個節(jié)點

13.點擊“繼續(xù)”，進入下一步安裝cdh到各個節(jié)點

14.點擊“繼續(xù)”，進入下一步主機檢查，確保所有檢查項均通過

點擊完成進入服務(wù)安裝向?qū)А?/p>

4.2 集群設(shè)置安裝向?qū)?/strong>

選擇需要安裝的服務(wù)

2.點擊“繼續(xù)”，進入集群角色分配

3.點擊“繼續(xù)”，進入下一步，測試數(shù)據(jù)庫連接

4.測試成功，點擊“繼續(xù)”，進入目錄設(shè)置，此處使用默認默認目錄，根據(jù)實際情況進行目錄修改

5.點擊“繼續(xù)”，進入各個服務(wù)啟動

6.安裝成功

7.安裝成功后進入home管理界面

“醉酒鞭名馬，少年多浮夸！嶺南浣溪沙，嘔吐酒肆下！摯友不肯放，數(shù)據(jù)玩的花！”

操屁眼的视频在线免费看,日本在线综合一区二区,久久在线观看免费视频,欧美日韩精品久久综

1 概述

2 前期準備

3 Cloudera Manager安裝

4 CDH安裝