Hadoop集群(CDH4)实践之 (3) Hive搭建

目录结构 Hadoop集群(CDH4)实践之 (0) 前言 Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建 Hadoop集群(CDH4)实践之 (2) HBaseZookeeper搭建 Hadoop集群(CDH4)实践之 (3) Hive搭建 Hadoop集群(CHD4)实践之 (4) Oozie搭建 Hadoop集群(CHD4)实践之 (5) Sqoop安

目录结构
Hadoop集群(CDH4)实践之 (0) 前言
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装

本文内容
Hadoop集群(CDH4)实践之 (3) Hive搭建

参考资料
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

环境准备
OS: CentOS 6.4 x86_64
Servers:
hadoop-master: 172.17.20.230 内存10G
– namenode
– hbase-master

hadoop-secondary: 172.17.20.234 内存10G
– secondarybackupnamenode,jobtracker
– hive-server,hive-metastore

hadoop-node-1: 172.17.20.231 内存10G
– datanode,tasktracker
– hbase-regionserver,zookeeper-server

hadoop-node-2: 172.17.20.232 内存10G
– datanode,tasktracker
– hbase-regionserver,zookeeper-server

hadoop-node-3: 172.17.20.233 内存10G
– datanode,tasktracker
– hbase-regionserver,zookeeper-server

对以上角色做一些简单的介绍：
namenode – 整个HDFS的命名空间管理服务
secondarynamenode – 可以看做是namenode的冗余服务
jobtracker – 并行计算的job管理服务
datanode – HDFS的节点服务
tasktracker – 并行计算的job执行服务
hbase-master – Hbase的管理服务
hbase-regionServer – 对Client端插入，删除，查询数据等提供服务
zookeeper-server – Zookeeper协作与配置管理服务
hive-server – Hive的管理服务
hive-metastore – Hive的元存储，用于对元数据进行类型检查与语法分析

本文定义的规范，避免在配置多台服务器上产生理解上的混乱：
以下操作都只需要在 Hive 所在主机，即 hadoop-secondary 上执行。

1. 安装前的准备
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建

2. 安装Hive
$ sudo yum install hive hive-metastore hive-server
$ sudo yum install hive-jdbc hive-hbase

3. 安装MySQL JDBC Connector
$ sudo yum install mysql-connector-java
$ sudo ln -s /usr/share/java/mysql-connector-java.jar /usr/lib/hive/lib/mysql-connector-java.jar

4. 安装MySQL
$ sudo yum install mysql-server
$ sudo /etc/init.d/mysqld start

$ sudo /usr/bin/mysql_secure_installation

 [...]Enter current password for root (enter for none):OK, successfully used password, moving on...[...]Set root password? [Y/n] yNew password: hiveserverRe-enter new password: hiverserverRemove anonymous users? [Y/n] Y[...]Disallow root login remotely? [Y/n] N[...]Remove test database and access to it [Y/n] Y[...]Reload privilege tables now? [Y/n] YAll done!

5. 创建数据库并授权
$ mysql -u root -phiveserver

 mysql> CREATE DATABASE metastore;mysql> USE metastore;mysql> SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.10.0.mysql.sql;mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hiveserver';mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'%';mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'%';mysql> CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hiveserver';mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'localhost';mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'localhost';mysql> CREATE USER 'hive'@'127.0.0.1' IDENTIFIED BY 'hiveserver';mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'127.0.0.1';mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'127.0.0.1';

6. 配置hive-site.xml
$ sudo vim /etc/hive/conf/hive-site.xml

 <?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="http://heylinux.com/archives/configuration.xsl"?>  javax.jdo.option.ConnectionURL  jdbc:mysql://hadoop-secondary/metastore  the URL of the MySQL<p>本文来源gao!daima.com搞$代!码#网#</p> database  javax.jdo.option.ConnectionDriverName  com.mysql.jdbc.Driver  javax.jdo.option.ConnectionUserName  hive  javax.jdo.option.ConnectionPassword  hiveserver  datanucleus.autoCreateSchema  false  datanucleus.fixedDatastore  true  datanucleus.autoStartMechanism   SchemaTable  hive.metastore.uris  thrift://hadoop-secondary:9083  IP address (or fully-qualified domain name) and port of the metastore host  hive.aux.jars.path  file:////usr/lib/hive/lib/hbase.jar,file:///usr/lib/hive/lib/zookeeper.jar,file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.5.0.jar,file:///usr/lib/hive/lib/guava-11.0.2.jar  hbase.zookeeper.quorum  hadoop-node-1,hadoop-node-2,hadoop-node-3

7. 启动Hive
$ /etc/init.d/hive-metastore start
$ /etc/init.d/hive-server start

8. 创建Hive所需的HDFS目录
$ sudo -u hdfs hadoop fs -mkdir /user/hive
$ sudo -u hdfs hadoop fs -mkdir /user/hive/warehouse
$ sudo -u hdfs hadoop fs -ls -R /user
$ sudo -u hdfs hadoop fs -chown -R hive /user/hive
$ sudo -u hdfs hadoop fs -chmod -R 1777 /user/hive/warehouse

$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hadoop-mapred
$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hive-hive
$ sudo chown -R hive:hive /var/lib/hive/.hivehistory

9. 至此，Hive的搭建就已经完成。

喜欢 (0)赏

[搞代码]

分享 (0)