Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Hive Installation
09-19-2016, 11:29 AM (This post was last modified: 07-05-2018 05:19 PM by zheny.)
Post: #1
Hive Installation
Follow introduce how to install Hive by yum on Centos 7 with CDH yum repo.

1. Install package

1.1 Install on NameNode
Code:
yum install hive hive-metastore hive-server2 hive-jdbc hive-hbase  -y

1.2 Install on DataNode
Code:
yum install hive hive-server2 hive-jdbc hive-hbase  -y

2. Install postgresql or Mysql as base database in Hadoop Namenode.
Follow introduce postgresql

2.1 Install postgresql
Code:
yum install postgresql-server -y

2.2 Initial postgresql
Code:
service postgresql initdb

2.3 Modify the configuration file
Modify /var/lib/pgsql/data/postgresql.conf. change filed 'listen_addresses' to "listen_addresses='*' " and 'standard_conforming_strings' to 'off'.

Modify /var/lib/pgsql/data/pg_hba.conf and add
host all all 0.0.0.0/0 trust

if you find can not login the database after set the password with user, change the content in above file about 'local' line and modify the last field to md5.

2.4 start the service and configure to start with boot
Code:
service postgresql start
Code:
chkconfig postgresql on

2.5 Download the jdbc driver
https://jdbc.postgresql.org/download.html , please be aware of the java version you installed and choose the best version for you. if download a incorrect version ,then hive can not work well.

rename the driver file to postgresql-jdbc.jar and place to /usr/lib/hive/lib/
this step should be done on each node

2.6 Create database and user
Code:
su postgres
type psql and type
Code:
CREATE USER hiveuser WITH PASSWORD 'redhat';
hiveuser is the username and redhat is the password. change according your requirement.
Code:
CREATE DATABASE metastore owner=hiveuser ;
Code:
GRANT ALL privileges ON DATABASE metastore TO hiveuser ;

Code:
\q
exit the pstgresql

2.7 import the database scheme.
Code:
rpm -qa | grep hive
to fine the hive version you installed.
such as hive-1.1.0
Code:
cd /usr/lib/hive/scripts/metastore/upgrade/postgres
there will be a file such as hive-schema-1.1.0.postgres.sql. then import:
Code:
psql -U hiveuser -d metastore -f hive-schema-1.1.0.postgres.sql
hiveuser is the user created before, type the password then the import progress will start.


3. Configure Hive
Modify the /etc/hive/conf/hive-site.xml , if not exist then create it.
Follow is a template instanceļ¼š
Code:
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:postgresql://linux01/metastore</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>org.postgresql.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hiveuser</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>redhat</value>
</property>

<property>
  <name>datanucleus.autoCreateSchema</name>
  <value>true</value>
</property>

<property>
  <name>datanucleus.autoCreateTables</name>
  <value>true</value>
</property>

<property>
<name>hive.metastore.schema.verification</name>
<value>true</value>
</property>

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>


according the real env to change 'linux01'(hostname), 'hiveuser'(database username), 'redhat'(password).

and add
Code:
<property>
  <name>hadoop.proxyuser.hive.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.hive.groups</name>
  <value>*</value>
</property>
to /etc/hadoop/conf/core-site.xml(if the hadoop conf dir is /etc/hadoop/conf)

4. start the service
Code:
service hive-metastore start;service hive-server2 start


5. configure the user .bashrc file
Code:
export JAVA_HOME=/usr/java/default/
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:/usr/lib/hbase:/usr/lib/hive/lib/:/usr/lib/hadoop/lib/
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export HADOOP_CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:/usr/lib/hbase:/usr/lib/hive/lib/:/usr/lib/hadoop/lib/

#Hadoop variables
export HADOOP_HOME=/usr/lib/hadoop
export HIVE_HOME=/usr/lib/hive
export HBASE_HOME=/usr/lib/hbase
export HADOOP_COMMON_HOME=/usr/lib/hadoop
#export HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
export HADOOP_HDFS_HOME=/usr/lib/hadoop-hdfs
#export HADOOP_MAPRED_HOME=/usr/lib/hadoop/hadoop-mapreduce
#export HADOOP_HDFS_HOME=$HADOOP_COMMON_HOME
export YARN_HOME=$HADOOP_COMMON_HOME
export PATH=$PATH:$HADOOP_COMMON_HOME/bin
export PATH=$PATH:$HADOOP_COMMON_HOME/sbin
#export HADOOP_LIB=/usr/lib/hadoop/lib/
export HADOOP_LIB=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:/usr/lib/hbase:/usr/lib/hive/lib/:/usr/lib/hadoop/lib/
export PATH=$PATH:${HIVE_HOME}/bin
export HIVE_LIB=$HIVE_HOME/lib
export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_YARN_HOME=/usr/lib/hadoop-yarn
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop


6 use hive
just type
Code:
hive -e 'create table t(id int);'
Code:
hive -e 'select * from t;'
Code:
hive -e 'select id from t;'
to see if hive works well or not.

if type hive and report "unable to instantiate org.apache.hadoop.hive.ql.metadata.session hive metastore" , please verify if the jdbc driver is match with your java version.
Find all posts by this user
Quote this message in a reply
07-05-2018, 12:27 PM
Post: #2
RE: Hive Installation
about step 1, I suggest to change it as:
1. Install package
1.1 Install on NameNode
Code:
yum install hive hive-metastore hive-server2 hive-jdbc hive-hbase  -y
1.2 Install on DataNode
Code:
yum install hive hive-server2 hive-jdbc hive-hbase  -y
RR xwcwt
Find all posts by this user
Quote this message in a reply
07-05-2018, 01:43 PM
Post: #3
RE: Hive Installation
(07-05-2018 12:27 PM)zheny Wrote:  about step 1, I suggest to change it as:
1. Install package
1.1 Install on NameNode
Code:
yum install hive hive-metastore hive-server2 hive-jdbc hive-hbase  -y
1.2 Install on DataNode
Code:
yum install hive hive-server2 hive-jdbc hive-hbase  -y
RR xwcwt

Good to me.
Find all posts by this user
Quote this message in a reply
07-05-2018, 02:21 PM
Post: #4
RE: Hive Installation
about 2.3
Quote:2.3 Modify the configuration file
Modify /var/lib/pgsql/data/postgresql.conf. change filed 'listen_addresses' to '*' and 'standard_conforming_strings' to 'off'.

the 'listen_addresses' should be changed to " '*' "
to avoid misunderstanding, I suggest to modify it as
Quote:2.3 Modify the configuration file
Modify /var/lib/pgsql/data/postgresql.conf. change filed 'listen_addresses' to "listen_addresses='*' " and 'standard_conforming_strings' to 'off'.
RR xwcwt
Find all posts by this user
Quote this message in a reply
07-05-2018, 02:40 PM
Post: #5
RE: Hive Installation
(07-05-2018 02:21 PM)zheny Wrote:  about 2.3
Quote:2.3 Modify the configuration file
Modify /var/lib/pgsql/data/postgresql.conf. change filed 'listen_addresses' to '*' and 'standard_conforming_strings' to 'off'.

the 'listen_addresses' should be changed to " '*' "
to avoid misunderstanding, I suggest to modify it as
Quote:2.3 Modify the configuration file
Modify /var/lib/pgsql/data/postgresql.conf. change filed 'listen_addresses' to "listen_addresses='*' " and 'standard_conforming_strings' to 'off'.
RR xwcwt

Good to me.
Find all posts by this user
Quote this message in a reply
07-05-2018, 03:44 PM (This post was last modified: 07-05-2018 04:12 PM by zheny.)
Post: #6
RE: Hive Installation
in step 2.3
Quote:if you find can not login the database after set the password with user, change the content in above file about 'local' line and modify the last field to md5.
after changed 'peer' to 'md5', you need to type passwd to login or change user
however, there is no default passwd, so we need set a passwd manually by, for example,
Code:
alter user postgres password 'new passwd';
in my installation, I set the passwd as our default passwd for root

RR xwcwt
Find all posts by this user
Quote this message in a reply
07-05-2018, 04:12 PM
Post: #7
RE: Hive Installation
in 2.6,
I find there must be a ' ' and ';' after the commend otherwise the commend cannot work

RR xwcwt
Find all posts by this user
Quote this message in a reply
07-05-2018, 05:10 PM
Post: #8
RE: Hive Installation
I realize step 2.5 should be done on each node, not only on the namenode

RR xwcwt
Find all posts by this user
Quote this message in a reply
07-05-2018, 05:12 PM
Post: #9
RE: Hive Installation
(07-05-2018 04:12 PM)zheny Wrote:  in 2.6,
I find there must be a ' ' and ';' after the commend otherwise the commend cannot work

RR xwcwt

I feel you are saying command.

It is good to me add the token as you said above.
Find all posts by this user
Quote this message in a reply
07-05-2018, 05:13 PM
Post: #10
RE: Hive Installation
(07-05-2018 05:10 PM)zheny Wrote:  I realize step 2.5 should be done on each node, not only on the namenode

RR xwcwt

Fine to me if you think it is needed.
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump: