Installing Stinger Technical Preview in HDP 2.0 Sandbox

Yesterday I tried to install Stinger on Hortonworks HDP 2.0 Sandbox. Below are the steps I followed. I used the Sandbox 2 for Hyper-V.

Installing Stinger phase 3 preview

Import the sandbox 2 VM and make sure that it can access to the internet.

Start the VM and log into it using Alt+F5 keys. Download Stinger Quickstart Bundle using wget. Remember the url is case sensitive.

wget https://github.com/cartershanklin/StingerQuickstart/blob/master/Stinger-Preview-Quickstart.tgz?raw=true

untar the downloaded file

tar -xzf Stinger-Preview-Quickstart.tgz

Change the directory name to StingerPreview. We are going to use that directory for multiple times.

mv Stinger-Preview-Quickstart StingerPreview

Installing and Configuring Tez

Download the Stinger Phase 3 Preview Tez tarball:

wget http://public-repo-1.hortonworks.com/HDP-2.1.0.0/repos/centos6/tars/tez-0.2.0.2.1.0.0-92.tar.gz

Extract the Stinger Phase 3 Preview Tez tarball:

tar -C /opt -xzf tez-0.2.0.2.1.0.0-92.tar.gz

Create a directory on HDFS for Tez:

hadoop fs -mkdir -p /apps/tez
hadoop fs -chmod 755 /apps/tez

Deploy Tez files to HDFS:

hadoop fs -copyFromLocal /opt/tez-0.2.0.2.1.0.0-92/* /apps/tez/

Deploy a sample Tez configuration from the Quickstart Bundle:

cp StingerPreview/configs/tez-site.xml.sandbox /etc/hadoop/conf/tez-site.xml

Installing and Configuring Hive 0.13

We have to use Hive 0.13 preview to the Hadoop cluster because this version of Hive is designed to work with Tez 0.2.

Download the Stinger Phase 3 Preview Hive tarball:

wget http://public-repo-1.hortonworks.com/HDP-2.1.0.0/repos/centos6/tars/apache-hive-0.13.0.2.1.0.0-92-bin.tar.gz

Extract the Stinger Phase 3 Preview Hive tarball:

tar -C /opt -xzf apache-hive-0.13.0.2.1.0.0-92-bin.tar.gz

Create a directory on HDFS for Hive with appropriate permissions.

hadoop fs -mkdir -p /user/hive
hadoop fs -chmod 755 /user/hive

Deploy Hive files to HDFS:

hadoop fs -put /opt/apache-hive-0.13.0.2.1.0.0-92-bin/lib/hive-exec-*.jar /user/hive/hive-exec-0.13.0-SNAPSHOT.jar

Configure the environment:

You must run below commands before starting hive.

export HIVE_HOME=/opt/apache-hive-0.13.0.2.1.0.0-92-bin
export HIVE_CONF_DIR=$HIVE_HOME/conf
export PATH=$HIVE_HOME/bin:$PATH
export HADOOP_CLASSPATH=/opt/tez-0.2.0.2.1.0.0-92/*:/opt/tez-0.2.0.2.1.0.0-92/lib/*
export HADOOP_USER_CLASSPATH_FIRST=true

Deploy Hive configuration:

To use Hive on Tez, certain configuration options must be set in hive-site.xml. This file must also contain information including login and password to your metastore. Copy an existing hive-site.xml.

cp /etc/hive/conf.erver/hive-site.xml /opt/apache-hive-0.13.0.2.1.0.0-92-bin/conf/hive-site.xml

Running Hive and Ensuring a Correct Install

If you want to run Hive with Stinger settings, you will need to start Hive with an additional parameter that contains in the stinger.settings. The file stinger.settings is provided in the Stinger Quickstart Bundle’s configs folder.

hive -i StingerPreview/configs/stinger.settings -hiveconf hive.optimize.tez=true

stinger settings

To ensure the proper configurations have been made, in the hive command line, Issue the command set hive.vectorized.execution.enabled;. You should see the output as true.

set hive vectorized execution enabled

Now you would be able to query Hive which will use Stinger.

Reference: http://public-repo-1.hortonworks.com/HDP-LABS/Projects/Stinger/StingerTechnicalPreviewInstall.pdf

Leave a Reply

Close Menu