實作HBase 操作介面

[尚須改進]
- 使用 Pool connection
- 加入 Spark 計算


功能:
0. 操作選項清單 (HBaseGet.java)
[已有功能]
- 可選擇計算或查詢
- 指令核對，錯誤指令將重新詢問
- 可選擇是否結束

[期望改進]
- 導入圖形介面(選單、參數方塊...等)

1. 計算         (CountJob.java)
[基本功能]
- 自行選擇Table, Column, 及輸出位置, row range尚須手動修改code
- 自動檢驗輸入的Table name, Column Family Name, Column Name

[期望改進]

- 可計算:
- Average
- 輸出結果 文字檔 至 HDFS

[期望改進]
1. conf.set("mapred.jar", ...) 因為要local端位址，還要想辦法修改(是否可自動?)
2. 把 平均或標準差的計算再拉到另外class

2. 查詢         (ScanHTable.java)
[已有功能]
      - 可設定簡單filter

      [期望改進]
      - filter 值得輸入方式

3. Create Table
     [已有功能]
     - 可手動輸入 tablename, cf

    [期望改進]
    - 自動創建大量cf之表格

4. Delete Table
     [已有功能]
     - 可手動輸入 tablename, cf

    [期望改進]
    - 是否要獨立disable?

5. List Table
     [已有功能]
     - 列出現存table name


* 6. Import Data to Table
*- Online
- Offline (HBaseimporttsv_v4.jar)

Java compare part of String method

Java 中判斷 String 中是否包含 subString，意即比對部分字串
比如要判斷Sting "ERROR543"中，是否包含 "ERROR"

可用

public class test {
public static void main(String[] args){

String a = "ERROR543";
if(a.indexOf("ERRO") != -1)
System.out.println("yes");
else
System.out.println("NO");
}
}

Eclipse with Hadoop plugin

[Reference]
https://github.com/winghc/hadoop2x-eclipse-plugin  (package download)
http://blog.jourk.com/compile-hadoop-eclipse-plugin-2-4-1-plugin.html  (method)
http://blog.csdn.net/yueritian/article/details/23868175  (Debug)


Software:
eclpise:  eclipse-java-luna-R-linux-gtk-x86_64.tar.gz
Hadoop: 2.4.1
#Because Hadoop2.4.1 doesn't provide the plugin for eclipse, so we need to create one
Plugin create:
Download: https://github.com/winghc/hadoop2x-eclipse-plugin
1.Modify: 
(this tool now(2014/09/15)is set for hadoop2.2.0 version, so it need to be modify for 2.4.1v.)
[Change 1]
  Find ~/hadoop2x-eclipse-plungin-master/src/contrib/eclipse-plugin/build.xml
  Add commons-collections-3.2.1.jar setting :
...
<copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<!--     Add this Line         -->
<copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<!--                           -->
<copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/>
...

    Change
lib/commons-lang-2.5.jar
 to
        lib/commons-lang-2.6.jar

 Add
lib/commons-collections-${commons-collections.version}.jar, 
 
[Change 2]
Find ~/hadoop2x-eclipse-plungin-master/ivy/libraries.properties
Add
commons-collections.version=3.2.1

2.Jar file
$cd src/contrib/eclipse-plugin
$ant jar -Dversion=2.4.1 -Declipse.home=/home/hdp2/eclipse -Dhadoop.home=/home/hdp2/hadoop-2.4.1

Then the jar file will appear in the ~/hadoop2x-eclipse-plungin-master/build/contrib/eclipse-plugin

# -Dversion : the installed Hadoop version
  -Declipse.home : the dir of ECLIPSE_HOME
  -Dhadoop.home : the dir of HADOOP_HOME

3. Move the plugin to eclipse/plugins
$cp build/contrib/eclipse-plugin/hadoop-eclipse-plugin.2.4.1.jar /home/hdp2/eclipse/plugins

4. Start eclipse with debug parameter:
$ eclipse/eclipse -clean -consolelog -debug

[備註] 此指令很重要，若eclipse開啟後點選 map/reduce location 沒反應，可以從terminal去看error info!

5. Eclipse開啟後
- Windows > Open Perspective > Other > Map/Reduce
- Windows > Show View > Other > MapReduce Tools > Map/Reduce Locations
- 點選右下角藍色大象 > 設定Hadoop server Location
[備註] 若點選大象時，沒跳出設定畫面，請查詢terminal上面錯誤訊息為何

- Setting:
Location name : master
MapReduce Master >>  Host: 192.168.0.7  Port: 8032
DFS Master     >>  Host:(use M/R Master host) Port:9000
User name : hduser (hadoop user name)

Eclipse Submit Job from Hadoop Client to remote Hadoop Master

[Software]
    Hadoop2.4.1
    Eclipse IDE for Java Developers Luna Release (4.4.0)
[Problem]
If you want to submit job from local side to remote Hadoop server through runngin java application on eclipse directly without send jar file to Hadoop server(ex. eclipce on 192.168.0.51 and Hadoop master on 192.168.0.7) 

1. Point to the jar file which will be create by the following step
add following code to your which set conf. to point to the jar file which will be create by the following step

conf.set("mapred.jar", "/home/hdp2/workspace/HBaseGet/HbaseGet_v3.jar");

[Reference]
http://stackoverflow.com/questions/21793565/class-not-found-exception-in-eclipse-wordcount-program

[NOTICE 1]
The "/home/hdp2/workspace/HBaseGet/HbaseGet_v3.jar" is the jar file location at local side

[NOTICE 2]
If you don't do this step, Eclipse may occur following error:
[Error]
... Class org.apache....  Map not found ...


2. Set yarn master locaiton
 add following to " yarn-site.xml "

<property>  
<name>yarn.resourcemanager.address</name>  
<value>master:8032</value>  
</property>  
<property>  
<name>yarn.resourcemanager.scheduler.address</name>  
<value>master:8030</value>  
</property>  
<property>  
<name>yarn.resourcemanager.resource-tracker.address</name>  
<value>master:8031</value>  
</property>  

3. Set Configuration for Client Side
 Select "Run Configuration" > "Classpath" > "Advanced" > "Add External Folder"  > Select the Hadoop and HBase conf folder(ex. $HADOOP_HOME/etc/hadoop and $HBASE_HOME/conf)

[IMPORTANT !]
Conf settings in Both Hadoop and HBase conf folder MUST be consistent  with the conf in the Hadoop and HBase master!!
# Simple method is copy the conf folder from  master to local

[Notice]
Because the Eclipse is running locally without the correct server configurations,
so if you don't do this step, it will occur following errors:
[Error 1]
Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[Error 2]
"eclipse"  ... Retrying connect to server: 0.0.0.0/0.0.0.0:8032 ....

4. Export Jar File
"File" > "Export..." > "Java" > "JAR file" > select resource to export > "Finish"

5. Run Application
"Run Configuration" > Set "Arguments" > "Run"


(6. Staging problem ( Multiple users in Hadoop )
If you are the Hadoop client not master(ex. client_username="hdp2", Hadoop_username="hduser"), it may occur error when execute app like:

2012-10-09 10:06:31,233 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:root cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=READ_EXECUTE, inode=”system”:mapred:supergroup:rwx——

[NEED TO DO]
http://amalgjose.wordpress.com/2013/02/09/setting-up-multiple-users-in-hadoop-clusters/
1. Create a new user(hdp2) at Hadoop master server
$sudo adduser hdp2

2. Create a user for client on HDFS
$hdfs dfs -mkdir /user/hdp2

3. Using new username(hdp2) to login Hadoop Master server, and execute any example jar(such as WordCount)
It will create the staging folder of new user in HDFS/tmp/
(Don't know why......)
4. Change priority
$hdfs dfs –chown –R hdp2:hadoop /tmp/hadoop-yarn/staging/hdp2/
$hdfs dfs -chmod 777 /tmp/

Hadoop-2.4.1 Example(WordCount) on Eclipse

[Software]
    Hadoop2.4.1
    Eclipse IDE for Java Developers Luna Release (4.4.0)


1. Open a Map/Reducer Project

2. Add lib jar:
- Right click the project > Build Path > Configure Build Path > Java Build Path > Libraries
 > Add External JARS  (including jars in following dir):
  - share/hadoop/common
  - share/hadoop/common/lib
  - share/hadoop/mapreduce
  - share/hadoop/mapreduce/lib
  - share/hadoop/yarn
  - share/hadoop/yarn/lib
  --------additional-----------
  - HDFS lib
  - HBase lib

3. On this project, add new:
- Mapper: Mp.java

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Mapper.Context;

public class Mp extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(LongWritable ikey, Text ivalue, Context context)
throws IOException, InterruptedException {
String line = ivalue.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while(tokenizer.hasMoreTokens()){
word.set(tokenizer.nextToken());
context.write(word, one);
}

}

}

- Reducer: Rd.java

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.Reducer.Context;


public class Rd extends Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text _key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
// process values
int sum =0;
for(IntWritable v : values){
sum += v.get();
}
context.write(_key, new IntWritable(sum));
}

}

- MapReduce Driver: WC.java

import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class WC { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); @SuppressWarnings("deprecation") Job job = new Job(conf, "wordcount"); //Job job = Job.getInstance(conf, "wordcount"); job.setJarByClass(WC.class); // TODO: specify a mapper job.setMapperClass(Mp.class); // TODO: specify a reducer job.setReducerClass(Rd.class); // TODO: specify output types job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); // TODO: specify input and output DIRECTORIES (not files) FileInputFormat.setInputPaths(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); if (!job.waitForCompletion(true)) return; } }

4. Create jar file
- File > Export > JAR file
- Select resources and Jar File Loctaion

5. Run application
- Seclect "Run Configurations" >  Check "Java Application", "Name", "Project", "Mainclass"
- Enter "Arguments" > add "file1 Output" in Program arguments
 [備註] 因為main裡面沒有指定 input output, 所以這邊必須設定給app,
  相當於用terminal 執行 $ hadoop jar project.jar file1 Output1  ,
  如果不加路徑，預設input 及output位置在本機的 $ECLIPSE_WORKSPACE/PROJECT_FOLDER

- Click "Run"

[ 問題 ]  如何Run appliction 於現有的Hadoop系統上，而非 Local端

===> 2014/09/18  目前測試，必須export jar file 丟到master上運行是OK的。

[Solution]

Spark vs Yarn (Simple Grep example)

Environment:
-HARDWARE:
Command:
-cat /proc/cpuinfo # 顯示CPU的資訊
-cat /proc/meminfo # 顯示記憶體的資訊
-sudo smartctl -i /dev/sda # 顯示硬碟型號及規格, apt-get install smartmontools


HOSTNAME IPADDRESS CPU CORE MEM DISK OS
----------------------------------------------------------------------------------------------------------------
master 192.168.0.7 4 8 3.5GB 500GB Ubuntu 14.04.1 LTS
regionserver2 192.168.0.23 2 4 3.5GB 500GB Ubuntu 14.04.1 LTS

-SOFTWARE:
-Hadoop 2.4.1
-Spark 1.0.2
-Scala 2.10.4
-java version: 1.7.0_65

Test Info:
-INPUT
Total Size: 2.8GB
INFO: Linux Redhat / Fedora, Snort NIDS, iptables firewall log file(2006-allog.1 ~ 2006-allog.9)
Date Collected: Sep - Dec 2006
DOWNLOAD: http://log-sharing.dreamhosters.com/   (Bundle 5)

* put data into HDFS
$hdfs dfs -put DATA_DIR/DATA_FOLDER /user/hduser/LogFile
$hdfs dfs -du /user/hduser  # Get the size of "LogFile" folder

-Example : GREP (Count "Dec" in log file)
Using Spark:
$spark-shell --master yarn-client
scala> val textFile = sc.textFile("/user/hduser/LogFile/2006-allog.1")
scala> textFile.filter(line => line.contains("Dec")).count()
scala> exit

Using Hadoop:
$hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar grep LogFile/2006-allog.1 OutPut "Dec"

# This example will execute two jobs, Grep and Sort. We only check the running time of Grep job.

Result

Data Size   spark Hadoop
-------------------------------------------------------------- 
119MB(2006-allog.1) 3.8 sec 21 sec
686MB(2006-allog.9) 5.8 sec 60 sec
2.8GB(LogFile) 25.3 sec        194 sec

Scala HelloWorld Example

[Software]
    Hadoop2.4.1
    Scala 2.10.4

[Hello World] by script
1. Create scala file "Hello.scala" as follow:

object Hello{
 def main(args: Array[String]) {
   println("Hello")
 }
}

2. Compile the scala file
- $ scalac Hello.scala
- It will create 2 files: Hello.class & Hello$.class

3. Execute(at the same dir with Hello.class)
[method 1]
- $ scala Hello.scala
[method 2]
- $ scala -cp . Hello

[Def]
- def : defines a method
- val : defines a fixed value (which cannot be modified)
- var : defines a variable (which can be modified)

OpenTSDB Installation and StartUp

Features
[Reference]
- http://opentsdb.net/index.html

[OpenTSDB Features]
- Scalable, distributed time series database
-

[Download]
- https://github.com/OpenTSDB/opentsdb/releases
- Version: opentsdb-2.0.0
or use command
- $ git clone git://github.com/OpenTSDB/opentsdb.git

Installation and Start Up
- Reference:
- http://opentsdb.net/docs/build/html/installation.html
1. Requirement Install:
    -A Linux system
    -Java Development Kit 1.6 or later
    -GnuPlot 4.2 or later
    -Autotools
    -Make
    -Python
    -Git
    -An Internet connection

2. Install openTSDB:
- $sh opentsdb-2.0.0/build.sh
(If compilation was successfuly, there is a tsdb jarfile in ./build along with a tsdb script)
- $cd build
- $make install or $ ./build.sh
[NOTE]
If error shows as:
|+ ./bootstrap 
|exec: 17: autoreconf: not found
install dh-autoreconf:
$ sudo apt-get install dh-autoreconf

3. Start openTSDB:
1. In src/opentsdb.conf, modify:
tsd.network.port = 8099
tsd.storage.hbase.zk_quorum = 192.168.0.7:2222
tsd.http.staticroot = /home/hduser/opentsdb/build/staticroot
tsd.http.cachedir = /home/hduser/opentsdb/cachedir/

2. Create Table in HBase:(Must execute the command line at the server which install HBase)
- $ env COMPRESSION=NONE HBASE_HOME=/usr/lib/hbase/hbase-0.98.5-hadoop2/ src/create_table.sh

3. Start:
- $ build/tsdb tsd --config src/opentsdb.conf
- TSD's web interface:
http://127.0.0.1:8099   (port is set in "tsd.network.port" )

4. Test openTSDB with simple collector
1. Register metrics at tsdb_uid of HBase
- $ tsdb mkmetric <metric_string1> <metric_string2> ... <metric_stringN>
ex: (Create two metric, "proc.loadavg.1m", "proc.loadavg.5m")
 $ tsdb mkmetric proc.loadavg.1m proc.loadavg.5m

2. Test Collector: Get server avg load and show at TSD web UI
[Reference]
- http://www.slideshare.net/thecupoflife/opentsdb-in-a-real-enviroment
- http://zhengbin.blog.51cto.com/2989505/1273330
- http://opentsdb.net/docs/build/html/user_guide/quickstart.html (about mysql)

1. Create a collector file (Collect local machine info "loadavg")
- loadavg-collector.sh
#!/bin/bash
set -e
while true;
do awk -v now=`date +%s` -v host=`hostname` \
'{ print "put proc.loadavg.1m " now " " $1" host=" host;
print "put proc.loadavg.5m " now " " $2 " host=" host }' /proc/loadavg
 sleep 15
done | nc -w 30 192.168.0.7 8079

[NOTE]
- "set -e" --> causes the shell to exit if any subcommand or pipeline returns a non-zero status
- "awk '{print $1 $2}' /proc/loadavg" --> print out column 1 and 2 all value in /proc/loadavg
- "now=`date +%s` -v host=`hostname`" --> create variables
- "nc -w 30 192.168.0.7 8079" --> connecte to "tsdb_host port"
- "|" --> linux commnd for let two cmds could execute at same time (?)

               2. Run the collector
- $ chmod +x loadavg-collector.sh
- $ sh loadavg-collector.sh (or $ nohup loadavg-collector.sh --> this will output info to nohup.out)

3. Open TSD web UI (192.168.0.7:8079)
- Setup the time line (ex: From 2014/09/24 To now)
- Metric: proc.loadavg.1m

The result diagram will show out, and you can use mouse to select a scope of diagram to show.

[Error]
-  "Request failed: Bad Request: No such name for 'metrics': 'tsd.'"

- [Solution]
- Think
http://grokbase.com/t/cloudfoundry.org/vcap-dev/126b11e3w6/tsdb-configuration-in-vcap-tools-dashboard
Says: There are 2 cases under which the above error happens.
1. collector is not running, so no metrics are pushed to tsdb
2. there is no any web application running in cloudfoundry, so no "frameworks" metrics are pushed to tsdb
- Do
check whether the collectors on monitored hosts are running?

Tcollector
* Start tcollector on hosts
- http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html#installation-of-tcollector
- http://books.google.com.tw/books?id=i5IFvlnfqi8C&pg=PA139&lpg=PA139&dq=opentsdb+monitor+hbase+table&source=bl&ots=kOpk1mpmCx&sig=0LsJOVd22zu2-SAM14CUhgMecMo&hl=zh-TW&sa=X&ei=U14iVPOIOMy48gWuuILQAw&ved=0CEMQ6AEwBQ#v=onepage&q=tcollector&f=false

[!] collector is set up on the host which we need to monitor.(Not Hbase system)

[!] it may right tmp data (to file in /proc/)

Nagios Monitor

1. Install
$sudo apt-get install apache2 nagios3 nagios-nrpe-plugin
(1. Select "Internet Site" for "General type of mail configuration"
2. Select "OK")
3. Set web loggin Password )
$sudo apt-get install nagios3-doc
$sudo apt-get install nagios-nrpe-server

2. Start Nagios
$sudo /etc/nagios3/nagios3 -v nagios..cfg
(  Check no Errors  )

$sudo nano /etc/nagios3/conf.d/hosts.cfg
--------------------------------------------------------

define host{
  use                     generic-host   ; Name of host template to use
  host_name           master
  alias                    master
  dress                192.168.0.7
}
define host{
  use                     generic-host     ; Name of host template to use
  host_name          regionserver2 
  alias                   regionserver2
  address               192.168.0.23
}

--------------------------------------------------------

$sudo nano /etc/nagios3/conf.d/hostgroup_nagios2.cfg
-----------------------Add----------------------------

define hostgroup {
        hostgroup_name  Hadoop_Cluster
        alias           Hadoop
        members         master, regionserver2
 }

---------------------------------------------------------
$sudo /etc/init.d/nagios3 restart

Login:   http://192.168.0.7/nagios3   account: nagiosadmin , password:


[REFERENCE]

http://www.cnblogs.com/junrong624/p/3653988.html   (Installation)

HBase Count Table Rows (Using Java Jar File)

[Software]
    Hadoop2.4.1
    Eclipse IDE for Java Developers Luna Release (4.4.0)
    HBase0.98.5

/*
 * Version:
 * v1 : count rows of appoint table, only map task, output: counter "ROWS"
 */
import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
import org.apache.hadoop.hbase.mapreduce.TableMapper;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;

import org.apache.hadoop.util.GenericOptionsParser;

public class HbaseGet {
private static byte[] tablename;
private static byte[] familyname;
private static byte[] columnname;

public static class GetMap
extends TableMapper<Text, LongWritable> {//in Java: Text=>String, LongWritable=>long

public static enum Counters {Rows, Times};

@Override
public void map(ImmutableBytesWritable rowkey, Result result, Context context)
throws IOException {
byte[] b = result.getColumnLatest(Bytes.toBytes("m0"),  Bytes.toBytes("Tj.00")).getValue();
String msg = Bytes.toString(b);
if(msg != null && !msg.isEmpty())
context.getCounter(Counters.Rows).increment(1);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = HBaseConfiguration.create();
String[] otherArgs = new GenericOptionsParser(conf,args).getRemainingArgs();
if(otherArgs.length !=3){
System.err.println("Wrong number of arguments:"+ otherArgs.length);
System.err.println("Usage: hadoop jar HBaseGet.jar HbaseGet <tablename> <CF> <CN>");
System.exit(-1);
}
tablename  = Bytes.toBytes(otherArgs[0]);
familyname = Bytes.toBytes(otherArgs[1]);
columnname = Bytes.toBytes(otherArgs[2]);

Job job = new Job(conf, otherArgs[0]);
job.setJarByClass(HbaseGet.class);

Scan scan = new Scan();
scan.addColumn(familyname,columnname);
TableMapReduceUtil.initTableMapperJob(
Bytes.toString(tablename),
scan,
GetMap.class,
ImmutableBytesWritable.class,
Result.class, //Single row result of a Get or Scan query
job);
job.setOutputFormatClass(NullOutputFormat.class);
job.setNumReduceTasks(0);
System.exit(job.waitForCompletion(true)?0:1);
}
}