顯示具有 Tcollector 標籤的文章。 顯示所有文章
顯示具有 Tcollector 標籤的文章。 顯示所有文章

2014年10月13日 星期一

Java TCP Server receive Linux nc command

[Problem]

Linux 的 nc 指令(netcat)可透過 TCP/UDP 來傳送資料或訊息,但我目前無法使用 nc 或其他Linux指令來創造一個MultiThread TCP Server,僅能透過 nc 做一對一的傳送資料
如  Put Data into Remote HBase by using Linux Bash Script ,因此改用Java 寫一個MultiThreaded Server

------------------------------------------------------------------------------------------------------------
[Concept]


Reference:
How to write RAW data to a file using Java? e.g same as: nc -l 8000 > capture.raw
Multithreaded Server in Java
-------------------------------------------------------------------------------------------------------------
[Code]

tcpserver.java  (Main Class)

package tcpserver;

public class tcpserver {
public static void main(String args[]){
MultiThreadedServer server = new MultiThreadedServer(9075);
System.out.println("***************************");
System.out.println("** TCP Collection Server **");
System.out.println("***************************");
new Thread(server).start();
}
}

MultiThreadedServer.java

package tcpserver;

import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;

public class MultiThreadedServer implements Runnable{
protected int          serverPort   = 50200;
    protected ServerSocket serverSocket = null;
    protected boolean      isStopped    = false;
    protected Thread       runningThread= null;

    public MultiThreadedServer(int port){
        this.serverPort = port;
        System.out.println("System port setting : " + port);
    }

    public void run(){
     
    synchronized(this){
            this.runningThread = Thread.currentThread();  //Avoid another same Server(?)
        }
   
        openServerSocket();
     
        System.out.println("[INFO] Start Listening!");
        while(!isStopped()){
            Socket clientSocket = null;
            try {
                clientSocket = this.serverSocket.accept(); //wait client
            } catch (IOException e) {
                if(isStopped()) {
                    System.out.println("[ERROR] Server Stopped.") ;
                    return;
                }
                throw new RuntimeException("[ERROR] Error accepting client connection", e);
            }
            // Create a New Thread for client
            new Thread(
            new WorkerRunnable(clientSocket)
            ).start();
        }
        System.out.println("Server Stopped.") ;
    }


    private synchronized boolean isStopped() {
        return this.isStopped;
    }

    public synchronized void stop(){
        this.isStopped = true;
        try {
            this.serverSocket.close();
        } catch (IOException e) {
            throw new RuntimeException("Error closing server", e);
        }
    }

    private void openServerSocket() {
        try {
            this.serverSocket = new ServerSocket(this.serverPort);
            System.out.println("[INFO] Create a ServerSocket");
        } catch (IOException e) {
            throw new RuntimeException("Cannot open port 8080", e);
        }
    }
}

WorkerRunnable.java

package tcpserver;

import java.io.*;
import java.net.*;

public class WorkerRunnable implements Runnable{
protected Socket clientSocket = null;

public WorkerRunnable(Socket clientSocket){
this.clientSocket = clientSocket;
}

public void run(){
try{
byte[] buff = new byte[1024];
int bytes_read=0;
InputStream data = clientSocket.getInputStream(); // get input data
while(( bytes_read = data.read(buff)) != -1){     // print out
System.out.println(new String(buff));
//String str = new String(buff);
}
data.close();
}
catch(IOException e){
e.printStackTrace();
}
}
}

-------------------------------------------------------------------------------------------------------------
[Result]

2014年9月30日 星期二

OpenTSDB Installation and StartUp

Features
[Reference]
- http://opentsdb.net/index.html

[OpenTSDB Features]
- Scalable, distributed time series database
-

[Download]
- https://github.com/OpenTSDB/opentsdb/releases
- Version: opentsdb-2.0.0
or use command
- $ git clone git://github.com/OpenTSDB/opentsdb.git

Installation and Start Up
- Reference:
- http://opentsdb.net/docs/build/html/installation.html
1. Requirement Install:
    -A Linux system
    -Java Development Kit 1.6 or later
    -GnuPlot 4.2 or later
    -Autotools
    -Make
    -Python
    -Git
    -An Internet connection

2. Install openTSDB:
- $sh opentsdb-2.0.0/build.sh
(If compilation was successfuly, there is a tsdb jarfile in ./build along with a tsdb script)
- $cd build
- $make install or $ ./build.sh
[NOTE]
If error shows as:
|+ ./bootstrap 
|exec: 17: autoreconf: not found
install dh-autoreconf:
$ sudo apt-get install dh-autoreconf

3. Start openTSDB:
1. In src/opentsdb.conf, modify:
tsd.network.port = 8099
tsd.storage.hbase.zk_quorum = 192.168.0.7:2222
tsd.http.staticroot = /home/hduser/opentsdb/build/staticroot
tsd.http.cachedir = /home/hduser/opentsdb/cachedir/

2. Create Table in HBase:(Must execute the command line at the server which install HBase)
- $ env COMPRESSION=NONE HBASE_HOME=/usr/lib/hbase/hbase-0.98.5-hadoop2/ src/create_table.sh

3. Start:
- $ build/tsdb tsd --config src/opentsdb.conf
- TSD's web interface:
http://127.0.0.1:8099   (port is set in "tsd.network.port" )

4. Test openTSDB with simple collector
1. Register metrics at tsdb_uid of HBase
- $ tsdb mkmetric <metric_string1> <metric_string2> ... <metric_stringN>
ex: (Create two metric, "proc.loadavg.1m", "proc.loadavg.5m")
 $ tsdb mkmetric proc.loadavg.1m proc.loadavg.5m

2. Test Collector: Get server avg load and show at TSD web UI
[Reference]
- http://www.slideshare.net/thecupoflife/opentsdb-in-a-real-enviroment
- http://zhengbin.blog.51cto.com/2989505/1273330
- http://opentsdb.net/docs/build/html/user_guide/quickstart.html (about mysql)

1. Create a collector file (Collect local machine info "loadavg")
- loadavg-collector.sh
#!/bin/bash
set -e
while true;
do awk -v now=`date +%s` -v host=`hostname` \
'{ print "put proc.loadavg.1m " now " " $1" host=" host;
print "put proc.loadavg.5m " now " " $2 " host=" host }' /proc/loadavg
 sleep 15
done | nc -w 30 192.168.0.7 8079

[NOTE]
- "set -e" --> causes the shell to exit if any subcommand or pipeline returns a non-zero status
- "awk '{print $1 $2}' /proc/loadavg" --> print out column 1 and 2 all value in /proc/loadavg
- "now=`date +%s` -v host=`hostname`" --> create variables
- "nc -w 30 192.168.0.7 8079" --> connecte to "tsdb_host port"
- "|" --> linux commnd for let two cmds could execute at same time (?)

               2. Run the collector
- $ chmod +x loadavg-collector.sh
- $ sh loadavg-collector.sh (or $ nohup loadavg-collector.sh --> this will output info to nohup.out)

3. Open TSD web UI (192.168.0.7:8079)
- Setup the time line (ex: From 2014/09/24 To now)
- Metric: proc.loadavg.1m

The result diagram will show out, and you can use mouse to select a scope of diagram to show.

[Error]
-  "Request failed: Bad Request: No such name for 'metrics': 'tsd.'"

- [Solution]
- Think
http://grokbase.com/t/cloudfoundry.org/vcap-dev/126b11e3w6/tsdb-configuration-in-vcap-tools-dashboard
Says: There are 2 cases under which the above error happens.
1. collector is not running, so no metrics are pushed to tsdb
2. there is no any web application running in cloudfoundry, so no "frameworks" metrics are pushed to tsdb
- Do
check whether the collectors on monitored hosts are running?

Tcollector
* Start tcollector on hosts
- http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html#installation-of-tcollector
- http://books.google.com.tw/books?id=i5IFvlnfqi8C&pg=PA139&lpg=PA139&dq=opentsdb+monitor+hbase+table&source=bl&ots=kOpk1mpmCx&sig=0LsJOVd22zu2-SAM14CUhgMecMo&hl=zh-TW&sa=X&ei=U14iVPOIOMy48gWuuILQAw&ved=0CEMQ6AEwBQ#v=onepage&q=tcollector&f=false

[!] collector is set up on the host which we need to monitor.(Not Hbase system)

[!] it may right tmp data (to file in /proc/)