fairscheduler修改無效

[Version]
hadoop 0.20.205.0

[Problem]
之前一直在嘗試修改fairscheduler的source code, 利用ant package指令編譯完後產生jar file 在HADOOP_HOME/build/contrib/fairscheduler中, 根據官網手冊, 把fairscheduler的jar檔放進HADOOP_HOME/lib中(照過去經驗與想法上也是如此, 要被引用的class理應都該放到lib中)

http://hadoop.apache.org/common/docs/r0.20.205.0/fair_scheduler.html

Installation

To run the fair scheduler in your Hadoop installation, you need to put it on the CLASSPATH. The easiest way is to copy the hadoop-*-fairscheduler.jar from HADOOP_HOME/build/contrib/fairscheduler to HADOOP_HOME/lib. Alternatively you can modify HADOOP_CLASSPATH to include this jar, in HADOOP_CONF_DIR/hadoop-env.sh

然而怎麼去修改src都沒發現有任何改變, 從HADOOP_HOME/logs/fairscheduler/hadoop-hadoop-fairscheduler-adalab.log中也沒發現加入的東西

然後又發現, 當我把fairscheduler從HADOOP_HOME/lib中拿掉後, 觀察http://<JobTracker>:50030/scheduler, FairScheduler依然在執行！
代表Hadoop看的lib 不是 HADOOP_HOME/lib !

後來一一拿掉hadoop中所有的fairscheuduler jar 檔, 發現當拿掉HADOOP_HOME/share/hadoop/lib/中的那個, FairScheduler才停止運作, 意即系統看的是這個路徑的lib

[Solution]
1. 檢查HADOOP_HOME/conf/hadoop-env.sh , 發現HADOOP_CLASSPATH被註解掉便啟用,
    export HADOOP_CLASSPATH=/home/hadoop/hadoop-0.20.205.0/lib:${HADOOP_CLASSPATH}
    但這是個錯誤的嘗試, 因為已經知道他看得不是HADOOP_HOME/lib

2. 查看了hadoop資料夾看是否有人引用了" /share"這個路徑,  發現

HADOOP_HOME/bin/hadoop:
＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝
if [ -e $HADOOP_PREFIX/share/hadoop/hadoop-core-* ]; then
......
  # add libs to CLASSPATH
  for f in $HADOOP_PREFIX/share/hadoop/lib/*.jar; do
    CLASSPATH=${CLASSPATH}:$f;
  done
......
else
......
  # add libs to CLASSPATH
  for f in $HADOOP_HOME/lib/*.jar; do
    CLASSPATH=${CLASSPATH}:$f;
  done
＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝

if [ -e file]; then
    A
else
    B

指的是, 如果指定的file存在, 則作A, 否則作B,

所以hadoop執行時先看到了share這個資料夾有存在, 所以指定了share中的hadoop lib給系統,換言之就是預設是不會去看放在HADOOP_HOME/lib中的jar檔, 也就是為何把fairscheduler的jar檔放進去沒用!

最簡單的方式就是把編譯好的jar file放到HADOOP_HOME/share/hadoop/lib中, 而不是HADOOP_HOME/lib.

放置到HADOOP_HOME/share/hadoop/lib後確實有看到修改的跡象：

HADOOP_HOME/logs/fairscheduler/hadoop-hadoop-fairscheduler-adalab.log
...

2012-03-05 18:49:22,803    PREEMPT_VARS    default    MAP    0    0
2012-03-05 18:49:22,803    PREEMPT_VARS    default    REDUCE    0    0
2012-03-05 18:49:22,803    HELLOOOOOOOOOOOOOOOOOOOOOO!!!!!
2012-03-05 18:49:23,303    PREEMPT_VARS    default    MAP    0    0
2012-03-05 18:49:23,303    PREEMPT_VARS    default    REDUCE    0    0
2012-03-05 18:49:23,303    HELLOOOOOOOOOOOOOOOOOOOOOO!!!!!
2012-03-05 18:49:23,368    HEARTBEAT    tracker_algoq-ProLiant-DL380-G6:localhost/127.0.0.1:60564
2012-03-05 18:49:23,368    RUNNABLE_TASKS    0    0    0    0
2012-03-05 18:49:23,368    INFO    Can't assign another MAP to HEEEEEEEEEEEELLLLLOOOOO!!!!!!!tracker_algoq-ProLiant-DL380-G6:localhost/127.0.0.1:60564
2012-03-05 18:49:23,368    HEEEEEEEEEEEELLLLLOOOOO!!!!!!!

... 

[Reference]
hadoop scheduler的編譯的編譯