• 【hbase】使用thrift with python 访问HBase


    HBase 版本: 0.98.6

    thrift   版本: 0.9.0

    使用 thrift client with python 连接 HBase 报错:

     1 Traceback (most recent call last):
     2   File "D:workspacePythonpyhelloworld.py", line 27, in <module>
     3     tables = client.getTableNames()
     4   File "E:mazhongsoftpythonlibhbaseHbase.py", line 788, in getTableNames
     5     return self.recv_getTableNames()
     6   File "E:mazhongsoftpythonlibhbaseHbase.py", line 798, in recv_getTableNames
     7     (fname, mtype, rseqid) = self._iprot.readMessageBegin()
     8   File "E:mazhongsoftpythonlib	hriftprotocolTBinaryProtocol.py", line 126, in readMessageBegin
     9     sz = self.readI32()
    10   File "E:mazhongsoftpythonlib	hriftprotocolTBinaryProtocol.py", line 203, in readI32
    11     buff = self.trans.readAll(4)
    12   File "E:mazhongsoftpythonlib	hrift	ransportTTransport.py", line 58, in readAll
    13     chunk = self.read(sz-have)
    14   File "E:mazhongsoftpythonlib	hrift	ransportTTransport.py", line 155, in read
    15     self.__rbuf = StringIO(self.__trans.read(max(sz, self.DEFAULT_BUFFER)))
    16   File "E:mazhongsoftpythonlib	hrift	ransportTSocket.py", line 94, in read
    17     raise TTransportException('TSocket read 0 bytes')
    18 thrift.transport.TTransport.TTransportException: None

    查找原因,过程如下:
    1) 客户端代码

    1 transport = TTransport.TBufferedTransport(TSocket('192.168.0.10', 9090))
    2 protocol = TBinaryProtocol.TBinaryProtocol(transport)
    3 client = Hbase.Client(protocol)
    4 transport.open()

    2) hbase-site.xml 配置如下

    1   <property>
    2      <name>hbase.regionserver.thrift.framed</name>
    3      <value>true</value>
    4   </property>
    5   <property>
    6      <name>hbase.regionserver.thrift.compact</name>
    7      <value>true</value>
    8   </property>

    3) thrift server日志如下:(/var/log/hbase/hbase-hbase-thrift-<hostname>.log)

     1 Wed Jan 14 14:54:43 CST 2015 Starting thrift on <hostname>
     2 core file size          (blocks, -c) 0
     3 data seg size           (kbytes, -d) unlimited
     4 scheduling priority             (-e) 0
     5 file size               (blocks, -f) unlimited
     6 pending signals                 (-i) 191956
     7 max locked memory       (kbytes, -l) unlimited
     8 max memory size         (kbytes, -m) unlimited
     9 open files                      (-n) 32768
    10 pipe size            (512 bytes, -p) 8
    11 POSIX message queues     (bytes, -q) 819200
    12 real-time priority              (-r) 0
    13 stack size              (kbytes, -s) 10240
    14 cpu time               (seconds, -t) unlimited
    15 max user processes              (-u) 1024
    16 virtual memory          (kbytes, -v) unlimited
    17 file locks                      (-x) unlimited
    18 INFO  [main] util.VersionInfo: HBase 0.98.6-cdh5.3.0
    19 INFO  [main] util.VersionInfo: Subversion file:///data/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hbase-0.98.6-cdh5.3.0 -r Unknown
    20 INFO  [main] util.VersionInfo: Compiled by jenkins on Tue Dec 16 19:13:29 PST 2014
    21 INFO  [main] thrift.ThriftServerRunner: Using default thrift server type
    22 INFO  [main] thrift.ThriftServerRunner: Using thrift server type threadpool
    23 INFO  [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties
    24 INFO  [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
    25 INFO  [main] impl.MetricsSystemImpl: HBase metrics system started
    26 INFO  [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    27 INFO  [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    28 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context thrift
    29 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
    30 INFO  [main] http.HttpServer: Jetty bound to port 9095
    31 INFO  [main] mortbay.log: jetty-6.1.26.cloudera.4
    32 INFO  [main] mortbay.log: Started HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:9095
    33 DEBUG [main] thrift.ThriftServerRunner: Using compact protocol
    34 DEBUG [main] thrift.ThriftServerRunner: Using framed transport
    35 INFO  [main] thrift.ThriftServerRunner: starting TBoundedThreadPoolServer on /0.0.0.0:9090; min worker threads=16, max worker threads=1000, max queued requests=1000

    由以上红色文字部分可知,是因为thrift 的server端和client端的协议不匹配造成的。

    解决方案有如下两个:
    方案一:修改hbase-site.xml,禁用TFramedTransport和TCompactProtocol功能,即:

    1   <property>
    2      <name>hbase.regionserver.thrift.framed</name>
    3      <value>false</value>
    4   </property>
    5   <property>
    6      <name>hbase.regionserver.thrift.compact</name>
    7      <value>false</value>
    8   </property>

    重启thrift 服务器: service hbase-thrift restart

    方案二:修改客户端代码

    1 transport = TFramedTransport(TSocket('192.168.0.10', 9090))  
    2 protocol = TCompactProtocol.TCompactProtocol(transport)  
    3 client = Hbase.Client(protocol)  
    4 transport.open()  

    如果报错未找到TFramedTransport和TCompactProtocol,请查看/usr/lib/python2.6/site-packages/thrift目录下有没有protocol/TCompactProtocol.py文件,检查transport/TTransport.py文件中有没有类TFramedTransport。如果没有,可以在thrift官网下载源码包 thrift-0.9.2.tar.gz,用其中的lib/py/src/覆盖/usr/lib/python2.6/site-packages/thrift/目录即可。

  • 相关阅读:
    关于overflow:hidden的作用(溢出隐藏、清除浮动、解决外边距塌陷等等)
    float详解
    五 使用并美化网页文本
    代码校验工具 SublimeLinter 的安装与使用
    Sublime Text3 快捷键
    word选择+快捷键
    Word1-提取图片文字
    javascript window对象属性和方法
    word教程字体和段落设置
    Word2 word输入公式+文字转表格
  • 原文地址:https://www.cnblogs.com/zhangningbo/p/4303422.html
Copyright © 2020-2023  润新知