• Hadoop 2.6 日志文件和MapReduce的log文件


    ref:https://blog.csdn.net/infovisthinker/article/details/45370089

     mr-jobhistory-daemon.sh命令是在${HADOOP_INSTALL}/sbin/目录下面。启动完了,用jps命令可以看到有JobHistoryServer的进程启动。
    启动了HistoryServer后,就可以看到Tracking URL里面的History了。
    根据博主所写,测试了一下,发现启动historyserver后,多出了mapred-root-historyserver-bigdata.log / out 2个文件。

    [root@bigdata userlogs]# /home/admin/Downloads/hadoop-2.10.0/sbin/mr-jobhistory-daemon.sh start historyserver
    starting historyserver, logging to /home/admin/Downloads/hadoop-2.10.0/logs/mapred-root-historyserver-bigdata.out
    [root@bigdata userlogs]# cd ..
    [root@bigdata logs]# ls
    hadoop-root-datanode-bigdata.log                           mapred-root-historyserver-bigdata.log
    hadoop-root-datanode-bigdata.out                           mapred-root-historyserver-bigdata.out
    hadoop-root-datanode-localhost.localdomain.log             SecurityAuth-root.audit
    hadoop-root-datanode-localhost.localdomain.out             userlogs
    hadoop-root-datanode-localhost.localdomain.out.1           yarn-root-nodemanager-bigdata.log
    hadoop-root-namenode-bigdata.log                           yarn-root-nodemanager-bigdata.out
    hadoop-root-namenode-bigdata.out                           yarn-root-nodemanager-localhost.localdomain.log
    hadoop-root-namenode-localhost.localdomain.log             yarn-root-nodemanager-localhost.localdomain.out
    hadoop-root-namenode-localhost.localdomain.out             yarn-root-nodemanager-localhost.localdomain.out.1
    hadoop-root-namenode-localhost.localdomain.out.1           yarn-root-resourcemanager-bigdata.log
    hadoop-root-secondarynamenode-bigdata.log                  yarn-root-resourcemanager-bigdata.out
    hadoop-root-secondarynamenode-bigdata.out                  yarn-root-resourcemanager-localhost.localdomain.log
    hadoop-root-secondarynamenode-localhost.localdomain.log    yarn-root-resourcemanager-localhost.localdomain.out
    hadoop-root-secondarynamenode-localhost.localdomain.out    yarn-root-resourcemanager-localhost.localdomain.out.1
    hadoop-root-secondarynamenode-localhost.localdomain.out.1  yarn-root-resourcemanager-localhost.localdomain.out.2
    hadoop-root-secondarynamenode-localhost.localdomain.out.2  yarn-root-resourcemanager-localhost.localdomain.out.3

     我们cat一下 mapred-root-historyserver-bigdata.log,下面是部分显示内容:

    STARTUP_MSG:   build = ssh://git.corp.linkedin.com:29418/hadoop/hadoop.git -r e2f1f118e465e787d8567dfa6e2f3b72a0eb9194; compiled by 'jhung' on 2019-10-22T19:10Z
    STARTUP_MSG:   java = 1.8.0_241
    ************************************************************/
    2020-04-11 17:03:07,796 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT]
    2020-04-11 17:03:09,022 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2020-04-11 17:03:09,146 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
    2020-04-11 17:03:09,146 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobHistoryServer metrics system started
    2020-04-11 17:03:09,152 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: JobHistory Init
    2020-04-11 17:03:09,831 INFO org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://bigdata:9000]
    2020-04-11 17:03:10,272 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Perms after creating 504, Expected: 504
    2020-04-11 17:03:10,301 INFO org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://bigdata:9000]
    2020-04-11 17:03:10,313 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Perms after creating 493, Expected: 1023
    2020-04-11 17:03:10,313 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Explicitly setting permissions to : 1023, rwxrwxrwt
    2020-04-11 17:03:10,325 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing Jobs...
    2020-04-11 17:03:10,344 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Found 0 directories to load
    2020-04-11 17:03:10,345 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Existing job initialization finished. 0.0% of cache is occupied.
    2020-04-11 17:03:10,346 INFO org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage Init
    2020-04-11 17:03:10,553 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
    2020-04-11 17:03:10,567 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 10033
    2020-04-11 17:03:10,843 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
    2020-04-11 17:03:10,853 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
    2020-04-11 17:03:10,853 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
    2020-04-11 17:03:10,956 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2020-04-11 17:03:10,976 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
    2020-04-11 17:03:10,985 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.jobhistory is not defined
    2020-04-11 17:03:10,990 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
    2020-04-11 17:03:10,992 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context jobhistory
    2020-04-11 17:03:10,992 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
    2020-04-11 17:03:10,992 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
    2020-04-11 17:03:10,994 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /jobhistory/*
    2020-04-11 17:03:10,994 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
    2020-04-11 17:03:11,491 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
    2020-04-11 17:03:11,493 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 19888
    2020-04-11 17:03:11,494 INFO org.mortbay.log: jetty-6.1.26
    2020-04-11 17:03:11,687 INFO org.mortbay.log: Extract jar:file:/home/admin/Downloads/hadoop-2.10.0/share/hadoop/yarn/hadoop-yarn-common-2.10.0.jar!/webapps/jobhistory to /tmp/Jetty_bigdata_19888_jobhistory____38atwj/webapp
    2020-04-11 17:03:13,251 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@bigdata:19888
    2020-04-11 17:03:13,251 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app jobhistory started at 19888
    2020-04-11 17:03:13,276 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 1000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
    2020-04-11 17:03:13,295 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 10020
    2020-04-11 17:03:13,333 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.HSClientProtocolPB to the server
    2020-04-11 17:03:13,336 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
    2020-04-11 17:03:13,355 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 10020: starting
    2020-04-11 17:03:13,494 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryClientService: Instantiated HistoryClientService at bigdata/192.168.0.108:10020
    2020-04-11 17:03:13,511 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
    2020-04-11 17:03:13,526 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 10033: starting
    2020-04-11 17:03:13,577 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
    2020-04-11 17:03:40,856 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
    2020-04-11 17:03:40,867 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
    2020-04-11 17:06:10,853 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
    2020-04-11 17:09:10,853 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
  • 相关阅读:
    卷积神经网络与典型结构
    机器学习之信息熵
    机器学习读书笔记第三章(1):线性模型
    神经网络之:S型神经元
    mysql只保留一条有效数据,删除其他重复的数据
    mysql索引
    mysql自定义函数收集
    MySql中循环的使用
    WCF的例子
    C盘满了如何清理
  • 原文地址:https://www.cnblogs.com/watermarks/p/12681022.html
Copyright © 2020-2023  润新知