SAPKBA 2356465

Symptom

• You start SAP HANA and you get one of the following (or similar) error messages:

  • FAIL: process hdbdaemon HDB Daemon not running
  • FAIL: process hdbindexserver HDB Indexserver not running
  • FAIL: process hdbxsengine HDB XSEngine not running

• You can find similar entries in SAP HANA service trace files as follows:

  [54558]{-1}[-1/-1] 2020-07-06 18:51:47.616031 f PersistenceLayer PersistenceController.cpp(00723) : startup failed exception 1: no.2000004 (Basis/IO/FileAccess/impl/FileDescriptors.cpp:334)
   Cannot open file ""<root>/logsegment_000_directory.dat" ((closed, mode= RW, access= rw-rw-r--, flags= DIRECT|MUST_EXIST|LOCK), factory= (root= "/hana/log/<SID>/mnt00001/hdb00001/" (access= rw-rw-r--, flags= AUTOCREATE_DIRECTORY, usage= LOG, fs= xfs, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=256,min_submit_batch_size=32,max_submit_batch_size=128))", rc=2: No such file or directory
  exception throw location:
   1: 0x00007fa67fdd4121 in FileAccess::LocalFile::open()+0x460 at FileDescriptors.cpp:334 (libhdbbasis.so)
   2: 0x00007fa67fdd46c8 in FileAccess::LocalFileFactory::open(ltt::basic_string<char, ltt::char_traits<char>, ltt::integer_constant<64> > const&, FileAccess::Mode, int, int)+0x294 at LocalFile.cpp:150 (libhdbbasis.so)
   3: 0x00007fa681302d28 in DataRecovery::LoggerImpl::addPartition(DataRecovery::LoggerPartitionInfo const&)+0xad4 at DirectoryLogSegmentFactory.cpp:403 (libhdbdataaccess.so)
   4: 0x00007fa6812dbe26 in DataRecovery::LoggerImpl::LoggerImpl(ltt::allocator&, DataRecovery::LoggerFactory const&, Diagnose::StatisticsData*, Configuration::ConfigurationHandler&, DataAccess::DuplicateTIDChecker&, DataAccess::Savepoint*, ltt::smartptr_handle<DataAccess::LogShippingHandler>, bool, bool, bool, bool, bool, bool, ltt::guid const&, ltt::guid const&)+0x1212 at LoggerImpl.cpp:1061 (libhdbdataaccess.so)
   5: 0x00007fa6812fa2f8 in DataRecovery::LoggerFactory::createLogger(DataAccess::PersistenceManager&, bool, bool, bool)+0x234 at LoggerImpl.cpp:4278 (libhdbdataaccess.so)
   6: 0x00007fa681074654 in DataAccess::PersistenceManagerImpl::createLogger(bool, bool)+0x50 at PersistenceManagerImpl.cpp:2681 (libhdbdataaccess.so)
   7: 0x00007fa681082b00 in DataAccess::PersistenceManagerImpl::prepareRecoveryOrReplay(bool, bool)+0x50 at DiagTopic.hpp:383 (libhdbdataaccess.so)
   8: 0x00007fa6812b420b in DataAccess::PersistenceManagerImpl::prepareLogRecovery(DataAccess::LogRecoveryCallback&, bool, bool, unsigned long)+0xc7 at PersistenceManagerImpl.cpp:5921 (libhdbdataaccess.so)
   9: 0x00007fa68107f771 in DataAccess::PersistenceManagerImpl::replayOrClearLog(bool, bool, bool, unsigned long&, unsigned long&)+0xf0 at PersistenceManagerImpl.cpp:3071 (libhdbdataaccess.so)
  10: 0x00007fa68104e66e in DataAccess::PersistenceManager::replay(bool)+0x18a at PersistenceManager.cpp:929 (libhdbdataaccess.so)
  11: 0x00007fa6854af865 in PersistenceLayer::PersistenceSystem::start(NameServer::ServiceStartInfo const&, bool, bool)+0x81 at PersistenceSystem.cpp:581 (libhdbpersistence.so)
  12: 0x00007fa6854effa5 in PersistenceLayer::PersistenceFactory::startPersistence(NameServer::ServiceStartInfo&, bool, bool, DataAccess::TablePreloadWriteCallback*, ltt::smartptr_handle<DataAccess::DisasterRecoveryPrimaryCallback>, DataAccess::DisasterRecoveryESCallback*)+0xb1 at PersistenceFactory.cpp:538 (libhdbpersistence.so)
  13: 0x00007fa69f41c3f0 in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, DataAccess::TablePreloadWriteCallback*, DataAccess::TablePreloadReadCallback*, Backup::RecoverCbc_Federation*)+0x5c0 at smartptr_handle.hpp:510 (libhdblogger.so)
  14: 0x00007fa6a223714f in NameServer::Topology::initPersistence(NameServer::ServiceStartInfo&, bool, bool, TREX_ERROR::TRexError*, bool, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >, NameServer::ServiceStartInfo::RequestAction)+0x23b at Topology.cpp:324 (libhdbns.so)
  15: 0x00007fa6a240fe04 in NameServer::TREXNameServer::loadTopology(NameServer::LoadTopologyMode, NameServer::ServiceStartInfo&, Backup::Backup_ExtendedRecoveryInformation*, bool)+0xf50 at TREXNameServer.cpp:11183 (libhdbns.so)
  16: 0x00007fa6a23b7ea4 in NameServer::TREXNameServer::assign(NameServer::ServiceStartInfo&)+0x24a0 at TREXNameServer.cpp:2023 (libhdbns.so)
  17: 0x000055bd0e5dac93 in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0x5f0 at TREXIndexServer.cpp:1068 (hdbnameserver)
  18: 0x000055bd0e613c7a in TRexAPI::AssignThread::run(void*)+0x46 at TREXIndexServer.cpp:535 (hdbnameserver)
  19: 0x00007fa682aada29 in TrexThreads::PoolThread::run()+0x765 at PoolThread.cpp:410 (libhdbbasement.so)
  20: 0x00007fa682ab0376 in TrexThreads::PoolThread::run(void*&)+0x12 at PoolThread.cpp:155 (libhdbbasement.so)
  21: 0x00007fa67fd62726 in Execution::Thread::staticMainImp(void**)+0x3f2 at Thread.cpp:539 (libhdbbasis.so)
  22: 0x00007fa67fd62e86 in Execution::Thread::staticMain(void*)+0x22 at ThreadMain.cpp:31 (libhdbbasis.so)
  23: 0x00007fa67fd5fba9 in Execution::pthreadFunctionWrapper(Execution::PthreadWrapperInfo*)+0x375 at Thread.cpp:1078 (libhdbbasis.so)
  24: 0x00007fa67f840744 in start_thread+0xc0 (libpthread.so.0)
  25: 0x00007fa67e4d3aad in __clone+0x69 (libc.so.6)

• You can also find similar entries in the SAP HANA service trace files as follows:
  

  [53555]{-1}[-1/-1] 2020-07-05 22:41:44.414681 i EventHandler EventManagerImpl.cpp(00620) : New event reported: 'MissingFileEvent[id= 341705, info= LogSegment /hana/log/<SID>/mnt00001/hdb00001/logsegment_000_00000695.dat]'
  [53555]{-1}[-1/-1] 2020-07-05 22:41:44.414690 e PersistenceManag PersistenceManagerImpl.cpp(05199) : Missing 'LogSegment' file: /hana/log/<SID>/mnt00001/hdb00001/logsegment_000_00000694.dat, this might be fatal for future restarts.

Environment

• SAP HANA, platform edition 1.0 SPS09 and higher;
• SAP HANA, platform edition 2.0;

Cause

The log directory/log segments were either moved or deleted.

Resolution

• In case the file logsegment_000_directory.dat is missing, you can check whether the file is really missing under log segment directory /hana/log/<SID>/mnt00001/hdb0000X/.
• If there is no such file or you cannot find the missing log segments, you should recover the database from a backup. Check SAP HANA Recovery;
• If recovering the database from a backup is not possible: as of SAP HANA 1.0 SPS09, you can re-create the directory. Before doing so, please understand that
• There are no means to recover files that were lost or deleted;
• The old directory is saved by appending a timestamp to its name;
• If there were changes that were not saved into the data volumes since the last shutdown (redo work), it might incur in data loss;
• You can re-create the log directory with the following command, while logged as <sid>adm:
  
  hdblogdiag directory --recreate /hana/log/<SID>/mnt00001/hdb0000X/
  
  
• You can only execute the command when the DB is offline, otherwise invalid data may occur;
• For the root cause of who/what possibly moved/deleted the files, you have to contact your Hardware/OS vendor.

Keywords

SAP HANA, sap_hana, hanaplatform, hana_platform, edition, hdblogdiag directory, log segment, mnt00001, logsegment, Cannot open file, startup failed exception, log_segment, can't open, can not, missing, missed, lost, deleted, removed, moved, changed, os