Elasticsearch crash due to corrupted data

Published by bolmaster2

Elasticsearch doesn’t start? First thing to remember is where is it started from? Homebrew services? Let’s check that…

brew services list

Name              Status  User Plist
elasticsearch@2.4 started joel /Users/joel/Library/LaunchAgents/homebrew.mxcl.elasticsearch@2.4.plist

Realize elasticsearch is yellow ☝️…

Ok. So let’s check if it’s running?

ps aux | grep elasticsearch

Nope… Nothing there.

Ok, so let’s see what the logs say? Check the launch agent plist file. They are located under ~/Library/LaunchAgents/ according to the homebrew documentation. You can also find it from the brew services list command.

cat ~/Library/LaunchAgents/homebrew.mxcl.elasticsearch\@2.4.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>KeepAlive</key>
    <false/>
    <key>Label</key>
    <string>homebrew.mxcl.elasticsearch@2.4</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/opt/elasticsearch@2.4/bin/elasticsearch</string>
    </array>
    <key>EnvironmentVariables</key>
    <dict>
    </dict>
    <key>RunAtLoad</key>
    <true/>
    <key>WorkingDirectory</key>
    <string>/usr/local/var</string>
    <key>StandardErrorPath</key>
    <string>/usr/local/var/log/elasticsearch@2.4.log</string>
    <key>StandardOutPath</key>
    <string>/usr/local/var/log/elasticsearch@2.4.log</string>
  </dict>
</plist>

There it is (StandardErrorPath), let’s check that out…

tail -f -n 1000 /usr/local/var/log/elasticsearch@2.4.log

Realize there’s broken files ☝️ which CorruptStateException implies:

Exception in thread "main" ElasticsearchException[failed to read [id:29, legacy:false, file:/usr/local/var/elasticsearch/elasticsearch_joel/nodes/0/_state/global-29.st]]; nested: IOException[failed to read [id:29, legacy:false, file:/usr/local/var/elasticsearch/elasticsearch_joel/nodes/0/_state/global-29.st]]; nested: CorruptStateException[Format version is not supported (resource SimpleFSIndexInput(path="/usr/local/var/elasticsearch/elasticsearch_joel/nodes/0/_state/global-29.st")): 1 (needs to be between 0 and 0)];
Likely root cause: org.elasticsearch.gateway.CorruptStateException: Format version is not supported (resource SimpleFSIndexInput(path="/usr/local/var/elasticsearch/elasticsearch_joel/nodes/0/_state/global-29.st")): 1 (needs to be between 0 and 0)

Delete shit

rm -rf /usr/local/var/elasticsearch

Restart it!

brew services restart elasticsearch@2.4

Check if it’s running:

ps aux | grep elasticsearch
joel             97960   0,1  1,6  6920728 263744   ??  S    10:23am   0:06.96 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/local/Cellar/elasticsearch@2.4/2.4.6/libexec -cp /usr/local/Cellar/elasticsearch@2.4/2.4.6/libexec/lib/elasticsearch-2.4.6.jar:/usr/local/Cellar/elasticsearch@2.4/2.4.6/libexec/lib/* org.elasticsearch.bootstrap.Elasticsearch start
joel             98305   0,0  0,0  4267752    716 s006  R+   10:23am   0:00.00 grep elasticsearch

Boom 💥. Work completed 😅.