How to recover monasca node after "disk full"

This document (7023932) is provided subject to the disclaimer at the end of this document.

Environment

SUSE OpenStack Cloud 8

Situation

After the disk went full where the log files resides, some of the services are down

Resolution

zookeper:

Move/remove the empty log file, then restart zookeeper.service

monasca storm:

Open the /etc/storm/storm.yaml file. Verify there is a "storm.zookeeper.servers:" entry with an IP address specified (if not, then storm is misconfigured and may need to be fixed).

Using that IP, run '/usr/bin/zkCli.sh -server <ip>:2181', this will start a command line interface.

Run 'ls /storm' and verify there is a list.

Should look like looks like [assignments, backpressure, blobstoremaxkeysequencenumber, credentials, nimbuses, logconfigs, leader-lock, storms, errors, supervisors, workerbeats, blobstore]

Removing with 'rmr /storm' command and run 'quit' to log out of zkCli.sh should clear out the broken references. Restart zookeeper.service,storm-nimbus.service and storm-supervisor.service to get it recreated.

Cause

Following error message in logs:

zookeper:

java.io.EOFException 
at java.io.DataInputStream.readInt(DataInputStream.java:392) 
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) 
...
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)

Issue is that an empty logfile (size is zero) or with invalid headers exists:

/var/lib/zookeeper/data/version-2/

monasca storm:

org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader 
nimbus from seed hosts [localhost]. Did you specify a valid list of nimbus 
hosts for config nimbus.seeds?

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.