SUSE Cloud 2.0 Chef Crowbar RuntimeError: Forked convergence run failed.

This document (7014304) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Cloud 2.0

Situation

On a new deployment of SUSE Cloud 2.0 it was found that the Chef Crowbar service was repeatedly dying.

When verifying the /var/chef/cache/chef-stacktrace.out file the following was listed :

RuntimeError: Forked convergence run failed
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/client.rb:172:in `run'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application.rb:140:in `run_chef_client'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:274:in `run_application'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:267:in `loop'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:267:in `run_application'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application.rb:72:in `run'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19

Resolution

The resolution to the problem of the chef crowbar service dying, was to simply enlarge the /var file system from 1 GB, to at least 10 GB, to accommodate the system requirements.

Cause

A /var file system with a size of only 1 GB was deployed, which is too small to accommodate the system requirements.

It is very important to realize that the /var file system is not 'just' only considered to be used for log files. This also contains quite some data, and with SUSE Cloud 2.0, includes data stored by Rabbitmq.

(Note: The Rabbitmq behavior is documented on the RabbitMQ website. Please read the section on Disk-Based Flow Control.

Additional Information

The problem of RabbitMQ becoming unresponsive is visible when looking at the :

/var/log/rabbitmq/rabbit@<server_name>.log-<date> log file.

This log file contained a series of events as listed below :

=INFO REPORT==== 10-Nov-2013::00:28:39 ===
Disk free space insufficient. Free bytes:999919616 Limit:1000000000

RabbitMQ is used by chef, once RabbitMQ stops working, this will in turn block chef, with chef being blocked, this will in turn block crowbar again.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.