SUSE Support

Here When You Need Us

SUSE Cloud 2.0 Chef Crowbar RuntimeError: Forked convergence run failed.

This document (7014304) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Cloud 2.0

Situation

On a new deployment of SUSE Cloud 2.0 it was found that the Chef Crowbar service was repeatedly dying.

When verifying the /var/chef/cache/chef-stacktrace.out file the following was listed  :

RuntimeError: Forked convergence run failed
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/client.rb:172:in `run'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application.rb:140:in `run_chef_client'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:274:in `run_application'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:267:in `loop'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:267:in `run_application'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application.rb:72:in `run'
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19


Resolution

The resolution to the problem of the chef crowbar service dying, was to simply enlarge the /var file system from 1 GB, to at least 10 GB, to accommodate the system requirements.

Cause

A /var file system with a size of only 1 GB was deployed, which is too small to accommodate the system requirements.

It is very important to realize that the /var file system is not 'just' only considered to be used for log files. This also contains quite some data, and with SUSE Cloud 2.0, includes data stored by Rabbitmq.

(Note: The Rabbitmq behavior is documented on the RabbitMQ website. Please read the section on Disk-Based Flow Control.

Additional Information

The problem of RabbitMQ becoming unresponsive is visible when looking at  the :
 /var/log/rabbitmq/rabbit@<server_name>.log-<date> log file.

This log file contained a series of events as listed below :
=INFO REPORT==== 10-Nov-2013::00:28:39 ===
Disk free space insufficient. Free bytes:999919616 Limit:1000000000


RabbitMQ is used by chef, once RabbitMQ stops working, this will in turn block chef, with chef being blocked, this will in turn block crowbar again.


Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7014304
  • Creation Date: 17-Dec-2013
  • Modified Date:03-Mar-2020
    • SUSE Open Stack Cloud

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

tick icon

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

tick icon

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.

tick icon

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.