rsyslog discards/delays local messages - if syslog remote target is down

l9931022:~ # cat /etc/rsyslog.d/ora_audit.conf
if ( $syslogfacility-text == 'local1' ) and ( $syslogseverity == 4 /* warning */ )  then {
        $WorkDirectory /var/spool/rsyslog       # where to place spool files
        $ActionQueueFileName RemoteQueue        # unique name prefix for spool files
        $ActionQueueMaxDiskSpace 1G             # 1gb space limit (use as much as possible)
        $ActionQueueSaveOnShutdown on           # save messages to disk on shutdown
        $ActionQueueType LinkedList             # run asynchronously
        $ActionResumeRetryCount -1              # infinite retries if host is down
        $ActionQueueTimeoutEnqueue 0            # causes that the message will be discarded immediately if the queue is full
        *.* @@aaa.bbb.ccc.ddd:514               # IP of remote syslog server:port 514
        stop
}

Above setup discards any messages, if queue size limit is reached (in this setup 1 GB) immediately !

The initial customer configuration requested a disk-assisted ActionQueue with limited 
disk space and did not specify, what client-side rsyslog daemon should do, when the queue 
becomes full. 

With default values, the queue is trying not to drop any message unless necessary and as 
a consequence is not able to quickly handle the big load from reproducer.

Rsyslog has one main message queue for incoming messages. They are processed and stored in 
ActionQueues belonging to each action configured in rsyslog.conf.

If one of the ActionQueues is full and rsyslog cannot put a message there, the message is 
by default not discarded immediately, but it's placed back to main queue and rsyslog tries 
to deliver it later. So, a single full ActionQueue can significantly slow down the whole 
rsyslog daemon. Then it tries to throttle the data senders and free some queue space in 
the meantime. The throttling can slow down the whole server or even make it unresponsible, 
when processes like sshd are waiting on write to syslog socket.

To prevent that, the administrator has to configure proper discard rules. 
(as described in the "Resolution" example)

Rsyslog has fairly extensive options for specifying, what to discard and when.

For example, "$ActionQueueTimeoutEnqueue 0" option causes, that the message will be discarded 
immediately, if the queue is full, "$ActionQueueDiscardSeverity" could be set to drop all 
queued low-priority messages, if the treshold  "$ActionQueueDiscardMark" is reached or even 
increase the maximum number of threads, handling this queue, using "$ActionQueueWorkerThreads" 
to improve performance.