Exceptionally long voice queue length queuing to cbann

EXCEPTIONALLY LONG VOICE QUEUE LENGTH QUEUING TO CBANN GENERATOR
EXCEPTIONALLY LONG VOICE QUEUE LENGTH QUEUING TO CBANN DRIVERS

WARNING channel.c: Exceptionally long voice queue length queuing to IAX2/****-11606 Phones drop offline, The CLI becomes unresponsive to any commands and the only way to get the system back is to kill the processes and reset asterisk. It's usually triggered by someone 'applying settings' to the system, but it happens once out of every 50 or so reloads. I have been unable to find any references to out of order log entries in my searches.I'm having an issue a couple of times a week where all of a sudden Asterisk becomes unresponsive. I'm assuming whatever is causing the log to get out of order is also causing the dropped audio. Plus, those 3 second anomalies probably average out in the graphs.

I don't see anything funny in the historic or real-time stats vCenter is feeding me, although I fully admit I may be looking in all the wrong places.

Asterisk C.3.7.2 (Business Edition, 1.4 trunk).

No over committed resources & CPU/Disk IO allocations are set to 'High' for this VM.

VMware HA cluster, services around 50 VMs.

VERBOSE logger.c: - Agent/436 answered environment: DEBUG res_musiconhold.c: SIP/peertrunk3-00002937 Opened file 0 '/home/asterisk/env/agentwaiting/agent-waiting-long' DEBUG rtp.c: Difference is 30512, ms is 3834

EXCEPTIONALLY LONG VOICE QUEUE LENGTH QUEUING TO CBANN GENERATOR

DEBUG channel.c: Started silence generator on 'SIP/peertrunk3-00001029'

DEBUG channel.c: Scheduling timer at 160 sample intervals DEBUG channel.c: Set channel SIP/peertrunk3-00001029 to write format slin

EXCEPTIONALLY LONG VOICE QUEUE LENGTH QUEUING TO CBANN DRIVERS

DEBUG devicestate.c: No provider found, checking channel drivers for Agent - 436 DEBUG devicestate.c: Notification of state change to be queued on device/channel Agent/436 DEBUG chan_agent.c: Set write format, result '0' DEBUG chan_agent.c: Set read format, result '0' DEBUG chan_agent.c: Waited for stream, result '0' DEBUG channel.c: Scheduling timer at 0 sample intervals DEBUG channel.c: Avoiding initial deadlock for channel '0x7f31c8787b10' DEBUG channel.c: Scheduling timer at 138 sample intervals DEBUG manager.c: Manager received command 'StopMonitor' DEBUG channel.c: Set channel SIP/peertrunk3-00001029 to write format gsm DEBUG channel.c: Stopped silence generator on 'SIP/peertrunk3-00001029' WARNING channel.c: Exceptionally long voice queue length queuing to 12:20:45] DEBUG channel.c: Scheduling timer at 0 sample intervals DEBUG rtp.c: Got RTCP report of 44 bytes DEBUG rtp.c: Difference is 31288, ms is 3931 DEBUG res_musiconhold.c: SIP/peertrunk3-000013d1 Opened file 0 '/home/asterisk/env/agentwaiting/agent-waiting-long' DEBUG app_dial.c: Dunno what to do with control type 25

DEBUG chan_sip.c: = No match Their Call ID: Their Tag Our tag: as28e70c40 DEBUG rtp.c: Got RTCP report of 64 bytes DEBUG app_queue.c: Device 'SIP/SIP/peertrunk4-0000211d' changed to state '4' (Invalid) but we don't care because they'r$ Log snipit: DEBUG devicestate.c: Changing state for SIP/SIP/peertrunk4-0000211d - state 4 (Invalid) In the log below, you'll see it jump around between 12:20:45 and 12:20:48. In the RTP captures, the 1 second of garbled audio consists of many 'Wrong Timestamp' markers. The symptom discovered today is, during these short periods of audio problems the log records are out of order, that is the second in the timestamp jumps around forward and backwards. Flow data and Wireshark playback shows it's originating from this system. Every once and a while we experience bouts of 3 seconds worth of lost audio, followed by 1 sec of garbled audio. I have an Asterisk server that serves upwards of 450 agents, it's been in production for a little over a year.

Exceptionally long voice queue length queuing to cbann