SCOPTEL IP PBX Software - Troubleshooting ¶

PDF Version

General Troubleshooting
Log files
SIP Response Codes 1XX - SIP Response Codes 2XX
SIP Response Codes 3XX - SIP Response Codes 400-416
SIP Response Codes 417-439 - SIP Response Codes 440-494
SIP Response Codes 5XX - SIP Response Codes 6XX
OS CLI Commands 1-3
OS CLI Commands 2-3
OS CLI Commands 3-3
Asterisk CLI Commands
OS CLI Commands 3-3
Asterisk CLI Commands
Database Repair Commands
MySQL Repair is Taking Too Long
SCOPTEL GUI is Slow
SCOPTEL GUI Will Not Restart
SCOPTEL GUI Fatal Error and Dead Call Processing
Debugging Authentification Failures 1-3
Debugging Authentification Failures 3-3
Debugging Call Failures with ‘database show’
Debugging Call Failures with SIP Cause Codes
SCOPSTATS System Monitoring Logs

General Troubleshooting ¶

Most telephony errors are caused by one of the following causes:

Bad authentication credentials
DNS resolution
Class of Service misconfigurations
Firewall issues
Asterisk bugs
User error (DND, call forwarding etc.)

CLI tools are used to debug most of these issues

Log files ¶

Server Logs: /var/log/messages
SMTP Logs: /var/log/maillog
Hardware Logs: dmesg
Asterisk Logs: /var/log/asterisk/messages
Asterisk Files: /etc/asterisk
SCOPSERV PHP logs: /tmp/php2.log
Asterisk Core Dumps (GDB files): dir /tmp/core*

SIP Response Codes 3XX ¶

Redirection	3xx
300	Multiple Choices
301	Moved Permanently
302	Moved Temporarily
305	Use Proxy
380	Alternative Service

SIP Response Codes 4XX ¶

Request Failure	4xx
400	Bad Request
401	Unauthorized
402	Payment Required
403	Forbidden
404	Not Found
405	Method Not Allowed
406	Not Acceptable
407	Proxy Authentication Required
408	Request Timeout
410	Gone
412	Conditional Request Failed	[RFC3903]
413	Request Entity Too Large
414	Request-URI Too Long
415	Unsupported Media Type
416	Unsupported URI Scheme
417	Unknown Resource-Priority	[RFC4412]
420	Bad Extension
421	Extension Required
422	Session Interval Too Small	[RFC4028]
423	Interval Too Brief
424	Bad Location Information	[RFC6442]
428	Use Identity Header	[RFC4474]
429	Provide Referrer Identity	[RFC3892]
430	Flow Failed	[RFC5626]
433	Anonymity Disallowed	[RFC5079]
436	Bad Identity-Info	[RFC4474]
437	Unsupported Certificate	[RFC4474]
438	Invalid Identity Header	[RFC4474]
439	First Hop Lacks Outbound Support	[RFC5626]
440	Max-Breadth Exceeded	[RFC5393]
469	Bad Info Package	[RFC6086]
470	Consent Needed	[RFC5360]
480	Temporarily Unavailable
481	Call/Transaction Does Not Exist
482	Loop Detected
483	Too Many Hops
484	Address Incomplete
485	Ambiguous
486	Busy Here
487	Request Terminated
488	Not Acceptable Here
489	Bad Event	[RFC6665]
491	Request Pending
493	Undecipherable
494	Security Agreement Required	[RFC3329]

SIP Response Codes 5XX ¶

Server Failure	5xx
500	Server Internal Error
501	Not Implemented
502	Bad Gateway
503	Service Unavailable
504	Server Time-out
505	Version Not Supported
513	Message Too Large
580	Precondition Failure	[RFC3312]

SIP Response Codes 6XX ¶

Global Failure	6xx
600	Busy Everywhere
603	Decline
604	Does Not Exist Anywhere
606	Not Acceptable

OS CLI Commands ¶

`SCOPSERV_yum update`	executes a full software update
`SCOPSERV_yum install <package name>`	installs a package from repositories
`SCOPSERV_dumpkey`	displays license and admin password
`dkms status`	shows install status of dahdi wanpipe mISDN drivers
`asterisk -r`	opens Asterisk CLI for Asterisk command line debugging
`asterisk -vr`	opens Asterisk CLI for Asterisk command line debugging with increased verbosity
`lsof -i :5555`	shows which service is using port 5555
`rpm -qa <package name>`	shows package version
`rpm -e <package name> --nodeps`	removes package without uninstalling dependencies
`rpm -qa \|grep <package name>`	shows installed packages version
`chown -R SCOPSERV:SCOPSERV /<path>`	changes ownership recursively on a path
`kill -KILL <pid>`	kills a process ID
`htop`	shows all processes and consumption
`top`	shows all processes and consumption
`ngrep -d any port 25 -W BYLINE`	network trace on port 25 to debug SMTP issues
`network trace on port 25 to debug SMTP issues`	Linux DNS lookup for MX record on specified domain
`rm -rf /root/.ssh/known_hosts`	rm -rf /root/.ssh/known_hosts
`wget <package path>`	CLI command to download a file from path
`ngrep -d port any 69`	network trace on port 69 to display TFTP traffic
`cat /<path>`	display contents of file to screen
`nano /<path>`	Linux text editor to edit path
`service <service name> restart`	restart network without rebooting server
`service <service name> stop`	stop service name without rebooting server
`service <service name> status`	display service name status
`reboot`	reboot server
`shutdown -h now`	shutdown server
`lynx <website address>`	Linux command line web browser
`lynx <website address>`	Ping an IP address
`lynx <website address>`	trace an IP address through multiple hops
`tcpdump -nq -s 0 -i eth0 -w /tmp/sip.pcap port not ssh`	saves a SIP pcap trace to /tmp using eth0
`ps auwwwwx \|grep crond`	prints all crond PID’s
`rpm -e --allmatches <package name> -nodeps`	removes the package name without dependencies
`rpm -e <package name> --nodeps -noscripts`	removes a broken package forcibly
`route`	prints the routing table
`ifconfig`	The “ifconfig” command allows the operating system to setup network interfaces and allow the user to view information about the configured network interfaces.

Asterisk CLI Commands ¶

`core show channels`	display active channels
`sip show peers`	show all SIP peers status
`sip show peer <peer>`	show details of SIP peer
`database show`	display astDB info (useful to see if a phone is in DND status
`queue show`	displays all ACD queues info
`pri show spans`	displays status of all PRI spans
`pri intense debug span <span number>`	shows Q.931 and SABME messages
`pri set debug off span <span number>`	stop intense PRI debugging on span number
`hangup request <channel>\|<all>`	Request that a channel be hung up. The hangup takes effect the next time the driver reads or writes from the channel. If ‘all’ is specified instead of a channel name, all channels will see the hangup request.
`core show hints`	This command lists registered hints
`agi set debug on`	prints agi debug messages (always useful when creating a support ticket)
`sip set debug <on/off>`	prints SIP messages to the CLI

Database Repair Commands ¶

mysqlcheck -A -o -r

Automatically optimizes and repairs mysql tables. If crashed on Slave (HA) then this need to be done on slave and not master

If you have to rebuild queue_log db :

service SCOPSERV_queuelog stop
`` /var/www/SCOPSERV/telephony/scripts/queue2sql.pl –recover /var/log/asterisk/queue_log ``
service SCOPSERV_queuelog start

Update CDR database direction myql SCOPSERV (enter) update cdr set calldir='unknown';

MySQL Repair is Taking Too Long ¶

If the Server is not configured in High Availability Mode (standard installation):

mysql SCOPSERV

truncate cel;
truncate phones_history;
truncate horde_histories;

If the Server is configured in High Availability Mode:

mysql SCOPSERV_repl

truncate cel;
truncate phones_history;
truncate horde_histories;

SCOPTEL GUI is Slow ¶

mysqlcheck -A -o -r : Automatically optimizes and repairs mysql tables. If crashed on Slave (HA) then this need to be done on slave and not master

SCOPSERV_yum update : Older packages, especially prior to SCOPSERV_realtime nodejs changes are not fully optimized Telephony>Configuration>Reports (CDR/ACD)>Archiving>Enable Archiving :[x]

../_images/Module19ScopTELTroubleshooting1.png

SCOPTEL GUI Will Not Restart ¶

mysqlcheck -A -o -r	Automatically optimizes and repairs mysql tables. If crashed on Slave (HA) then this need to be done on slave and not master
service SCOPSERV stop	Stops the GUI service and realtime
lsof -i :5555	Displays the PID that is bound to the SCOPSERV service port
kill -KILL <PID output of lsof -I 5555>	Kills the PID bound to the SCOPSERV port
service SCOPSERV start	Starts the GUI once the binding port is freed

SCOPTEL GUI Fatal Error and Dead Call Processing ¶

Most often this is caused by a full hard drive
You must delete files on the hard drive to restore call processing
Usually this is caused by:
- Too many local backups
- Too many recording files in /var/spool/asterisk/monitor/
- Too many files in /var/log/asterisk/
- Too many files in /tmp/
To locate the largest folders do:

cd /	Change to root directory
du * -s	Print Disk Usage Summary and locate the largest folder

Use the ‘cd’ command to change into the largest directory
Use the Linux ‘rm’ command to remove files
Keep drilling down from / and use the ‘du * -s’ command to locate the largest folders and keep deleting files with the ‘rm’ command until you have freed enough space in the file system to restore normal service.

Debugging Authentications Failures ¶

Here is a typical authentication failure in Asterisk. The reason for this failure is that peer 228 does not exist!

`[root@demo ~]# asterisk -vr`¶

[Aug  1 14:41:46] Running as group 'SCOPSERV'
[Aug  1 14:41:46] Connected to Asterisk 1.8.12.0 currently running on demo (pid = 6006)
Verbosity is at least 3
Core debug is at least 3
[2012-08-01 14:50:06] NOTICE[6260]: chan_sip.c:24974 handle_request_register: Registration from '"228"<sip:228@SCOPSERV.local>' failed for '192.168.192.55:57744' - No matching peer found

Here is a typical authentication failure in Asterisk. The reason for this failure is that the password does not match!

`[root@demo ~]# asterisk -vr`¶

[Aug  1 14:41:46] Running as group 'SCOPSERV'
[Aug  1 14:41:46] Connected to Asterisk 1.8.12.0 currently running on demo (pid = 6006)
Verbosity is at least 3
Core debug is at least 3
[2012-08-01 14:53:49] NOTICE[6260]: chan_sip.c:24974 handle_request_register: Registration from '"227"<sip:227@SCOPSERV.local>' failed for '192.168.192.55:63316' - Wrong password

Here is a typical authentication success in Asterisk

`[root@demo ~]# asterisk -vr`¶

[Aug  1 14:41:46] Running as group 'SCOPSERV'
[Aug  1 14:41:46] Connected to Asterisk 1.8.12.0 currently running on demo (pid = 6006)
Verbosity is at least 3
Core debug is at least 3
[2012-08-01 14:56:00] NOTICE[6260]: chan_sip.c:20812 handle_response_peerpoke: Peer '227' is now Reachable. (7ms / 2000ms)

Debugging Call Failures With ‘database show’¶

In this first example a call from 227 to 221 fails to ring 221

[2012-08-01 15:02:05] == Extension Changed 227[default-local] new state InUse for Notify User 221
[2012-08-01 15:02:05] -- Executing [221@default-default:9] Macro("SIP/227-00000005", "default-dial,SIP/221,221,default,,en,u221@default,twWxXkKg,,default_default,,Local/0@default-local/n,vm") in new stack
[2012-08-01 15:02:05] -- Executing [s@macro-default-dial:1] NoOp("SIP/227-00000005", ""CALL TO LOCAL EXTENSION FROM 227(227)"") in new stack
[2012-08-01 15:02:05] -- <SIP/227-00000005>AGI Script agi://127.0.0.1:4573/dial completed, returning 0
[2012-08-01 15:02:05] -- Executing [s@macro-default-dial:4] ExecIf("SIP/227-00000005", "1?Macro(all-vm,default,b,221@default,Local/0@default-local/n,vm)") in new stack
[2012-08-01 15:02:05] -- Executing [s@macro-all-vm:1] NoOp("SIP/227-00000005", "CALL TO VOICEMAIL") in new stack
[2012-08-01 15:02:05] -- Executing [s@macro-all-vm:2] VoiceMail("SIP/227-00000005", "221@default,b") in new stack

database show the extension has enabled the DND feature code! demo*CLI> database show

/CustomDevstate/dnd-default-221 : BUSY
/DND/Local/*78@default          : On
/DND/SIP/221                    : On

Debugging Call Failures with SIP Cause Codes ¶

In this second example a call from 227 to 221 fails to ring 221

[2012-08-01 15:24:27] -- Executing [221@default-default:1] Set("SIP/227-00000009", "CDR(tenant)=default") in new stack
[2012-08-01 15:24:27] -- Executing [221@default-default:9] Macro("SIP/227-00000009", "default-dial,SIP/221,221,default,,en,u221@default,twWxXkKg,,default_default,,Local/0@default-local/n,vm") in new stack
[2012-08-01 15:24:27] -- Executing [s@macro-default-dial:1] NoOp("SIP/227-00000009", ""CALL TO LOCAL EXTENSION FROM 227(227)"") in new stack
[2012-08-01 15:24:27] -- Called SIP/221
[2012-08-01 15:24:27] -- Got SIP response 302 "Moved Temporarily" back from 192.168.100.100:5060
[2012-08-01 15:24:27] -- Now forwarding SIP/227-00000009 to 'Local/555@default-default' (thanks to SIP/221-0000000a)
[2012-08-01 15:24:27] NOTICE[30807]: app_dial.c:892 do_forward: Not accepting call completion offers from call-forward recipient Local/555@default-default-59dd;1
[2012-08-01 15:24:27] NOTICE[30807]: chan_local.c:899 local_call: No such extension/context 555@default-default while calling Local channel
[2012-08-01 15:24:27] NOTICE[30807]: app_dial.c:980 do_forward: Forwarding failed to dial 'Local/555@default-default'
[2012-08-01 15:24:27]  == Everyone is busy/congested at this time (1:0/0/1)

The SIP response code shows the extension has enabled a native SIP forward to 555 from the SIP phone! There is no 555 in the dial plan and the forward fails. The user must remove the bad forwarding attempt from their phone.

SCOPSTATS System Monitoring Logs ¶

The SCOPSTATS reporting engine can also display System Logs instead of accessing the Linux file system

../_images/Module19ScopTELTroubleshooting2.png

Table of Contents

Previous topic

Next topic

This Page

SCOPTEL IP PBX Software - Troubleshooting ¶

General Troubleshooting ¶

Log files ¶

SIP Response Codes 3XX ¶

SIP Response Codes 4XX ¶

SIP Response Codes 5XX ¶

SIP Response Codes 6XX ¶

OS CLI Commands ¶

Asterisk CLI Commands ¶

Database Repair Commands ¶

MySQL Repair is Taking Too Long ¶

SCOPTEL GUI is Slow ¶

SCOPTEL GUI Will Not Restart ¶

SCOPTEL GUI Fatal Error and Dead Call Processing ¶

Debugging Authentications Failures ¶

`[root@demo ~]# asterisk -vr`¶

`[root@demo ~]# asterisk -vr`¶

`[root@demo ~]# asterisk -vr`¶

Debugging Call Failures With ‘database show’¶

Debugging Call Failures with SIP Cause Codes ¶

SCOPSTATS System Monitoring Logs ¶