Wanpipe Asterisk Debugging
- Hardware Echo Canceler Echo/Fax/DTMF Debugging
- Software echo canceller debugging
- Hardware/Physical Layer Debugging
- Line/Connection Debugging
- Line Trace
- Is my Hardware echo canceller running?
- DTMF Recognition Debugging
- Asterisk/Wanpipe RBS Debugging
- Asterisk PRI Span Debugging
- Asterisk A500 BRI debugging
- WANPIPE® Echo Spike Generation/Debugging (Linux)
WANPIPE® Echo Spike Generation: Debugging (Linux)
When a system suffers from an intractable echo, it is often very difficult to diagnose the origins and possible solutions to the echo problem. WANPIPE® has a utility that allows the echo from a unit impulse to be examined and analyzed.
Important:
If you have Sangoma Card with Hardware echo canceller, the echo spike is not very important or useful.
In this case we suggest using hardware echo canceller tools to determine the echo problem:
Please read the Sangoma Hardware Echo canceler debugging howto .
To debug Echo problem during the phone call:
1. Install WANPIPE version beta1w-2.3.4 or later, use the following command when installing:
./Setup install --echo-debug
The Setup script will patch Zaptel driver, so it will have to be recompiled/reinstalled after WANPIPE installation is complete. Software Echo canceller should be enabled on the span, where the test will be done (or on all spans).
2. Make the phone call and check if there is echo.
Find a number that shows consistent echo for this test. The test should be done from the side where the echo is heard. Ideally, mute the phone on the remote side during the test. Note the Zap channel number on which the test call is done, for example channel 12. The rest of this document assumes the test call is active on Zap channel 12.
3. Run the following command:
#>wanpipemon -zap -c ses -zapchan 12
A test impulse is sent on zap channel 12. The Finite Impulse Response (FIR) is saved to the file: /etc/wanpipe/span_1_chan_12_before_ec.spike as a set of linear samples taken 8000 times per second.
4. Copy the data from /etc/wanpipe/span_1_chan_12_before_ec.spike into a spreadsheet.
Create a line chart of the data. The result should look similar to this picture:
The graph shows the response to a unit impulse (FIR). Note that the start of the FIR is at 55 taps (7ms) after the impulse, and the echo was attenuated by 80 taps (10 ms). If the echo canceller had been configured for 128 taps, the echo would have been properly cancelled. If the Echo Canceller had been set to 64 taps, much of the echo would have survived the echo cancellation, and there would have been audible echo on this call.
Line Debugging
When debugging your connection, you should start from the physical layer and move upwards.
1. Check if you are physically connected, type:
#>wanrouter status
If it says connected, move to the next step, else:
Check if your cables are plugged in.
Check if you are using the correct port (A102/4) (Port 1 is the lowest one, the active port will be red or green (connected/disconnected)
Check if you are using the proper cables. (Line from Telco uses straight T1/E1 Cable; Line to channel bank or to another PBX uses cross T1/E1 cable)
Check if you are using the proper coding/framing. (contact your Telco for this information)
2. Check if you are having errors/overruns, type repeatedly:
#>ifconfig <interface name>
#> wanpipemon -i <interface name> -c sc (for FreeBSD)
#> wanpipemon -i <interface name> -c Ta (for Linux)
Your interface name is usually w1g1, w2g1, w3g1,...
It is normal to have a few errors (usually occurs during line synchronization), if your number of overruns and errors is not incrementing go to step 3:
If your number of errors is increasing:
You could be using the wrong framing/coding mode.
Try replacing your cables.
If you have noise on the line, contact your telco to test the line.
If your number of overruns is increasing:
Go to the overruns debugging section:
http://wiki.sangoma.com/wanpipe-linux-asterisk-appendix#overruns
If this does not help your overruns issue, contact Sangoma Tech Support.
For A500- BRI cards, when connecting with Telco, Please confirm that you have configured your card in TE-MODE. Use "wanrouter hwprobe verbose command" to check mode.
For example :
[root@drv-test ~]# wanrouter hwprobe verbose
-----------------------------------------
| Wanpipe Hardware Probe Info (verbose) |
-----------------------------------------
1 . AFT-A500-SH : SLOT=0 : BUS=8 : IRQ=11 : PORT=1 : HWEC=64 : V=32
+01:TE
2 . AFT-A500-SH : SLOT=0 : BUS=8 : IRQ=11 : PORT=2 : HWEC=64 : V=32
+02:TE
3. Check if you are having line/bit errors, or alarms. type:
#>wanpipemon -i <interface name> -c Ta
Check if you have any alarms on.
RED (Red Alarm) - Physical Layer issue, go back to step 1.
YEL (Yellow Alarm) - Other end of the connection is down; Contact your telco or check the connected PBX/Channel Bank
ALOS/LOS (Loss of Signal) - Physical Layer issue, go back to step 1.
OOF (Out of Frame) - You could be using the wrong framing mode.
If all alarms are off, repeat command several times and check if any of the counters are incrementing. If counters are not incrementing abnormally, go to next step. Else:
If you have Bit errors or Line code violation incrementing, go to step 1.
If you have Framing bit errors or Bit errors, you could be using the wrong framing mode.
4. Launch Zaptel, type:
#>ztcfg -vvv
If you have no errors, proceed to the next step, otherwise:
If you have this message (Linux only):
Notice: Configuration file is zaptel.conf
line 0: Unable to open master device '/dev/zap/ctl'
The zap device was not ready, wait 2-3 seconds and try again. (See script to autoconfigure zaptel).
If you get the same message after 4-5 seconds, then you probably did not configure UDEV for Zaptel devices,
go to zaptel source directory (usually usr/src/zaptel), type:
#>make install-udev
Then try again.
If you get this error:
ZT_CHANCONFIG failed on channel XX: No such device or address (6)
Your zaptel.conf is not configured properly. Click here for sample zapata.conf configurations.
5. Launch Asterisk with added verbosity and command line
#>asterisk -vvvvvvvvvc
If you are in the Asterisk CLI, go to the next step, else:
If you get this error:
Jan 12 23:30:06 WARNING[6896]: chan_zap.c:912 zt_open: Unable to open '/dev/zap/channel': No such file or directory
:
:
:
Jan 12 23:30:06 WARNING[6896]: loader.c:554 load_modules: Loading module chan_zap.so failed!
Then your zapata.conf is not configured properly. Click here.
If you get something like:
Signalling requested is FXO KewStart but channel signalling is FXO Loopstart...
Then your /etc/zaptel.conf and /etc/asterisk/zapata.conf do not match.
6. In the Asterisk CLI, type:
#>zap show channels
Check that you have the correct amount of channels registered in Asterisk, and in the proper context.
You should an entry for each of your zap channels in addition to your pseudo channel, for example:
Chan Extension Context
pseudo from-incoming
1 from-pstn
2 from-pstn
3 from-pstn
If you are using a PRI ISDN connection, type:
#>pri show span <span number>
It should say:
*CLI> pri show span 2
Primary D-channel: 48
Status: Provisioned, Up, Active
:
:
N200 Counter: 3
If it says:
Status: Provisioned, Down, Active
You are not receiving frames, start a trace .
If D Channel goes up and down,
Recompile/reinstall zaptel.
Go to zaptel source directory and type:
#>make clean; make install
Line Trace using hardware d-channel HDLC (2.3.3 drivers or later)
1. Performing a trace on your sangoma card. (Assuming you have already passed steps 1-3 of Line Debugging)
Type:
#>wanpipemon -i <interface name> -c trd
If you see nothing:
Check if you are using hardware d-channel HDLC. The TDMV_DCHAN parameter should be set to 24 (T1) or 16 (E1).
If you see only outgoing packets:
Contact your telco, your line is not provisioned yet.
If you see incoming and outgoing packets:
Your card is configured properly, start a trace on Asterisk.
2. Performing a trace on Asterisk
In the Asterisk CLI, type:
#>pri show span <span number>
if it says:
pri command not found, then you did not install/configure libpri properly.
if it says:
no pri running on span <span number>
then your zaptel.conf and/or zapata.conf are not configured properly.
if it says:
provisioned, down, active
and you had incoming and outgoing packets when using wanpipemon trace, then your packets are not reaching Asterisk, recompile/reinstall zaptel (make sure zaptel modules are not loaded while doing this).
Software echo canceller debugging
Here are a few steps we have used to debug echo.
1. Find out if the Echo canceller is even ON. Sometimes the EC stays off for some reason.
Set echocancel = no, and make a call to a number that you know has bad echo.
Then make it with echocancel = yes. You should notice a _BIG_ difference. If you have to think about it, then there is no real difference and echo cancellation was off both times. The difference should be dramatic.
You can confirm by placing a call and using "zap show channel x" to see the echo cancellation state. If you see that echo cancellation is set to zero taps, then somehow echo cancellation is not enabled. If you see echo cancellation set to 128, 64 or 32 taps, then the global echo cancellation is enabled.
2. If the EC is working play with the gains.
The software echo cancellers are quite good at reducing echo about 25dB, but you will often have a little echo remaining. This echo residual can be reduced a lot by playing with the tx and rx gains. Try txgain=rxgain=-10 to start. It should have a dramatic effect on the residual echo, at the expense of volume. You can adjust the attenuation and will usually find a setting with reasonable volume and acceptable echo.
3. Limitations of the software EC: The EC in zaptel has been optimized for the lowest system load, so there have been several compromises made. Firstly the training sequence is run much less frequently than in a hardware EC, so training takes 10-15 seconds rather than about 1/4 second. So you will hear echo for the first 10-15 seconds, after which it will largely fade. There is nothing you can do about that in s/w, except use the echotraining feature.
The other problem is that the non-linear processor is missing entirely from the s/w EC so that there is always residual echo that has to be manipulated with the gains.
4. Try decreasing your txgain and rxgain in zapata.conf, and increase the volume on your handsets.
5. You can try a different software echo canceller.
In your zaptel source. modify your zconfig.h file.
Look for #define ECHO_CAN and try a different echo canceller.
6. If all other methods of reducing echo have failed, Asterisk also has an option in the zconfig.h to makethe echo cancellation more aggressive. You can enable it by uncommenting the following line:
#define AGGRESSIVE_SUPPRESSOR
Note that aggressive echo cancellation can create a walkie-talkie, half-duplex effect. This should be enabled only if all above methods have failed.
You will get better performance from good carrier-grade hardware EC such as used on our "d" models .
DTMF Recognition Debugging
When using Asterisk, the DTMF is detected and interpreted by Asterisk. If you are having DTMF recognition problems, you can try the following:
1. Set relaxdtmf=yes
In your zapata.conf, set "relaxdtmf=yes", and confirm that relax DTMF is set to yes on your channel by typing "zap show channel N" in your asterisk CLI.
2. Increase your rxgain
Increase your rxgain in your zapata.conf and see if this helps.
3. Check your voice quality
If you have a lot of noise on your line, this will affect your DTMF recognition performance. Try to reduce the noise on the line.
4. Check for echo.
If you have a lot of echo on the line, this will affect your DTMF recognition. Check if you have a lot of echo during calls, and if so, go through the echo cancelling cookbooks:
http://wiki.sangoma.com/wanpipe-linux-asterisk-debugging#SWEC
http://wiki.sangoma.com/wanpipe-linux-asterisk-appendix#confirmHWEC
Hardware echo canceller Fax/DTMF Debugging
Instructions for A500
If you are experiencing ECHO or DTMF/Fax problems on your system using a card only when the hardware echo canceller is enabled, please follow the instructions below on how to capture an echo debug file.
The hardware echo canceller chip has a debug feature where a DEBUG Monitor will capture chip debug data during active calls. The captured debug binary file can be sent to Sangoma for further processing and analysis.
Before proceeding you should also confirm that Is my Hardware echo canceller running?
1. Establish a call that has an echo problem noise or DTMF problem
2. Determine the call channel number by running "show channels" on Asterisk CLI:
CLI> show channels
3. Run a echo canceler debug utility on that channel
#>wan_ec_client wanpipe1 monitor <channel number>
Where <channel number> is a channel number obtained in step 2.
4. After 2 minutes of talk and noticeable echo, or 30 seconds of bad DTMF or noise.
5. Obtain the debug statistics using:
#>wan_ec_client wanpipe1 monitor
The wan_ec_client will write a binary file in your local directory.
6. Send the binary file back to techdesk@sangoma.com
Hardware echo canceler Fax/DTMF Debugging for A500
- Determine which Wanpipe number and channel number the call is using with the aid of the following link: HOW TO DETERMINE SPAN AND CHANNEL on A500
- Start the echo canceler monitoring tool by executing "wan_ec_client wanpipeX monitor Y" (where X is the wanpipe number and Y is the channel number found in step 1) from the Linux Command prompt.
- Keep the call active for at least 1 minute (2 minutes is better) by having a normal conversation (the weather is always a fun topic). You should here echo on the local phone.
- Stop the monitoring tool and create a binary file of the audio by running the command "wan_ec_client wanpipeX monitor" (where X is the wanpipe number found in step 1). This will take a second or so.
- The binary recording will be saved to the current working directory with the date and channel number in the title.
Hardware/Physical Layer Debugging
If you suspect that your card has a physical defect, run the patlooptest .
Debugging Asterisk/Wanpipe RBS
The wanpipemon utility provides the ability to debug RBS bits. The following command enables/disables WANPIPE RBS debugging and prints all debugging message into the /var/log/messages file. This feature is supported for all Sangoma digital AFT-series cards.
* To enable RBS debugging on reciever side (this command will print only changes to the RBS bit settings):
wanpipemon -i <interface name> -c derr
* To disable RBS debugging on reciever side:
wanpipemon -i <interface name> -c ddrr
* To enable RBS debugging on transmit side (this command will print only changes to the RBS bit settings):
wanpipemon -i <interface name> -c dert
* To disable RBS debugging on transmit side:
wanpipemon -i <interface name> -c ddrt
* Read the current RBS status bits from AFT-series cards:
wanpipemon -i <interface name> -c drr
* Read the current RBS status from the Wanpipe driver:
wanpipemon -i <interface name> -c dpr
In order to verify the RBS operations, you can run ./zttool utility and enable Sangoma RBS debugging at the same time. Zttoll shows what zaptel thinks is the RBS setting and the wanpipe utilities show what the sangoma driver thinks is the RBS setting. The two have to match at all times.
- In one window run: zttool
- In another window run: tail -f /var/log/messages
- Enable RBS debugging as stated above:
wanpipemon -i <interface name> -c derr #Enable rx rbs debugging
wanpipemon -i <interface name> -c dert #Enable tx rbs debugging
- Place a call on Asterisk with reproducable bad behaviour
- Compare zttool output versus the /var/log/messages
If driver rbs changes are identical to zttool rbs changes problem is with the telco.
If driver rbs changes differ from zttool rbs changes there could be a problem with the drivers.
- If zttool output doesnt match wanpipe rbs output please contact Sangoma Support.
Asterisk PRI Span Debugging
If you are having problems with a PRI span follow the steps below to troubleshoot the issue:
1)Stop Asterisk and then restart the wanpipe drivers using "wanrouter restart"
2)Check the output from "wanrouter status", is the line "connected", "connecting", or "disconnected"
-if the status is "connecting" then check that there are no major alarms on the line using "wanpipemon -i w1g1 -c Ta"
-->OOF alarm means the incoming signal isn't quite right, check your line settings E1/T1, line framing, and line coding (contact your telco if you are not 100% sure)
-->LOS or Loss of Signal alarm means there is no signal at all on the line; first check the cable then check with your telco that the line is live
-->Short Circuit alarms means that the pin out of the cable is wrong
-if the status is "connected" then check that there are no RAI alarms on the line by looking at /var/log/messages
-if the status is "disconnected" then check that there is a cable connected to the card and then restart the drivers again.
3)Check and make sure that the Wanpipe network interface is processing data properly. Run "ifconfig" and make sure that Rx and Tx packets are increasing for the wanpipe interface (the wanpipe interfaces use the naming pattern of "w1g1" for port 1, "w2g1" for port 2,etc). If the values are not increasing then please send the following to techdesk@sangoma.com:
-the output from "wanrouter version"
-the output from "wanrouter restart" (asterisk needs to be turned off for this so it may need to be done after hours)
-the file /var/log/messages (the entire file containing a wanrouter restart)
-the output from "wanrouter hwprobe verbose"
-the output from "wanrouter status"
-the file(s) /etc/wanpipe/wanpipeX.conf (where X is the wanpipe number)
-the output from "ifconfig", (re run a couple of times with a second or so delay in between each run)
-the output from "cat /proc/interrupts" (re run a couple of times with a second or so delay in between each run)
4)Run "ztcfg -vvv" to register the wanpipes with Zaptel, check for any error messages, if you see errors check your zaptel.conf file.
5)Run "asterisk" followed by "asterisk -r" to connect to the Asterisk console. Run "zap show channels" and make sure that Asterisk has all the Zap channels, if not check you zapata.conf or zapata-auto.conf and verify that these files correct register the Zap channels.
6)Run "pri show span 1", check whether the span is "Up and Active". If it is not "Up and active" turn on "pri intense debug span 1" in Asterisk to see the D-channel messages, make sure there are incoming and outgoing messages. Once you have confirmed that there incoming and outgoing frames confirm also that the Wanpipe drivers are seeing the same; from the Linux command prompt run "wanpipemon -i w1g1 -c trd" (NOTE: this will only work if Wanpipe is doing the HDLC framing, which it is by default)
If there are only incoming frames check the following link: Hardware HDLC
If there are only outbound frames then contact you telco and confirm that the D-channel has been activated.
If there are both incoming and outgoing frames and you are using Zaptel-1.4.X and Wanpipe-3.2.X check this link: HARDHDLC vs DCHAN
7)If the span is "up and active" then turn up the verbosity of Asterisk to 10 using "set verbose 10". Try to make a call out and look at the messages that Asterisk is generating, is the call transfered to the Zap channel? (i.e Dial(Zap/...)
8)If the call is transfered to a Zap channel then turn on "pri intense debug span 1" and try to make a call out. You will now see the PRI messages related to the call, check for a Cause Code and look it up online (ISDN Cause codes).
9)If you are still having problems then please send the following to techdesk@sangoma.com along with a brief description of the problem:
-the output from "wanrouter version"
-the output from "wanrouter restart" (asterisk needs to be turned off for this so it may need to be done after hours)
-the file /var/log/messages (the entire file containing a wanrouter restart)
-the output from "wanrouter hwprobe verbose"
-the output from "wanrouter status"
-the file(s) /etc/wanpipe/wanpipeX.conf (where X is the wanpipe number)
-the file /etc/zaptel.conf
-the file(s) /etc/asterisk/zapata*.conf
-the output from "ifconfig", (re run a couple of times with a second or so delay in between each run)
-the output from "cat /proc/interrupts" (re run a couple of times with a second or so delay in between each run)
-the output from "wanpipemon -i wXg1 -c Ta" (where X is the interface number 1,2,3,etc)
-log all output from the Asterisk console to a file:
=>open up the file /etc/asterisk/logger.conf
=>look for the line "messages=> notice,warning ..." and add this one line below "sangoma => notice,warning,error,event,verbose"
=>save the file and restart Asterisk
-increase the verbosity of Asterisk using "set verbose 10"
-the output from "pri show span X" from the Asterisk console, where X is the span having problems (i.e. 1,2,3,4,etc)
-run "pri intense debug span X" from the Asterisk console so that Asterisk captures debug information, where X is the span having problems (i.e. 1,2,3,4,etc)
-make a call out to reproduce the problem.
-turn off Asterisk logging to the file /var/log/asterisk/sangoma by:
=>open up the file /etc/asterisk/logger.conf
=>look for the line "sangoma => notice,warning,error,event,verbose", and either remove it or uncomment it
=>save the file and restart Asterisk
-send me the file /var/log/asterisk/sangoma
***To turn of the debugging in Asterisk run "pri no debug span X", where X is the span in question***
|
