Wireshark (also known as ethereal or tethereal) is a "network protocol analyzer. It lets you capture and interactively browse the traffic running on a computer network." If you want a definitive answer on what is causing problems with your VoIP calls, you need to learn how to use Wireshark. Thankfully, it's pretty simple for anyone with basic knowledge of networking.

There are two parts to successfully running a network trace: capturing the traffic and viewing the results. I'll go through capturing traffic on different operating systems and viewing the results on a Windows PC using Wireshark.

Capturing traffic on a Linux-based PBX

Assuming you're running CentOS, which is the default for most PBX software, you will need to install the Wireshark package:

yum install wireshark

Run the following commands on your PBX to capture all traffic, SIP (signalling) and RTP (audio), between the PBX and your provider's server into file /root/my.cap. While the capture is running, restart your PBX software so it will attempt to register with your provider. Then attempt an outgoing phone call and an incoming phone call. Type Ctrl-C after running these tests to stop the capture.

tshark host server.provider.com -w /root/my.cap

Run the following command to compress your capture file my.cap into a compressed my.cap.gz file:

gzip /root/my.cap

From your Windows PC, run the following to copy that capture file from your PBX's IP address to your Windows PC's C: drive. You will need to have the pscp program, which is available here. You will be prompted for the PBX's root password. Replace 192.168.1.2 with the IP address of your PBX.

pscp root@192.168.1.2:/root/my.cap.gz c:\

If you have a Windows PC on the same network as your Voice over IP adapter, you can use it to capture all traffic between the adapter and your provider. Since a network switch isolates the traffic on each port, you cannot capture another device's traffic from your PC (unless it supports port mirroring). Instead, the easiest way to capture traffic on your local network is to get a cheap Ethernet hub. A hub repeats all traffic coming into one port on all of the other ports. This means any device plugged into a hub can view the traffic to/from every other device on the same hub.

Plug your hub into your router and your PC and adapter into the hub, as shown below:

Tip: A more advanced method of capturing traffic would be to plug the hub into your cable modem, so it sits between the modem and router. However, anything else you plug into the hub (like the PC running Wireshark) will be unprotected and open to all traffic from the Internet. 

Start Wireshark and click on Capture Options. This will allow us to select the proper network card to capture traffic from, the host to capture traffic to/from (your provider's gateway) and a place to save the capture file. You must run the capture in "promiscuous" mode, which means Wireshark will "look" at any traffic it sees on the network, not just traffic to/from the host it's running on (your PC).

While the capture is running, reboot your adapter so it will attempt to re-register with your provider. Also, attempt to make an incoming and outgoing call. After running these tests, click on the "Stop the running live capture" button. Your capture will already be loaded for viewing.

Reading and understanding a VoIP traffic capture

Now that you've captured the traffic between your adapter (or PBX) and your provider, you have enough information to figure out what is causing the problem. For most people, the easiest thing to do now is send the capture file to your provider. An experienced technical support agent will be able to immediately tell you what is causing the problem and how to fix it. As I mentioned in an earlier post (VoIP, no dial tone, missed calls & port forwarding), there's a 90% chance the the problem is your router.

Tip: Unless you have significant networking experience, stop here and send the capture to your provider. It will be tremendously helpful to them in troubleshooting the issue and you can save yourself a few hours of time.

If you'd like to dig into the details and see exactly what is happening, then continue on. When you open your capture file, you'll see a split-screen with each Ethernet frame on the top, a user-readable drill-down of the contents in the middle, and the raw data on the bottom.

A working SIP REGISTER conversation

Scroll down in the top panel until you reach the first REGISTER attempt from your adapter to your provider. Select the frame and in the middle window, right-click on the "Session Initiation Protocol" section and pick "Expand Subtrees". Find the line starting with "Call-ID", right-click on it, and select Apply as Filter > Selected. The Filter box above the first window pane will now show something like "sip.Call-ID == "5b1c8fdb-90c4f2a3@10.0.1.100"", meaning only the frames that contain that Call-ID will be shown. This makes it easier to filter out one conversation from a capture that may have a lot of extraneous traffic. This is what a proper REGISTER attempt will look like:

Your adapter will attempt to register, but it doesn't send any Authorization data the first time. The provider will respond with a 401 Unauthorized and will include information such as the realm, nonce, algorithm values. The adapter will then use these values, along with your SIP password, to form an encrypted response which it will use when re-registering. Since the second register attempt has Authorization information, the provider will accept the registration and reply with a 200 OK.

You can dig into each SIP request and response to see exactly how the registration conversation works. Now that you know what a working REGISTER conversation looks like, you can identify one that is broken.

A broken SIP REGISTER conversation

The most common cause for an adapter not being able to register is a router/firewall issue. If the provider's 401 Unauthorized response never makes it back to the device, it can never properly send Authorization information to the provider to register.

As you can see in the registration attempt above, the adapter never gets the 401 Unauthorized, so it keeps trying without success. If your provider was looking at a trace from their end, they'd see a 401 Unauthorized being sent, but no REGISTER with an Authorization section, clearly indicating that the 401 was being blocked by the router/firewall.

Working outgoing & incoming calls

Most providers configure their adapters to only play a dial-tone when the adapter is registered. Therefore, once you are registered, you will probably have no issues making an outbound call. Also, an outbound call is a connection initiated from your private network, so there are usually no firewall issues to deal with either.

An outgoing call is initiated by an INVITE request sent from your adapter to the provider. The provider will reply with a 100 Trying and a 407 Proxy Authentication Required. This is similar to the 401 Unauthorized during the REGISTER request, except with a Proxy-Authenticate section instead of an Authorization section. Your adapter will re-send the INVITE with your credentials. The provider will respond with a 100 Trying, 183 Session Progress (which will cause you to hear ringing), and 200 OK when the call is answered. Your adapter will acknowledge the answer and the conversation will begin. Finally, when one of the sides hangs up, a few BYE messages will be sent and acknowledged.

Tip: You may not see this exact sequence of events, depending on how your provider handles the call and if it's actually answered. A busy or invalid number will return different responses than a 200 OK.

A working incoming call looks similar to an outgoing call, with INVITEs, 100s, 183s, and 200s.

Broken outgoing & incoming call

It's rare to have an issue with outgoing calls if you have a dial-tone. However, a capture will show what's going wrong if you do.

If you're not receiving incoming calls at all, chances are you won't see anything on the capture, which is still useful information! A common problem is that you will receive incoming calls for a few minutes after your adapter is first powered on (and registers), but then your incoming calls don't ring through. If you can capture this entire sequence (register, successful incoming call, failed incoming call 15 minutes later), it will clearly demonstrate that there is a router/firewall issue.

Analyzing calls with Wireshark

Wireshark has telephony-specific features that may come in handy for troubleshooting VoIP calls. After opening your capture file, go to Telephony > VoIP Calls. Wireshark will automatically detect all of the calls in your capture.

You can then click "Prepare Filter" to easily view just the frames associated with a particular call, click "Flow" to see the conversation between your adapter and provider, or click "Player" to listen to one or both sides of the conversation in the capture.

Also, you can go to Telephony > RTP > Show All Streams. Select an RTP stream (the audio from one side of a phone call), click Find Reverse, click on Analyze. You can now see and graph statistics like how many RTP packets were lost or the max and mean jitter.

Finally, if you want to cut/paste a UDP SIP conversation into a support ticket with your provider, you can right-click on one of the frames in the conversation and select Follow UDP Stream. Change the radio button to ASCII before copying or saving the data.

Conclusion

Now that you know how to capture and analyze your Voice over IP traffic using Wireshark, you have the ability to do some troubleshooting on your own that goes beyond rebooting the adapter. However, as I've mentioned before, sometimes it's far easier and faster to let your provider take over after you've provided them with a helpful capture showing the problem.

Please don't take this as an opportunity to hassle your provider over issues that don't cause any noticeable problems in your phone calls. There will be jitter; there will be dropped RTP packets. That is the nature of voice over IP and there are numerous methods of ensuring voice quality despite these network-level problems. Your provider is focused on providing you with clear phone calls, not on optimizing the statistics generated by Wireshark.

If you're interested in the inner-workings of voice over IP and SIP, there's no better way to dig in than running a Wireshark capture and figuring out exactly what is going on.

(Portions of this article may be taken from a similar posting I wrote for trixbox.org in 2008.)

Most residential VoIP services work perfectly without any additional configuration by the end-user.  They just plug in a Linksys PAP2T-NA (or similar device, referred to as adapter going forward) that's been pre-configured by the provider, and it handles the typical home network without any issues.

However, for those users who experience no dial tone or missed incoming calls, there is always one culprit: the home router.

Why You're Using a Router

For most end-users, the purpose of having a router is simple. They are assigned a single IP address from their ISP, but they have multiple devices. In order to share that IP, you need a router to perform the network address translation (NAT) between your private network and the public network (Internet). Let's assume the adapter's private IP is 192.168.1.101 and your public IP is 222.222.222.222.

When you plug your adapter into the router, it usually gets an IP address via DHCP. For example, the adapter will get IP address 192.168.1.101.

Tip: If you have manually set a static IP on the adapter, then you must also set the default gateway, network mask and a DNS server.

Next, the adapter will use DNS to lookup the IP address associated with the VoIP gateway that has been configured. For example, if your adapter has been configured with gateway.provider.com, and a DNS server of 8.8.8.8 (Google's public DNS server), the adapter will ask 8.8.8.8 for the IP address that is associated with gateway.provider.com. Let's assume the response is 74.63.41.218.

How Your Router is Supposed to Behave

Now your adapter will attempt to register to 74.63.41.218 using a SIP REGISTER packet. The packet will originate from 192.168.1.101 (your adapter) on UDP port 5060 (you can change this) and will be destined for 74.63.41.218 (your provider's gateway) on UDP port 5060 (you can't change this unless your provider says so). Your router will receive this packet, select a random public port for this conversation and forward the packet on to your provider. To your provider, this packet will look like it's coming from 222.222.222.222 (your public IP) on a random UDP port (43211), not the private IP and port that it originated from. If everything works properly, this packet will make it from the adapter, to the router, to the Internet, to the provider and the response will make it all the way back. Your router will remember that it used UDP port 43211 for this conversation and will continue to forward anything that comes into that port to the adapter's IP address (192.168.1.101) and UDP port (5060).

Tip: You can verify if it's working by looking at the status page of your adapter and confirming that the SIP Messages Sent and SIP Messages Received counters are both increasing.

Since routers typically "clean up" connections that aren't being used, the mapping between the adapter and random UDP port 43211 will be forgotten at some point. Typically, your adapter will register and then be inactive until you make or receive a call.  Since this could be a long time, it's likely that the router will determine the connection is inactive and the port mapping will be forgotten after a few minutes. Now when your provider attempts to send you an incoming call on UDP port 43211, the router will have closed the port and your caller will hear a busy signal or be sent to voicemail. To prevent this, your adapter will most likely be configured to send a NAT Keep-Alive message periodically.  This message will keep the conversation going during periods of inactivity, prevent the router from forgetting the port mapping, and ensure incoming calls will ring.

Troublesome Routers

However, sometimes there is a NAT configuration problem and packets make it from your adapter to the provider, but they never make it back. This is either because your provider is not properly detecting your public/private IP addresses (unlikely) or because your router is blocking the replies. Some routers remember the port mapping for a very short period of time or have some advanced firewall-type features (SIP ALG) which will always cause problems.

Tip: You should always shut off any VOIP or SIP-aware functionality in your router, because it is unlikely that the router engineers have implemented 100% SIP-compatible features.

Port Forwarding

The easiest solution is to purchase a router that is known for being SIP compliant. If purchasing a new router is not an option, then you can try port forwarding. When you configure port forwarding, you are hard-coding the mapping between the private IP:port and the public IP:port. This is a common solution because most routers can properly handle port forwarding, since it's a much simpler feature to implement.

Tip: If you're going to setup port forwarding, then you should set a static IP on your adapter. Otherwise, your adapter's IP could change via DHCP, making the hard-coded port forward rule invalid.

One of the problems with port forwarding is that it will forward all traffic, from any source (trusted or not) to the adapter. Remember, without port forwarding, the router assigned a random port and kept the established connection open between the two endpoints. There is a strong possibility that someone could use this vulnerability to compromise your VoIP client, especially if it's a PBX (Asterisk, Freeswitch, Trixbox, etc).

Warning: You should NEVER configure port forwarding if you are using a software PBX (Asterisk, Freeswitch, Trixbox, etc) or place the PBX in the DMZ. Your PBX will be compromised.

Another snag you may run into with port forwarding is if you have multiple adapters. Without port forwarding, each adapter will be assigned a random port on the public IP address by the router, even if they all are listening to UDP port 5060 on their private IP addresses. However, if you decide to use port forwarding, all inbound traffic to UDP port 5060 will only redirect to one adapter. This is where things start to get messy so you'll probably want to create a diagram so you can make sure every device is configured properly. Set each line on each adapter to use a different UDP port (5060, 5061, 5062, etc) and then setup port forwarding to forward the corresponding public UDP ports (5060, 5061, 5062, etc) to the adapter that's listening on that port.

Placing your adapter in the DMZ, which is effectively forwarding all ports to the device, is not recommended. Any malicious user that has access to every port on your adapter will be able to compromise it.

Before You Spend Hours Troubleshooting

The first thing you should do before tinkering with your setup is to open a ticket with your provider. They have the ability to see exactly what is causing you to not have a dial tone or miss incoming calls. They will be able to recommend settings for your router that will ensure your service will work properly. Remember, even if you tweak your adapter and router to work with one provider, there's no guarantee it will work with another provider. Your first step should always be to contact them -- it could save you hours of time.

What's the purpose of this blog? I'm not quite sure yet.