There has been a major update that fixes a couple of minor problems and adds a few interesting new features such as the ability to mess with the TCP MSS on packets and override the DF bit. There is also a feature to randomly vary the MTU used for fragmentation to get around defective packet filter implementations that may be upstream. For now, you'll need to poke around in the source to figure out how to use the new features, though here's a fresh example showing a few of them:
shaper \
pipe 250k mtu 1500 ethernet \
queue 8681 delay 0.05 \
queue 8682 delay 0.25 \
queue 8683 delay 0.50 mtu 320 wobble 32 ignore-df \
pipe 2500k mtu 1500 ethernet \
queue 8691 delay 0.05 \
queue 8692 delay 0.25 \
queue 8693 delay 0.50 mss 280
Feel free to send any specific questions on this. ( 2005-11-13 )
There was a bug in the original version posted here where select() timeouts and bandwidth management were not getting handled correctly on higher speed pipes. The version now linked here has been fixed. ( 2003-11-21 )
I ordered voice over IP service from Vonage a few months ago after reading some positive reviews in the Broadband Reports Voice over IP Forum. This service provides local and long distance calling over any reasonable performance IP network connection from standard phones. They provide a Cisco ATA-186 adapter, which is a modem-sized box that interfaces an analog phone to an ethernet network. After you connect the box up, you get a dial tone and place calls just like normal (well, almost -- 11-digit dialling is mandatory). Their rates and services are great compared to traditional phone service providers.
So, a couple of days after ordering I got my ATA-186 in the mail and hooked up. It worked pretty good. Until my ADSL Internet service got busy with other traffic. Then the calls got choppy, like a bad mobile phone connection. Mostly it was a problem for those who I was talking to, since my upstream speed (which carries my voice out) is lower than my downstream spead (1536 kb/sec down and 288 kb/sec up).
It was obvious what the problem was: voice packets getting discarded due to congestion where they hit the relatively slow upstream ADSL link. So, it was time to do some traffic shaping to ensure that the voice traffic always got the bandwidth it needed and other services got whatever was left.
Since I use a FreeBSD server as my router for sharing the ADSL service throughout the office, I figured it would be an easy matter to setup traffic shaping using FreeBSD's dummynet facility. I would setup a pipe of 288 kb/sec with two queues defined, one for voice (which would have a high priority) and one for everything else (which would have a lower priority). The voice traffic could be determined by looking at the IP address of the ATA if I did the shaping on the inside of the NAT (Network Address Translation) that runs on the server as well.
And I was right -- it was easy to set up. Only it didn't work too well. In fact, intially it wasn't any better at all. Dropping the configured pipe speed to 200 kb/sec made it pretty good. But that noticably slowed other traffic even when the phone wasn't being used, and slowed it a lot when the phone was in use.
When I started to think about what was going on, the problem was obvious: There was a huge amount of overhead involved in the ADSL link that dummynet knew nothing about. I didn't realize how bad it was until I started tracing the flow of packets and doing some math.
Dummynet does its calculations based on the IP packet size. Which is fine until you actually send the packet somewhere. Then, the packet is going to get encapsulated somehow, depending on what kind of network you have. In my case, the network connection is ADSL, which from the server involves two encapsulations. Once to ethernet to get from the server to the ADSL modem, and then in the modem, the ethernet is bridged by encapsulating it in ATM.
Each of these encapsulations adds overhead, but it is not a fixed overhead. In fact, the rules to figure out how much are somewhat complicated. For ethernet, 18 bytes get added to each packet:
Plus -- if the resulting packet is less than 64 bytes, the data is padded to make it 64 bytes long. Dummynet knows nothing about this, so when it does its calculations on a typical 172-byte voice packet, it is off by 18 bytes. That's more than 10 percent. It would be even worse for a small packet, due to the padding that would come into play.
So that's 10% error. Next the ethernet packet is encapsulated in ATM to be sent over the phone line. ATM sends data in fixed-length cells consisting of 48 bytes of data plus 5 bytes of header. So, each ATM cell has about 5/53 or another 10 percent overhead. Plus, if even one byte of a cell is used, all 48 bytes need to be sent. That's up to 47 extra bytes per packet depending on where the length falls. And on top of that, the briding protocol adds another 14 bytes per ethernet packet when it's encapsulated in ATM.
Continuing with our 172 byte IP packet example, which is now 190 bytes due to the ethernet headers: Adding the 14 bytes of bridging overhead makes 204 bytes, which requires 5 ATM cells, since each can hold only 48 bytes (204/48 is 4 plus another 12 bytes in a partially-filled 5th cell). Five ATM cells at 53 bytes each is... 265 bytes.
So by the time it gets on the wire, our original 172 byte packet is now taking up 265 bytes! That's 35% overhead -- no wonder dummynet's calculations were so far off.
I thought about fixing dummynet to be smarter about bandwidth calculations, but I really didn't want to fool around in the operating system kernel. If I did that, I'd either need to package up the changes and get them added to FreeBSD permanently, or I'd need to patch the operating system up myself every time I upgraded.
However, on FreeBSD, there's a facility called a Divert Socket, which allows a user process to read and write packets processed by the operating system's firewall rules. This is used in the standard FreeBSD address translation daemon, natd.
So I decided to write a traffic shaper as a user process, like natd, and make it know how to properly calculate encapsulation overhead. And while I was at it, I would make it work with strict prioritization, instead of weighted queueing like dummynet uses.
I could then install some simple firewall rules to send packets through my program, which would queue and discard as needed to maintain the proper traffic flow. Since the traffic shaper would manipulate the demand to exactly match the speed the ADSL line can take the packets, no packets would ever get discarded anywhere except in the traffic shaper program.
The resulting program is currently called simply 'shaper'. It's a rather compact, but fairly elegant piece of code all in one source file, shaper.c.
Shaper accepts a configuration on it's command line that defines one or more 'pipes' each with one or more 'queues'. These terms are used as they are in dummynet: A pipe is an abstraction that removes data from queues at a rate that corresponds to a certain bandwidth. A queue just that; a queue of packets waiting transmission through a pipe.
Pipes have a few characteristics:
Queues also have some characteristics:
As an example, my shaper command line looks like this (this is effectively a single command line, though I place it on more than one physical line for readability):
shaper \
pipe 288k mtu 550 ethernet atm \
queue 8671 delay 0.1 \
queue 8672 delay 0.1 \
pipe 1280k mtu 1500 ethernet atm \
queue 8681 delay 0.1 \
queue 8682 delay 0.1This starts the traffic shaper and sets up two pipes, both of which model a line that carries ethernet encapsulated in ATM. The first pipe is for the upstream (outbound) side of the line, which has a bandwidth (ATM) of 288 kb/sec and an MTU of 550 bytes. The second pipe is for the downstread (inbound) side of the line. This has a physical bandwidth of 1536 kb/sec, however, I shape the traffic to 1280 kb/sec. This helps ensure that any discards that need to take place take place in the traffic shaper, which can make the decision intelligently.
The first pipe has two queues feeding it, listening on ports 8671 and 8672. Each queue has a maximum delay of 0.1 seconds, meaning up to that amount of data can be queued before it it discarded. The second pipe is similar, just with different port numbers.
The queues are fed with firewall rules that look like this (I have other rules before, after, and between these, but they are not important to this discussion):
15000 divert 8671 ip from 192.168.231.230 to any out xmit ed0 15010 divert 8672 ip from not 192.168.231.230 to any out xmit ed0 ... 15050 divert 8668 ip from not 209.54.72.175 to any via ed0 ... 15080 divert 8681 ip from any to 192.168.231.230 in recv ed0 15090 divert 8682 ip from any to not 192.168.231.230 in recv ed0
The first two rules handle outbound packets, sending voice packets (identified by the IP address of the ATA, 192.168.231.230) to the first queue, and all other packets to the second queue. Queues are prioritized by the order they are defined when the daemon is started, so voice packets get priority over other packets since they are sent to the first queue. After processing in the shaper daemon, packets are reinserted in the firewall flow after the rule that originally diverted them. In this case, it's important that the two rules never match the same packet, otherwise that packet would go through both queues.
Download the source code, shaper.c.
Compile it with "cc -o shaper shaper.c" and install somewhere appropriate like /usr/local/bin or /usr/local/sbin.
Start the daemon at boot time out of rc.local and install appropriate firewall rules to match. See the examples above for syntax.
Due to overflow limitations with 32-bit integers used internally, the bandwidth of a pipe is limited to 2048 kb/sec and the maximum delay of a queue is limited to one second. These are pretty reasonable limits for instances where you would do this kind of traffic shaping
There are a few things that I've thought of that would be nice improvements to the traffic shaper, but are unnecessary for my application, so I haven't coded them yet:
If you'd like to work on any of these and would like my thoughts or if you'd like me to work on them for you (I might!) let me know.
If you have questions, comments, or requests on this software, please let me know. I can be emailed at david@madole.net.
I would also enjoy hearing from you if you are using this software or use another traffic shaping solution, especially with voice over IP.
|
Web hosting by Omd3.com Hosting Services |
David S. Madole 2003-11-02 0430 GMT |