home  wiki



This discussion is driving changes we are making to the frottle application. There have been three major things we wanted to address:
  1. Support for multiple interfaces on a machine
  2. Expose magic cookies in application as configuration parameters
  3. Introduce the concept of control domains

This has led to a number of other enhancements, the most significant being the desire to pass a control token from master to master. This was considered necessary to utilize frottle effectively in a mesh type network.

Multiple Interfaces

Nodes that form part of a routing backbone have multiple interfaces. As there is only one QUEUE target we needed to be able to define additional interfaces and split the incoming traffic into the appropriate set of queues for the outgoing interface.
A Node may be the master of one domain on one interface and a client in a second domain via a second interface.

Flexible configuration

The intial thought was we would be running multiple copies of frottle ona multi-interface machine. We wanted to have a way to simplify the configuration. This led to adding a number of command line args, pre- and post- script callouts and the externalization of some hard coded parameters.

Control Domains

It became clear as soon as we started looking at multiple interfaces that we needed to support the concept of a control domain. These were added and clients and masters specified in terms of the domain they belong to.

Control Token

This is new work to better support the use of frottle in a mesh environment. The problem faced here is that a mesh uses the same frequency and frottle sucks at controlling the traffic from a node that is not directly connected. A multi master approach ( as used in a routed infrastructure ) still has problems due to the single frequency used. We are now looking at using a co-master approach where masters pass control to the next master in the chain. The chain is a ring that supports splitting and healing.


Frottle Config File

DanFlett: Here's how I'd like to be able configure a multi-interface version of Frottle...

proposed frottle.conf
daemon 1
verbose 1
logfile /var/log/frottle.log
clientstatsfile /var/www/html/frottle/client.html
masterinfofile /var/www/html/frottle/master.html
client        wlan0    22,53           999     100
client        eth0     22,53,5001      -       100
client        eth2     -               1001
# CLIENTMODE #MODE-MASTER INTERFACE (SELF/CLIENT/NONE) TIMEOUT POLLPARAMS master eth1 self 100 60000,10,6000,7,5000,5,4000 master wlan1 - 200

You get the idea - leaving a paramater blank (if there are no parameters to be specified after it) or specifying '-' (if there are parameters to be specified after it) for a parameter sets it to default - where appropriate. This is similar to the way External linkShorewall sets out its External linkconfig files.

MW frottle patch

OK, here it is, everything you asked for and a little bit more. The major change from above is the introduction of domains. Define the domains and associate the master / client instance to a domain.


Note: These links were broken - I can find the ipk files in the files/frottle section but I can't find MwPre0.7 anywhere --DanFlett

This is beta code, I have tested the patch on Minitar MNWAPB (RTL8181 mips ) and openwrt(BROADCOM mipsel).

Change log

MwPre0.3 is now built with iptables 1.3.1. and staticly linked (no need for libpthread.o)

MwPre0.4 adds manual override of RF Rate setting in client for hosts that do not support /proc/net/wireless

MwPre0.5 adds configuration of rate bands for queues ( used in conjunction with pollparams ) and supports calling an external command to get rate

MwPre0.6 adds the priority client option ( allowing the specification of a bigger slice for a know high priority machine- such as an internet gateway or a trunk ) and adds multiple sets of Queues to finish of the multi-domain work.

MwPre0.7 has a primitive token passing mechanism for passing off control from master to master. Masters and Domains need to be defined in the .conf file.

openwrt version

You can now install the QUEUE module by itself as a package. Just do this:
ipkg install kmod-ipt-queue

insmod ip-queue

you should be ready to go.

Just a question: does the OpenWRT version need the iptable_filter.o module? I can't find it anywhere, so I'm guessing you don't need it. But the examples given on the Frottle homepage have it there.

README for the patch

This patch contains a number of small changes to frottle 0.2.1

To install:

   apply patch from inside the frottle directory
   patch -p1 < path-to-patch/frottle-0.2.1-mwPre0.1.diff

The changes were done in order to make configuring frottle easier in a
multi-interface environment.

1. New command line arguments
 frottle master | client <interface> <port>
--mode master | client This argument is used to select an alternative configuration file: master /etc/frottle-master.conf client /etc/fottle-client.conf Mode parameters still need to be set in the configuration file. --interface <interface> This argument allows the interface specified in the configuration file to be overridden.
--port <port> This argument allows the Master port specified in the configuration file to be overriden. Care should be used with this argument as only a correct combination of Master IP addr and Master port will allow a client to connect to a master instance.

2. Support for re-named binary.

If the frottle binary is launched using the name frottlemaster
either renamed, hard or soft linked then it will default to -mode master

3. New configuration file options

#pidfile /var/run/frottle.pid

If this option is defined the process ID will be written to the file specified.
No checking for multiple instance is done at this stage. If the file already
exists then the pid will be added to the file. On shutdown the file is unlinked
(this will be a problem in the case of multiple instances).
pre and post scripts are supported for Master and Client configurations.
The specified script will be called with the following arguments:
 scriptname interface masterport

#Master pre-script
#masterprescript /usr/local/frottle_master_pre.sh

#Client pre-script
#clientprescript /usr/local/frottle_client_pre.sh

#Master post-script
#masterpostscript /usr/local/frottle_master_post.sh

#Client post-script
#clientpostscript /usr/local/frottle_client_post.sh

#Client re-register timeout
#clientreregister 10

The client retry time was hard coded. this is now configurable. If the client
does not hear from the master after this time it assumes it has been dropped
and will try to re-connect. 10 was the hard coded default, may be a little high.

#Client timeout
#clienttimeout 60

The client will timeout if nothing is hear from the master after this time.
It will fall out of the frottle configuration. This was hard coded and
is now configurable.

#Client link speed
#clientlink 0

In some cases the IOCTL that is used to return the link speed will fail in
the client. This will leave the client at the fastest linkspeed default. This
may not be desirable. Setting clientlink to non zero ( set to the link rate )
it will override the internal default. Use this to reduce the TX window in
clients on a slow link to prevent them hogging the available time.

4. Bug fixes

Bandwidth allocation was testing for link speed >=5 else ==2 else other.
This was changed to >=5 else >=2 else other.

Errors in the management of the client concetion state meant that once a
conncetion was timed out it would repeatedly go through flushing the queued
packets when this was not necessary.

5. Known Issues
Need some more work on the thread join for multiple interface support. This is
only on shutdown and sems to work Ok.

Config file parsing logic has a error where it is looking for blank lines.
This is cosmetic but should be fixed at some stage.

The link speed setting from the wireless interface divides the value returned
from the IOCTL cal by 1000000. It may be it should be divided by 10,000,000.
As it stands if the value returned is 54,000,000 then it is going to be 54.
In the code the time slice is allocated according to >=5, >=2, <2. As it
stands all links will be in the top allocation category. ( this needs checking)

Netfilter interaction

Frottle uses the iptables QUEUE target to delay outgoing traffic. The QUEUE target was designed to allow userspace applications, like Frottle, to inspect and control network traffic.

The QUEUE target is explained here:
External linkhttp://www.netfilter.org/documentation/HOWTO//packet-filtering-HOWTO-7.html#ss7.4

Here is the relevant exerpt:

QUEUE is a special target, which queues the packet for userspace processing. For this to be useful, two further components are required:

  • a "queue handler", which deals with the actual mechanics of passing packets between the kernel and userspace; and
  • a userspace application to receive, possibly manipulate, and issue verdicts on packets.

The standard queue handler for IPv4 iptables is the ip_queue module, which is distributed with the kernel and marked as experimental.

The following is a quick example of how to use iptables to queue packets for userspace processing:

# modprobe iptable_filter
# modprobe ip_queue
# iptables -A OUTPUT -p icmp -j QUEUE

With this rule, locally generated outgoing ICMP packets (as created with, say, ping) are passed to the ip_queue module, which then attempts to deliver the packets to a userspace application. If no userspace application is waiting, the packets are dropped.

To write a userspace application, use the libipq API. This is distributed with iptables. Example code may be found in the testsuite tools (e.g. redirect.c) in CVS.

The status of ip_queue may be checked via:


The maximum length of the queue (i.e. the number packets delivered to userspace with no verdict issued back) may be controlled via:


The default value for the maximum queue length is 1024. Once this limit is reached, new packets will be dropped until the length of the queue falls below the limit again. Nice protocols such as TCP interpret dropped packets as congestion, and will hopefully back off when the queue fills up. However, it may take some experimenting to determine an ideal maximum queue length for a given situation if the default value is too small.

Presently, Frottle is only capable of acting on one network interface at a time on the host on which it is running. To be part of a truly scalable network, it would be desireable for Frottle to act on any and all of a hosts' wireless interfaces. For instance, a host may have one interface which is acting as a Access Point, and two more interfaces making client connections to two remote APs. It would be desireable for Frottle to be able to simultaneously act as a Master on the AP interface and as a Client on each of the client interfaces.

As mentioned in the exeprt above, the QUEUE target requires:
a userspace application to receive, possibly manipulate, and issue verdicts on packets.

Frottle does this, but currently it is able to manipulate the transmit order of packets in the QUEUE based on port number only. This is the controlled by the "HIPORT" paramater in the Frottle config file.

For Frottle to be able to act on different interfaces, it needs to be able to manipulate packets in the QUEUE based on their destination IP or Ethernet MAC address also. Packets being sent to one interface will be within a certain IP address range, packets being sent to another interface will be from a different IP address range.

LIBIPQ is the API that allows userspace applications to inspect an manipulate the packets in the QUEUE. Doing a search on Google turns up some info:
External linkGoogle Search - libipq
External linkGoogle Search - "libipq" address
A good place to start is here:
External linkQuick Intro to libipq

From reading the various pages out there on LIBIPQ, it appears it can read a field called "hw_addr" from the "ipq_packet_msg" record. This is the source MAC address of the packet. This is useful to us, as it tells us out which interface the packet is destined to leave the host. It is unclear at this point wether or not packets sent to the QUEUE from the FORWARD and OUTPUT chains will have "hw_addr" set (it looks like they don't). Perhaps QUEUEing packets from the POSTROUTING chain would work better (if this is allowed).

OK, we don't need "hw_addr", iptables MARK, routing table lookups or anything like that. The name of the outgoing is contained in the packet. LIBIPQ can read it from packets in the QUEUE.
From External linkSuperHac.com:

ipq_packet_msg structure
Just a dump of the ipq_packet_msg structure.

ipq_packet_msg structure defined in /usr/include/linux/netfilter_ipv4/ip_queue.h:

typedef struct ipq_packet_msg {
   unsigned long packet_id; /* ID of queued packet */
   unsigned long mark; /* Netfilter mark value */
   long timestamp_sec; /* Packet arrival time (seconds) */
   long timestamp_usec; /* Packet arrvial time (+useconds) */
   unsigned int hook; /* Netfilter hook we rode in on */
   char indev_nameIFNAMSIZ; /* Name of incoming interface */

char outdev_nameIFNAMSIZ; /* Name of outgoing interface */
   unsigned short hw_protocol; /* Hardware protocol (network order) */
   unsigned short hw_type; /* Hardware type */
   unsigned char hw_addrlen; /* Hardware address length */
   unsigned char hw_addr8; /* Hardware address */
   size_t data_len; /* Length of packet data */
   unsigned char payload0; /* Optional packet data */

} ipq_packet_msg_t;

"outdev_name" is what we're after. We can use this to put packets into subqueues (inside Frottle, in userspace) based on outgoing interface name.

Ongoing discussion

Dan's Testing

02 Feb 2006
Here's my config file for the new Node GHO WRT54G
root@nodegho:/etc# cat frottle.conf
  1. frottle.conf v0.2
  1. Melbourne Wireless patch Pre0.5
  2. See README.mwPre0.5 for details
  3. Source patch available
  4. http://www.melbournewireless.org.au/wiki/?FrottleDevDiscussion

  1. Defaults/ examples shown commented out
  1. By default, frottle expects to find this file as /etc/frottle.conf
  2. If started with -mode master it will look for /etc/frottle-master.conf
  3. If started with -mode client it will look for /etc/frottle-client.conf
  1. This file _will_ need editing for frottle to work.

  1. Multiple interface

  1. This section describes how to configure frottle to support multiple
  2. interfaces. The configuration of each interface is through defining one
  3. or more frottle domains, then a client or master line for each interface

domain north 999
domain south 999
domain centre 999

#client wlan0 test1 22,53 100
#client eth0 test2 22,53,5001 100
#client eth2 test3 -

master vlan0 north self 100 60000,10,6000,7,5000,5,4000
master vlan1 south self 100 60000,10,6000,7,5000,5,4000
master vlan2 centre self 100 60000,10,6000,7,5000,5,4000

  1. Original config method

  1. Act as a master
mastermode 1

  1. Act as a client
  2. Note a master can run a client this way but best to use selfclient option
#clientmode 0

  1. Run as a daemon
  2. (If you use this, use 'killall -TERM frottle' to stop frottle)
  3. or configure the pidfile option
#daemon 0

  1. Verbose messages
verbose 1

  1. UDP port to use for control messages
masterport 999

  1. File to write process id to
pidfile /var/run/frottle.pid

#### Client only settings ####

  1. Masters IP

  1. Max packets in each of the queues
#queuesize 100

  1. High priority tcp/udp ports (max. 10)
#hiports 22,53

  1. Wireless interface, for gathering stats from wireless tools
#winterface eth2

  1. Client timeout to re-try connection. If client does not hear from master
  2. in this many seconds it will send a new register packet.
  3. (seconds / default 10)
#clientretry 10

  1. Client lost comms timeout. If client does not hear form master in this
  2. many seconds it will flush queued packets and bypass queue until master
  3. responds. (seconds / default 60)
#clienttimeout 60

  1. Client override link speed. If wireless stats are not available
  2. (check /proc/net/wireless) then a clients link speed can be manually set.
  3. Value is Mbits/sec
#clientlink 54

  1. Client alternative rate command. If there is an alternative way to get a
  2. rate then this parameter can be set to a script that returns the rate value
  3. (in Mbits/sec)
  4. example for wrt54g : return ` wl rate | cut -d " " -f 3 `
  5. Note: clientlink takes precedence over ratecommand
#ratecommand /etc/rate.sh

  1. Optional stats file path/name (html format)

  1. Client pre-script
#clientprescript /usr/local/frottle_client_pre.sh

  1. Client post-script
#clientpostscript /usr/local/frottle_client_post.sh

#### Master only settings ####

  1. Act as a client to ourselves
  2. This gives the master much more bandwidth than 'clientmode 1'
  3. (which is also valid to do at the master)
#selfclient 0

  1. The time the master waits for clients to respond that
  2. they have completed their poll sequence (in mS).
#timeout 100

  1. Poll parameters (too complex to explain here, see 'man frottle')
#pollparams 60000,10,6000,7,5000,5,4000

  1. Override High queue min rate setting ( default 5 )
#ratequeuehigh 5

  1. Override Medium queue min rate seting ( default 2)
#ratequeuemedium 2

  1. Info file path/name (html format)
infofile /www/frottle.html

  1. Master pre-script
masterprescript /usr/local/frottle_master_pre.sh

  1. Master post-script
#masterpostscript /usr/local/frottle_master_post.sh

  1. High priority clients
  2. A client that should get a bigger slice of the pie
  3. IP ADDR Multi Comment
#priorityclient 2 Internet gateway
#priorityclient 4 Concentrator backbone

Here's my Master pre-script

root@nodegho:/etc# cat /usr/local/frottle_master_pre.sh

#debugging the paramater passing
echo 1 $1 2 $2 3 $3 4 $4

#allow control packets
iptables -A INPUT -p UDP --dport 999 -j ACCEPT
iptables -A OUTPUT -p UDP --sport 999 -j ACCEPT

#queue packets on these interfaces (that masters are running on)
for iface in vlan0 vlan1 vlan2; do
       iptables -t nat -A POSTROUTING -p ALL -o $iface -j QUEUE

When I run it I get the following error:

Exceeded the max number of Master configs (3)
Exceeded the max number of Master configs (3)
Exceeded the max number of Master configs (3)
Frottle: Args done, config flag is: 0x29

Frottle: Acting as a master
Frottle: Acting stupid
Frottle: Executing master pre-script: /usr/local/frottle_master_pre.sh
1 eth1 2 999 3 4
Frottle: Saving pid to file: /var/run/frottle.pid
Frottle: Master running on port 999 for domain[ default ]

Here are the problems I'm having:
I had tested that stuff but I may have introduced a couple of bugs. Looks like I hard coded the number of domains (3) and masters (4). It seems to choke on that in the arg processing and then further down uses the default config - thats wrong, it should identify the domain you are using. Think I only ever tested with 2 - Ok - so if you only define two Domains it adds the interfaces correctly. Looks like I stomp on the maxMasters when I malloc extra space for the domain queus of the third domain. There's a bug here
Oh, you probably want to have a different master port for each domain, as it is each master instance will get messages for all the instances which will cause some confusion. Each Master is running in a separate thread and listening to the port number, it won't be able to descriminate which interface the UDP packet came in on.

The domain is a structure that is used to hold some parameters that will be shared across master / client definitions. When you define a Master or a Client you specify the domain and it pulls in the parameters that are configured under that domain.
The real trickery is in the multiple configurations. As the config details are parsed it creates a thread for each unique Master and Client, Its like running multiple isolated frottle instances. But there isonly one Queue device so the parent thread handles incoming packets and places them in the appropriate queue structures for the Master / Client threads to dispatch.

That's probably a bug, though from the way it choked on your config it could be because it just dumped all the Master configs because it exceeded the hardcoded limit. Try configuring only two masters and see what happens - I'll check anyway.

Should be able to do this

Ah, sorry, I was lazy. There was a bug in the patch Makefile and I never went back and changed it, I tweaked it each time I upped the version number

Yep, will add it.
You probably haven't noticed yet but when you run a Master as a self client there is no client page or details for the machine itself. Yhael asked about fixing that but it's not an easy change. When the master runs as a self client there is no separate client thread, the master just gives itself a slice of the bandwidth every time it cycles through the registered clients. I work arround it by not running self client but configuring the client for the master as well

Dan's Hastily Scribbled Testing Notes

yep - I assume you want the same parameters passed in?

It shuldn't really. I have had a lot of problems with adding the multi interface and not breaking the original definitions. There are some places where I havent fully dual pathed the code so it falls back to a parameter that is defined using the original single-mode parameters. I'm going to change this so the single mode parameters are put in a domain "default" and then the code gets cleaner again.

I build on one platform and when it is mostly right I generate the patch , I don't want to do that untilthe major bugs are squished. But as soon as possible I'll generate the patch.

You should be able to specify this with winterface in the config file. I'll remove the default and generate an error if the interface is not specified

not sure will check the code
I knew you would ask for this. It depends on how the incoming messages are presented. If the thread only receives messages on it's address or if it receives all mesages that come in on that port. Usually in userspace listening to a port will get messages from all interfaces, that would be a real problem for frottle control messages - the extra interrupts to process and discard the unwanted ones would introduce too much overhead

I can add the dependancy in the ipkg description. That way it will bring in the kmod-ipt-queue that it finds in the installed ipkg list
This is the URL
I don't think ipkg checks dependencies from packages installed directly when the full URL is specified, but it's a nice thing to include anyway.

There's probably more than one !!

Show-stopping Bug in MwPre0.8
This occurs in both single and multi-modes on client and server
Frottle starts normally
Frottle: Args done, config flag is: 0xa

Frottle: Acting as a client to with 3 packet queues[ 100 ]
Frottle: Executing client pre-script: /usr/local/frottle_client_pre.sh eth1
1 eth1 2 999 3 4
Frottle: Saving pid to file: /var/run/frottle.pid
Frottle: Entering mainloop
Frottle: Client running on if[ vlan0 ] for domain[ default ]
Frottle: IOCTL SIOCGIWRATE failed, using defaults
Frottle: RF rate: 5 signal: -28 noise -93
Frottle: Client sent registration
Frottle: Client registered

It pauses, then as soon as any traffic is sent
Frottle: Error reading message from IPQ (Timeout)
Frottle: Error reading message from IPQ (Timeout)
Frottle: Error reading message from IPQ (Timeout)
Frottle: Error reading message from IPQ (Timeout)
Frottle: Error reading message from IPQ (Timeout)
Frottle: Undefined message type from IPQ
Frottle: Error reading message from IPQ (Timeout)
Frottle: Error reading message from IPQ (Timeout)
Frottle: Error reading message from IPQ (Timeout)
Frottle: Error reading message from IPQ (Timeout)
Frottle: Error reading message from IPQ (Timeout)
Frottle: Error reading message from IPQ (Timeout)

It fills the screen with these messages. I've got ip_queue loaded, and I'm pretty sure I've got all my pre-scripts working properly. I happens on the master and to the first client to connect to the master when the first traffic is sent.

This error indiates a return code of -1 from ipq_read. I suspect this is a version squew between what I am compiling with and the shared lib on your machine. In the past a timeout would have a return code of 0i and was used to do background processing

I'm using http://downloads.openwrt.org/whiterussian/packages/kmod-ipt-queue_2.4.30-brcm-2_mipsel.ipk from OpenWRT


echo iface $iface port $port

#/bin/echo "0" > /proc/sys/net/ipv4/conf/all/accept_redirects
insmod ip_queue

#allow control packets
iptables -A INPUT -p UDP -i $iface --dport $port -j ACCEPT
iptables -A OUTPUT -p UDP -o $iface --sport $port -j ACCEPT

#queue packets on these interfaces (that masters are running on)
iptables -A OUTPUT -p ALL -o $iface -j QUEUE
iptables -A FORWARD -p ALL -o $iface -j QUEUE


echo iface $iface port $port

#/bin/echo "1" > /proc/sys/net/ipv4/conf/all/accept_redirects
#rmmod ip_queue
iptables -D INPUT -p UDP -i $iface --dport $port -j ACCEPT iptables -D OUTPUT -p UDP -o $iface --sport $port -j ACCEPT

iptables -D OUTPUT -p ALL -o $iface -j QUEUE
iptables -D FORWARD -p ALL -o $iface -j QUEUE


echo iface $iface port $port
#/bin/echo "0" > /proc/sys/net/ipv4/conf/all/accept_redirects insmod ip_queue

#allow control packets
iptables -A INPUT -p UDP -i $iface --sport $port -j ACCEPT
iptables -A OUTPUT -p UDP -o $iface --dport $port -j ACCEPT
#queue packets on these interfaces (that clients are running on) iptables -A OUTPUT -p ALL -o $iface -j QUEUE iptables -A FORWARD -p ALL -o $iface -j QUEUE


echo iface $iface port $port

#/bin/echo "1" > /proc/sys/net/ipv4/conf/all/accept_redirects
#rmmod ip_queue
iptables -D INPUT -p UDP -i $iface --sport $port -j ACCEPT iptables -D OUTPUT -p UDP -o $iface --dport $port -j ACCEPT

iptables -D OUTPUT -p ALL -o $iface -j QUEUE
iptables -D FORWARD -p ALL -o $iface -j QUEUE

Back to MelbWirelessRouterProject

Version 22 (current) modified Tue, 03 Jul 2007 23:12:05 +1000 by Dan
[EditText] [Spelling] [Current] [Raw] [Code] [Diff] [Subscribe] [VersionHistory] [Revert] [Delete] [RecentChanges]
> home> about> events> files> members> maps> wiki board   > home   > categories   > search   > changes   > formatting   > extras> site map


 Remember me.

> forgotten password?
> register?
currently 0 users online
Node Statistics