A little something I've been working on...

NathanE · 7 Feb 2006 at 19:53

For about a year I've been working on a network bandwidth monitor program - a sort of "DU Meter" on steroids.

It's built in the shape of "client-server". Basically you install the "server", or what I like to call "sensor", on all the PCs in a network. Then using the client program (the GUI) the administrator can connect to each sensor in the network and view bandwidth statistics of each protocol.

I did a lot of work with the packet analysis, ugh, "engine", to make it track TCP sessions. This means, with the aid of deep packet inspection, that it can track BitTorrent traffic regardless of what port number the user has it configured to. Obviously it's been designed in such a way that I can add other special protocols like BitTorrent which don't use any standard port number.

The packet analysis "engine" is fully multi threaded using NT kernel's completion port threading model. A lot of work went into making the code as concurrent as possible. Multi-core CPUs ahoy

Client and Sensor communicate via a UDP connection using an XML schema as the protocol.

Client is C#.NET. Sensor is VC++ w/ Boost.

Does anybody think I could sell this software once it's finished?

Comments welcome.

Thanks for reading.

Beansprout · 7 Feb 2006 at 21:17

Cool - a kind of cross between Ethereal, Munin (sort-of) and Netlimiter.

'tis neat

I couldn't tell you if there were competing products though as it's not my market, but sure - put a site up and advertise it, see how you do :cool:

MNuTz · 7 Feb 2006 at 21:48

i was wondering if it would allow you to limit the amount of bandwidth for each PC?

I think its a great idea.

Especially if it has a stealth mode that i could use on everyones PCs that would allow me to monitor it without them knowing it was running!

If you need anyone to test it for id be happy to, as long as i could use it on my CV

NathanE · 7 Feb 2006 at 22:07

Beansprout said:
<snip>

Ta for the link, never seen that before.

Just found this web page http://www.generalconcepts.com/resources/monitoring/

There seems to be quite a few OSS solutions. But they all seem to be quite "Unix'ie" and not user friendly. Distinct lack of commercial software though... :eek:

MNuTz said:
<snip>

Well I made a design decision at an early stage that the program would not affect the latency of the network in any way, so unfortunately this design means it cannot perform traffic shaping. Sorry!

The finished Sensor will just run as a regular Windows service (e.g. Start->Run->"services.msc"). Not sure if that counts as "stealth mode" to you?

I'll keep the testing in mind. There's still quite a bit to be done yet before it's ready for public consumption

For instance, I need to wrap the Sensor up as a Windows service. At the moment it's just a console window

Ta for comments guys!

tolien · 7 Feb 2006 at 22:13

Sounds extremely interesting, but how much load does it put on the machines at either end (since most layer 7 filtering stuff I've seen adds a fairish bit)?

NathanE · 7 Feb 2006 at 23:30

You mean the load of the Sensor? Well I am yet to see it go above 1% CPU

, and that's on my crappy 1.8GHz P4 with numerous BitTorrent seeds going. Memory usage is also very very small (but obviously proportional to amount of connections that are open.) I implemented the TCP session tracking using an AVL binary tree so it is just about as efficient you can get.

The most costly thing to the Sensor is when a new session (e.g. a TCP connection) is opened. Once the session is setup, packets flowing on that session (even 1500 byte packets) don't really have any impact thanks to the completion port that minimises context switches and balances processing across available CPUs.

tolien · 7 Feb 2006 at 23:40

NathanE said:
You mean the load of the Sensor?

The bit that does the packet inspection

Well I am yet to see it go above 1% CPU , and that's on my crappy 1.8GHz P4 with numerous BitTorrent seeds going. Memory usage is also very very small (but obviously proportional to amount of connections that are open.)

Nice. Certainly sounds like it would give DUMeter et al a hell of a run for their money.

Visage · 8 Feb 2006 at 09:12

Would the same functionality not be available at the router level through SNMP?

Nix · 8 Feb 2006 at 09:26

Nice, looks like a lot of work went into that.

NathanE · 8 Feb 2006 at 13:33

Visage said:
Would the same functionality not be available at the router level through SNMP?

Possibly on really high-end corporate routers but not on anything us mere mortals can get ahold of

Ta for comments guys

tolien · 8 Feb 2006 at 14:01

NathanE said:
Possibly on really high-end corporate routers but not on anything us mere mortals can get ahold of

The 837 certainly won't expose stats for level7 packet inspection out through SNMP.
Could do it with an IPCop box (and some patches) though.

Veritas · 8 Feb 2006 at 14:42

Wow, that looks pretty cool.

Good luck if you do release it. I can imagine quite a few people would have a use for an application like this

...just wished I'd thought of it first...

NathanE · 9 Feb 2006 at 20:09

I've improved the GUI quite a bit. Finished the "Sensor Browser" that you caught a glimpse of on the previous screenshot. Also added more columns to the Protocol report. Also spent some time wrestling with .NET to let me hide and reorder ListView columns (had to resort to Win32 Interop in the end

)

Here's a 'collage' screenshot showing a few of the new UI features:

I also wrote a quick'n'dirty "alternative GUI" in (dare I say it) PHP. I did it as I wanted to make sure the XML schema could be read and used by other parsers/languages. It can. I think this is a pretty cool "feature" for zero time cost to me.

One thing I'm mulling over at the moment is using a UDP broadcast packet to detect all the Sensors on a subnet and then adding them to the Sensor Browser, automatically. Would be a neat "quick setup" feature, no?

PS: I'm thinking of releasing it as freeware at first. Then later on introducing a sort of "Pro" version

I don't know, it just seems these days you don't get anywhere unless you also have a free version in some shape or form. I guess, really, it's a marketing tool.

tolien said:
The 837 certainly won't expose stats for level7 packet inspection out through SNMP

Fair enough

Peve · 9 Feb 2006 at 22:27

Very impressive I don’t know much about this sort of stuff but it seems really good. Good Luck in the future!

McDaniel · 9 Feb 2006 at 22:54

lets hope it gets you a good mark on your project eh Nathan!

robjf · 9 Feb 2006 at 23:11

Maybe for the pay version you could implement bandwidth restrictions, i woud deffo pay for it

Beansprout · 9 Feb 2006 at 23:13

Lookin' great

MNuTz · 10 Feb 2006 at 14:38

robjf said:
Maybe for the pay version you could implement bandwidth restrictions, i woud deffo pay for it

it may be easier for him to implement plugins and then charge for them.

NathanE · 14 Feb 2006 at 22:38

Little update...

I've wrapped the sensor up into a Windows Service now. This can be controlled from the standard Windows MMC or through my GUI itself by clicking the Start/Stop/Pause buttons on the toolbar.

1. Added a subnet self-discovery function to the "Sensor Browser" pane. This places all the active sensors in the network subnet under its own folder on the browser.
2. Redesigned the ListView refreshing mechanisms under-the-hood to use what is called a "Virtual ListView" in .NET. They now update instantly, have zero flicker and don't lag the GUI.
3. Added two new protocols, Shoutcast and Secure Shell (SSH). Both of these are identified by deep packet inspection.
4. Added a new column to Interface List that shows how many sessions (e.g. TCP connections) are being tracked by the sensor.

I may release a beta soon

robjf said:
Maybe for the pay version you could implement bandwidth restrictions

Do you mean QoS/bandwidth throttling, or allocating a bandwidth quota/allowance and then once it's run out - it prevents further traffic occuring? The former can't be done on my system, as I was explaining earlier. The latter however could be done, but not in version 1

Somebody also suggested a function to disable Internet connectivity completely. I could do that in a later version.

MNuTz said:
it may be easier for him to implement plugins and then charge for them.

Plugins would be neat but it would hurt the performance of the system. I don't want to sacrifice performance

There's plenty of other things I could do for a licensed version - e.g. real-time graphs, reports, bandwidth logs/history etc. Or even just allowing the GUI to connect to non-localhost or more than 1-2 sensors?

Jotun · 14 Feb 2006 at 23:06

NathanE said:
I did a lot of work with the packet analysis, ugh, "engine", to make it track TCP sessions. This means, with the aid of deep packet inspection, that it can track BitTorrent traffic regardless of what port number the user has it configured to.

Can it track bittorrent data that has been encrypted? I know BitComet has a protocol header encrypt option, as I use it because my "Unlimited" ISP uses packet shaping, and also from what i've been reading, more and more clients are planning to implement encryption.... Assuming you've not already accounted for this anyway