Advice for c# winforms dev

[DanF]

I've been developing with c# for years now but it's all been web based so I have almost zero experience with winforms setup. I'm wanting to make up a pretty basic web crawler that can crawl over our website looking for any issue.

I've played around with the main crawling and parsing code in linqpad but I need to get it in to an actual app now, it'll be a fairly basic xaml .net 4 app.

Is there anywhere that gives good advice on which way to do threading/shared collections in winforms? I've looked around and there seems to be loads of ways now. My thoughts where to have part of the code do the actual scraping putting the response in to a shared container, then another process can parse this and put the errors and new links back in to other shared containers. The part that does the actual scraping will need to have some form of slicing/threading as well to get performance out of it.

 

[Haircut]

First of all, XAML != Winforms.
If you're doing XAML stuff I assume it's WPF you're looking at?

In terms of your requirement, it sounds as though Reactive Extensions (Rx) or TPL could be a good fit for what you're trying to do.
Neither of them are specific to Winforms or WPF though and I wouldn't have thought that stuff would be much different to how you'd do it in ASP.NET.

If using WPF you should be aware of how the dispatcher works for actually displaying stuff on the screen: http://msdn.microsoft.com/en-us/magazine/cc163328.aspx

 

[DanF]

Cheers this is what I was looking for, I knew I didn't want to block the main UI threads yeah it'll be a WPF app where it'd be nice to have counters updating showing how far it's got.

For the shared containers would ConcurrentBag and ConcurrentDictionary be good fits?
ConcurrentBag to hold url's to parse in FIFI style, the other to hold the response against the url for further processing.

 

[Haircut]

To be honest it doesn't sound as though you need ConcurrentBag or ConcurrentDictionary to store the requests & responses.
The bit that takes the time I would guess is the actual processing, so I would have a simple queue and then dequeue things from there on a single thread, then just use the taskpool or something to do the work.
Then when the stuff comes back marshal it back onto the dispatcher thread and put things into an ObservableCollection
That way WPF will get automatically notified when the contents of the collection changes.

That's all just speculation though, could be completely different based on your actual requirements though!

 

[DanF]

The bottleneck is going to be waiting for responses from the server, my current code running flat out doesn't even dent the cpu. Parsing the responses is also very quick compared to the data transfer but that might change depending on how much work we get it to do. Currently it's running 4 web requests at the same time in different threads and putting the data back in to ConcurrentDictionary.

 

[Haircut]

The bottleneck is going to be waiting for responses from the server, my current code running flat out doesn't even dent the cpu. Parsing the responses is also very quick compared to the data transfer but that might change depending on how much work we get it to do. Currently it's running 4 web requests at the same time in different threads and putting the data back in to ConcurrentDictionary.

That's what I mean. If you marshal the results back to the dispatcher thread before putting them in your container WPF can simply update the screen from a binding to that collection without it needing to worry about the fact that the underlying mechanism of fetching the data is multithreaded.