Advice for c# winforms dev

[DanF]

I've been developing with c# for years now but it's all been web based so I have almost zero experience with winforms setup. I'm wanting to make up a pretty basic web crawler that can crawl over our website looking for any issue.

I've played around with the main crawling and parsing code in linqpad but I need to get it in to an actual app now, it'll be a fairly basic xaml .net 4 app.

Is there anywhere that gives good advice on which way to do threading/shared collections in winforms? I've looked around and there seems to be loads of ways now. My thoughts where to have part of the code do the actual scraping putting the response in to a shared container, then another process can parse this and put the errors and new links back in to other shared containers. The part that does the actual scraping will need to have some form of slicing/threading as well to get performance out of it.

 

[DanF]

Cheers this is what I was looking for, I knew I didn't want to block the main UI threads yeah it'll be a WPF app where it'd be nice to have counters updating showing how far it's got.

For the shared containers would ConcurrentBag and ConcurrentDictionary be good fits?
ConcurrentBag to hold url's to parse in FIFI style, the other to hold the response against the url for further processing.

 

[DanF]

The bottleneck is going to be waiting for responses from the server, my current code running flat out doesn't even dent the cpu. Parsing the responses is also very quick compared to the data transfer but that might change depending on how much work we get it to do. Currently it's running 4 web requests at the same time in different threads and putting the data back in to ConcurrentDictionary.