C# threading- returning values

[SiD the Turtle]

Hi guys.

I need to read 400+ files from the Internet which of course takes time, most of which is latency requesting the file as they are very small files. Therefore I want to dabble in threading to say do 10 simultaneous requests at once.

The difficult part is that I read the files into a hashtable which is processed further down the line, but threads do not return values so I can't execute 10 threads that read into the same hashtable, not to mention 10 threads trying to access the same variable at the same time is likely to cause some very nasty problems.

I'm a bit stumped on how to proceed. Has anyone done anything similar in the past, how can I collate the data the separate threads generate into one source?

 

[SiD the Turtle]

Thanks for the info so far guys. Here is my program in a nutshell:

Read into a hashtable a list of URLs from SQL Server
Add to another hastable the source code of each of these URLs
Do some processing on each of these source strings.

The processing takes seconds, it is the fetching of the data from the Internet that takes a while, most of the time is idle time waiting for the remote server to respond to the request. Therefore I was considering using threads to that I could do a few simultaneous web requests at once to get the data back faster.

So imagine that each thread calls a function that gets a single web page and returns its source. I want to get all of these sources into one hashtable so that I can work on it after the threads have completed, but as I understand it threads must always return void.

So thanks for the advice, I'm experimenting locking the variable I have bought in by ref and also investigating delegates, as that's a new concept to me too!

 

[SiD the Turtle]

Righty ho here is a cut down version of what I have working so far:

        static string XblGetPage(String url, CookieContainer cookieJar)
        {
                //Code to get the page and return it as a string
        }

        delegate string XblGetPageDelegate(String url, CookieContainer cookieJar);

        static Hashtable XblGetSource(Hashtable gamertags, CookieContainer cookieJar)
        {
            Hashtable source = new Hashtable();

            XblGetPageDelegate foo = new XblGetPageDelegate(XblGetPage);
            Hashtable threads = new Hashtable();
            foreach (Int16 i in gamertags.Keys)
            {
                threads.Add(i, foo.BeginInvoke("http://live.xbox.com/en-GB/profile/Achievements/ViewAchievementSummary.aspx?compareTo=" + gamertags[i], cookieJar, null, null));
            }

            foreach (Int16 i in threads.Keys)
            {
                source.Add(i, foo.EndInvoke(threads[i] as IAsyncResult));
            }

            // etc...

Apart from error handling the only glaring problem is on the live system that would invoke 400+ threads (unless .Net has a thread cap?), so I will re-jig the code to create say 5 threads at once, add them to the source hashtable, then do the next five etc...

Thanks for the help guys, do critique the above in case I'm going something naughty.

Edit: I'm still learning about delegation, I think it would also be useful to check to see if the thread has finished, not just block on EndInvoke, or I may hold up processing. Lots to read up on

 

[SiD the Turtle]

Right, not really what I had in mind...

What do you mean?

 

[SiD the Turtle]

Thanks again guys, I haven't touched threading since operating system theory back at uni and that was... well all theory on pieces of paper

Thanks for the code, I'm going to step through them and get a better understanding of this all.