Pulling 59000 records out of SQL into a webpage

[HungryHippos]

Hi all,

At work I am working on a website which currently pulls data out of CSV's and displays in tables via ASP.NET C# code and Response.Write(), the CSV files are updated via scheduled tasks/scripts that run on the webserver.

Generally works OK but one of the CSV files contains 59000 records and as you can imagine dynamically writing a table using this method can take a bit of time, and doesn't handle that many records well at all.

I was considering feeding the data into a SQL table and using this instead, i'd imagine it would be quicker, what is the best way to achieve this taking speed as the main factor?

Some caveats:

• Columns need to be sortable and uses various field types such as text, numbers, dates.
• Data needs to be filterable, i.e. users can enter a value to search for in the fields and I can check if the row matches the filter before writing it out.
• On some columns, I would like it to check some thresholds and then display a different image depending if the threshold is good (green), warning (yellow) or bad (red) after the data in the table cell.
• Would be great if I could dynamically pick which fields to display, or otherwise deselect columns and then refresh the data easily.

Bit of a tall order really but not sure where to start, so would welcome some direction. I'm no pro at scripting in ASP.NET but I can take examples and figure them out/build up my knowledge as I go.

Thanks!

 

[Izi]

try the sql server compact edition: http://blogs.msdn.com/b/sqlserverco...er-compact-4-0-is-available-for-download.aspx

The just read the data from the csv in to the database using insert statement.

You could then bind the data to a gridview / listview and add sorting etc.

 

[PaulStat]

Mysql has a fast way of doing this

http://dev.mysql.com/doc/refman/5.1/en/load-data.html