What technologies for graphing real-time data on a website?

MrWhippy · 22 Mar 2020 at 18:07

Fancy learning something new and have a few project ideas, several of which could involve needing to graph/chart numeric data to a Web page. I've very little knowledge of code or appropriate technologies for a task like this. Any suggestions for sensible ways I might go about it?

For info, I'm an ex software engineer, not coded for a few years, and my experience is split between embedded systems and high performance processing. I'm not afraid to learn new stuff but I've no idea where to start. I've next to no knowledge of databases or Web technologies.

I do know c, c++, java, python.
Let’s say I have a numeric data stream being generated on my server, what might be a sensible way of setting up a Web server to display it? Do I need to be sending the data to a database? Is this a job for javascript or something else?

This is all for personal interest, just a little project to blow away the cobwebs, no serious or commercial objectives.

Cheers for any pointers.

BongoHunter · 22 Mar 2020 at 18:24

I started to answer this - bit more info is needed really to make a decent recommendation...

How frequently does the data change? And what does it look like (single metric times series?) - your technology choices might be informed by the type, volume and frequency of your data.

Have you looked at off the shelf solutions like Grafana or ELK?

Where does the data come from?

MrWhippy · 22 Mar 2020 at 19:04

Cheers. I guess it won't be very high data rate, one or two hz at most but could be much lower. Initially likely to be single time series data, numeric floating point, down the line it might be multiple streams. The data will be being generated by code running on the server, not written yet, might be a running executable or python script.

As I said I have a number of project ideas I fancy playing with, but several could require visualisation of time series data so I thought I'd better find out how to do that sort of thing.

So one project might be analogous to monitoring data from a weather station (temperature / humidity / barometric pressure etc). Interfacing to hardware to take measurements and data processing and analysis I'm capable of, but I've no experience of data visualisation methods.

Thinking ahead, further down the line I might want to be able to communicate back into my 'system' from the Web page - toggle switch / setting values etc. Does this influence the choice of methods? Cheers for any suggestions.

planty · 22 Mar 2020 at 21:55

If you want to stream data to frontend, websockets would be an obvious choice - I'm more of a .NET guy so would use signalr (https://github.com/SignalR/SignalR) for that, but not sure what the options are in the java/python world. Your webpage would use a js client, and then your web server can just push data down to it via the 'socket' as and when it needs to.

If you need to be displaying the 'historic' data when the page is reloaded then obviously you'd need to store that data in a database somewhere and request it on page load.

There are loads of js graphing libraries around - chartjs is good: https://www.chartjs.org/

MrWhippy · 23 Mar 2020 at 07:43

Thanks for the heads up, that gives me somewhere to start. I'll get reading.

GravyMonster · 23 Mar 2020 at 08:56

I've done similar in the past and the graphing part is the easy bit, use HighCharts. The tricky bit is getting the data to the client and keeping it scalable (in other words polling is out of the question). WebSockets is absolutely the way to go for this. Like @planty I used Azure SignalR so it scales to hundreds of thousands of concurrent users.

Dj_Jestar · 25 Mar 2020 at 12:40

If you're just messing about, try out Jupyter.

gpuerrilla · 6 Apr 2020 at 20:54

I was going to suggest Jupyter when I read the OP. Im a bit late to the party. I dabbled with a thing called Anaconda with Python for a moment. Something to have a peek at, you should be comfortable in no time giving your background.