Data Trend Detecting Maths?

tntcoder

tntcoder

tntcoder

Soldato
Joined
7 Apr 2004
Posts
4,212
Hi,

Say you have a real time stream of data like the current trade price for a given company's shares. What kind of area of mathematics can be used to detect if you are currently at a peak?

I guess you can do trivial things like graphing and comparing to historical data over recent days but im interested if there are more complex analysis algorithms that can say in real time "based on the last t minutes/hours the next price you get will be lower than the current trade price with probability X".

Obviously it can only be done to a rough degree of probability and im sure pro traders have such tools in their toolbox. Just searching for reading material on the topic :)

Is there any good books on this kind of mathmatical trend-detection for time/value data streams? If possible without requiring a maths or finance degree :p
 
What you are looking for is Time Series Analysis.

You can use Box-Jenkins methodology to project the mean values and confidence bounds (ARIMA models). Professional investment analysts may use something like the GARCH model to determine past volatility or make projections into the future.

For things as volatile as day to day share prices brownian motion models are used.
 
Last edited:
tntcoder said:
What kind of area of mathematics can be used to detect if you are currently at a peak?
Click to expand...
There is no such mathematics, else we'd not have unexpected crashes and people would know when to dump their stocks.

tntcoder said:
I guess you can do trivial things like graphing and comparing to historical data over recent days but im interested if there are more complex analysis algorithms that can say in real time "based on the last t minutes/hours the next price you get will be lower than the current trade price with probability X".
Click to expand...
There are mathematical methods which allow you to interpolate into the future based on behaviour seen in the past, though quite how you do it depends very much on what you're considering. Things like Kalman filters are good for predicting the motion of moving objects and are relatively simple on the scale of things, but they don't deal well with the 'jaggedy' nature of stocks.

In fact stock prices bear considerable resemblance to random walks but just because something visually looks similar doesn't mean you can just apply known methods unfortunately.

tntcoder said:
Obviously it can only be done to a rough degree of probability and im sure pro traders have such tools in their toolbox. Just searching for reading material on the topic :)
Click to expand...
While there's an army of PhDs employed to do this sort of stuff its also deeply working how horrific or useless at maths some people in charge of vast sums of money are.

tntcoder said:
Is there any good books on this kind of mathmatical trend-detection for time/value data streams? If possible without requiring a maths or finance degree :p
Click to expand...
Depends what you want to do. Do you want to just have an inkling of how the professional statisticians do it, ie you know a few equations but nothing of any use, or do you want to be able to construct your own models and use them to guide investments? If its the former then buy a book from WHSmiths. If its the latter then you're really going to need to devote serious time to it. Said professionals do it day in, day out and even then the models aren't something you'd really want to bet the farm on (though people regularly do).

Vague areas you'll want to look at are things which involve buckets of statistics. Here is a run down of Cambridge Part III mathematical courses provided by their pure maths department and one of the sections is titled "Operational Research, Statistics, Mathematical Finance", starting on page 37. It gives reading recommendations but since its aimed at people doing the 4th year at Cambridge its aiming pretty high and assuming the reader is capable enough to get on said 4th year. It'll give you areas to look at and perhaps find a book a little more introductory.
 
Lightnix said:
This is what we do instead of socialising.
Click to expand...

lol!! Come to my house and have a beer... Forget that stuff. Only select people need to know about that and they ain't invited :D That guy in the wheel chair with the robot voice (can't mind his name) has it all figured out anyway so relax.
 
BetaNumeric said:
There is no such mathematics, else we'd not have unexpected crashes and people would know when to dump their stocks.

There are mathematical methods which allow you to interpolate into the future based on behaviour seen in the past, though quite how you do it depends very much on what you're considering. Things like Kalman filters are good for predicting the motion of moving objects and are relatively simple on the scale of things, but they don't deal well with the 'jaggedy' nature of stocks.

In fact stock prices bear considerable resemblance to random walks but just because something visually looks similar doesn't mean you can just apply known methods unfortunately.

While there's an army of PhDs employed to do this sort of stuff its also deeply working how horrific or useless at maths some people in charge of vast sums of money are.

Depends what you want to do. Do you want to just have an inkling of how the professional statisticians do it, ie you know a few equations but nothing of any use, or do you want to be able to construct your own models and use them to guide investments? If its the former then buy a book from WHSmiths. If its the latter then you're really going to need to devote serious time to it. Said professionals do it day in, day out and even then the models aren't something you'd really want to bet the farm on (though people regularly do).

Vague areas you'll want to look at are things which involve buckets of statistics. Here is a run down of Cambridge Part III mathematical courses provided by their pure maths department and one of the sections is titled "Operational Research, Statistics, Mathematical Finance", starting on page 37. It gives reading recommendations but since its aimed at people doing the 4th year at Cambridge its aiming pretty high and assuming the reader is capable enough to get on said 4th year. It'll give you areas to look at and perhaps find a book a little more introductory.
Click to expand...

Really informative post this. Thank you for sharing
 
BetaNumeric said:
There are mathematical methods which allow you to interpolate into the future based on behaviour seen in the past, though quite how you do it depends very much on what you're considering. Things like Kalman filters are good for predicting the motion of moving objects and are relatively simple on the scale of things, but they don't deal well with the 'jaggedy' nature of stocks.
Click to expand...

Wow :eek: What are ye smoking my friend? :)
 
BetaNumeric said:
There is no such mathematics, else we'd not have unexpected crashes and people would know when to dump their stocks.

There are mathematical methods which allow you to interpolate into the future based on behaviour seen in the past, though quite how you do it depends very much on what you're considering. Things like Kalman filters are good for predicting the motion of moving objects and are relatively simple on the scale of things, but they don't deal well with the 'jaggedy' nature of stocks.

In fact stock prices bear considerable resemblance to random walks but just because something visually looks similar doesn't mean you can just apply known methods unfortunately.

While there's an army of PhDs employed to do this sort of stuff its also deeply working how horrific or useless at maths some people in charge of vast sums of money are.

Depends what you want to do. Do you want to just have an inkling of how the professional statisticians do it, ie you know a few equations but nothing of any use, or do you want to be able to construct your own models and use them to guide investments? If its the former then buy a book from WHSmiths. If its the latter then you're really going to need to devote serious time to it. Said professionals do it day in, day out and even then the models aren't something you'd really want to bet the farm on (though people regularly do).

Vague areas you'll want to look at are things which involve buckets of statistics. Here is a run down of Cambridge Part III mathematical courses provided by their pure maths department and one of the sections is titled "Operational Research, Statistics, Mathematical Finance", starting on page 37. It gives reading recommendations but since its aimed at people doing the 4th year at Cambridge its aiming pretty high and assuming the reader is capable enough to get on said 4th year. It'll give you areas to look at and perhaps find a book a little more introductory.
Click to expand...

Thanks, really useful!

I have done work with data mining before, and I guess the principle here is similar. I used algorithms like Bayes classifiers, neural nets, Multi Layer Perceptrons, Decision Trees etc to classify new data based on trends seen in the past. The only problem is these statistical routes ive used before fail quite badly on this kind of data.

Im not really interested in it for investment use, I just have some other data that has similar entropy levels and trends to stock data and some prediction based on past data sets is just generally really interesting. Just looking to learn some algorithms I could potentially use for automated 'AI style' software for data analysis.

Plenty to read now though :)
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom