Big data export cost and latency

[D.P.]

With a totally non-tech head on, I imagine this is purely value. Some of our services are more value driven than cost and a very good analogy is something like Getty.

A photo that was taken 5 years ago might cost £10 to use for editorial, £50 for low res web, £100 medium res, £200 high res, £500 large scale print use, £1000 TV broadcast, etc. Same photo, same work that went into it, just a different resolution, file size and usage, all of which are minimal or no additional cost whatsoever. You're paying for value.

Totally get that, this specific data in infinitely more valuable real-time then even an hour old. But we are trying to work out a pricing model that is based on operating costs and not value to allow us to develop our market and clients, at which point market value is easy for us.

 

[D.P.]

Is the weekly/monthly etc data as granular as real-time?

I'd suspect that if the main aim of their business is supplying real-time data to their clients, after probably an hour the data is likely now worthless to them, and they probably only store it for archival/record purposes. If you come along and are now interested in the 'old' data, then this company will see another revenue stream, even if it's considerably smaller than usual.

Yes, same granularity. Potential that they can compress data beter once it is no longer real-time.

As mentioend in post above, most cleints have much more valuer for the real-time but the older data is not worthless and isn't purely archival.

 

[touch]

Of your list, only the last point, actually sending data to us would have any real computational cost though.

You said the amount of data transferred would reduce if the amount of requests increased.
You either need to send the entire dataset at each request (which is a lot of bandwidth) or you need to filter the dataset to select only the new data since the last request (which is processing time). You can have it both ways.

As to pricing, well simply we don;t have the capital to pay their market rate for the data so ideally we have a revenue share model and wouldn't pay a dime until we have paying clients but since they incur operating costs then a cost plus model makes more sense. This is pretty standard, several of are clients run this kind of pricing model with the data we provide.

Your "cost plus" model might make sense to you but it doesn't make any sense for them.
They obviously have clients who do pay the market rate for the data - why would they risk upsetting those clients by giving you a much cheaper price?

 

[Semple]

So it is probably a combination of cheaper storage costs for older data

Was just about to add this as another point.

If the company only has a small number of clients accessing weekly/monthly data dumps then storage costs will be significantly cheaper than those in use for storing/backing-up the real-time data.

Going by the comments in this thread, there's quite a number of factors that will influence the cost of this data.

And going on Beansprouts comment, if it's a niche market, they know they can probably sell the data at whatever cost they like, even if the pricing model doesn't quite make sense.

If what they're selling isn't an easily-available commodity then you can't just rock up and ask for it at "cost price" because "cost price" is completely irrelevant - and thinking the cost is just the bandwidth is surely offensive to them as presumably work goes into building whatever it is. The cost of building a website can be 50k but the monthly bandwidth bill could be a fiver!

 

[D.P.]

You said the amount of data transferred would reduce if the amount of requests increased.
You either need to send the entire dataset at each request (which is a lot of bandwidth) or you need to filter the dataset to select only the new data since the last request (which is processing time). You can have it both ways.

The data is requested by timestamps, so there is some filtering required but the data should already be indexed by timestamp. At least it would be odd if it wasn't. But either, they don;t need to store anything about prior requests, each data export would just be given updated query times.

Your "cost plus" model might make sense to you but it doesn't make any sense for them.
They obviously have clients who do pay the market rate for the data - why would they risk upsetting those clients by giving you a much cheaper price?

Firstly, it is not our pricing model, this is their pricing model they are proposing to us so of course it makes sense to them.
Secondly, What their other clients pay is largely irrelevant. The value oif the data is whatever the client finds value in, and different clients in different industry will have vastly different values and so their cost will change.

Thirdly, this is entirely irrelevant to the discussion and is far more complex than can be discussed here, and is under NDA so is entirely pointless to speculate. I am only interested in technologies and services for big data achievable and export and how this relates to the cost. I have had some reasonable advice that would certainly explain a good deal of the cost differences.

 

[Beansprout]

Dear Amazon, 1Gb network transfer only really costs fractions of a penny and all I'm doing is streaming cat videos so please sell it to me at that cost price for that bandwidth, not your list price of 10p, because I don't place any value on all the kit you invested in to deliver the bandwidth in the first place.

At some point you have to admit defeat here....

 

[D.P.]

Dear Amazon, 1Gb network transfer only really costs fractions of a penny and all I'm doing is streaming cat videos so please sell it to me at that cost price for that bandwidth, not your list price of 10p, because I don't place any value on all the kit you invested in to deliver the bandwidth in the first place.

At some point you have to admit defeat here....

More irrelevance, and as it happens we get over 100K's worth of AWS for free.

You obviously have very little understanding of how companies actually negotiate.

 

[Beansprout]

More irrelevance, and as it happens we get over 100K's worth of AWS for free.

You obviously have very little understanding of how companies actually negotiate.

100Ks of what, Kat videos?

I thought it was a fair analogy so that seems a bit uncalled for, I'm done here, like I said before, good luck.