SEO is a minefield and a ever changing space. Always has been and always will be. This is a quick reply so may be a bit rough and is my opinion after years of working in or around the industry. All of the below are some of the gotchas to look for there are more of course not all SEO's are created equal.
If you have done some yourself you will have realised the amount of time it takes and the skills needed to do it right and know what it is worth to you & your company.
A SEO calculator would be impossible to put together I would have thought, too many variables. Strength of competition, skills and experience of the SEO. Overheads of the company, you will get better value but less security from a home based SEO than a company with large offices in city centres. You are paying for that fancy office & the sports cars in the car park!
Set a realistic budget & expectations and see what you can get for it, then if the results you get are good up the budget.
If they are giving you prices without doing some research then alarm bells time.
No two keywords/sites are the same and can take a huge amount of work or a little.
If they have solid pricing then alarm bells time
As above by solid pricing I mean set pricing a rougn price is ok, but some have solid pricing structures. Packages are setup to suit them not to help you.
If they won't at least give you a clue of what they will be doing then alarm bells time
They won't usually give you details but should at least give you a idea.
If they are guaranteeing results then run like hell.
The only people who can guarantee results are the SE's NO ONE else can. & is the reason that there are so many bad SEO's out there.
If they mention backdoors/shortcuts run like hell
Those that are found don't last long and the sites that use them get dropped.
Realise that they may or should be wanting to make changes to your site, unless it is bang on. Some people get real arsey about that.
Also they should be talking to you about conversion optimisation. Waste of time ranking #1 on google for cheap air fairs, getting 1 million hits per day and selling 1 ticket!
Be wary off Seo's doing performance based, (where you are contracted to pay once the agreed requirements are met. i.e keyword ranking pos 10-7 = £x, 6-4=£x, 3-1=£x for 12 moths etc..) they will rush to get it there which can and has lead to problems.
Hope that helps you find the right person.
SEO is a skill that takes a lot of time to learn and keep up to date. The tools you need can be expensive. It is not about spamming links with regurgitated content.
You need solid foundations for the longterm, unless your business model is one of rank the site and sell!
A good SEO will understand other related tech and be able to help you out in all related areas.
Of course speak to as many as you need to until you find someone you are comfortable with.
Eeeek went on a bit long there! Sorry