Realworld Unit Tests

[D.P.]

As a scrubby CS student who is replying to this mainly as some kind of terrible alternative to revision, I would probably remove the ranf(); calls and pass those values in as arguments. Then you could test against whatever list of data you want, with a known list of outputs, and not have to do any crazy statistical anythings against the output of that function.

I appreciate the input but I don't think this approach would actual serve as a valid test. I know I can create unit tests for any sub-function and replace the call to ranf() with known values to test these sub-functions.

However, what I really want to test is if that function as a whole returns the correct results, i.e. normally distributed random number. The only way to test for this is to use statistical tests over large sample sizes.


You can remove the randomness, replace calls to rand/ranf etc. or use a fixed random seed to the same values are returned. However, you then fail to test the scope of the returned values.

E.g., it may so happen to be with the fixed seed and the extent of your testing by chance everything woks as expected. With a different seed (or after many additional calls to rand() the function gives an erroneous result.

 

[ZombieFan]

I can test for normality with test like Kolmogorov-Smirnov, Shapiro–Wilks, etc. But that involves exporting the data and processing in a statistical package like R/Matlab

Forgot about this thread

You don't have to export the data to use these methods. There are libraries available which you can use in your unit test project to check from within the code. For example, here is a C# library which performs a Kolmogorov-Smirnov test against a sample of data - http://www.extremeoptimization.com/...s.OneSampleKolmogorovSmirnovTest_Members.aspx

Do some digging and you will probably find a similar library in your language of choice.

 

[D.P.]

Forgot about this thread

You don't have to export the data to use these methods. There are libraries available which you can use in your unit test project to check from within the code. For example, here is a C# library which performs a Kolmogorov-Smirnov test against a sample of data - http://www.extremeoptimization.com/...s.OneSampleKolmogorovSmirnovTest_Members.aspx

Do some digging and you will probably find a similar library in your language of choice.

yeah, that is my plan to find a suitable C++ library that doesn't have too many other dependencies (like to keep thing lean and clean without an large dependency tree).

Exporting the data would just allow it to be tested very easily in something like R.

 

[aln]

Most Unit test implimentations will likely only catch obvious mistakes, although that'll depend how much effort you put into the tests. Generally it's a saftey net that devs aren't making anything stupidly broken live (or rather, they shouldn't even be pushing such crap to the repo).

For example, lets say you have a function that's supposed to return a random float, you don't know what the result is but you can test whether or not it returns a float. When a junior comes along and changes it so it returns an int, having said function unit tested will tell you and him very quickly that it's broken. Obviously, as you've said, this is trivial and not the best example in the world, because he's just as likely to make it return the wrong float (and in a static typed language is probably not going to compile anymore), so you can and would delve deeper where appopriate, but the point is, you _if_ miss that on a code review (or you don't do reviews) the difference between you staring at the screen wondering why the hell it's broken and having the ability to be told instantly is a good thing. Even better, said dev should be unit testing before they commit, saving you time from every needing to see these trivial mistakes.

See you're making calls on the basis that you know yourself and all your guys are good and know the code base, throw some churn into your staff levels and even fairly basic unit tests will probably more than make up for the time spent. If you want to waste time reviewing simple mistakes, by all means go without them, but complaining it's not always a silver bullet is probably missing the point.

 

[Randomface74]

after writing thousands of unit tests I believe they're not worth the cost to the business - better off getting the product to market sooner

 

[aln]

after writing thousands of unit tests I believe they're not worth the cost to the business - better off getting the product to market sooner

I'd say thats an argument against TDD not unit testing, as long as you have some simple tests the infrastructure is there and you can add more later (which means "will never happen" in many companies).

That said, unit tests aren't really about helping you, they're about helping the guy who comes after you. Have you ever hit a new project you don't completely understand that has unit tests? I have, and I appreciated them.

 

[Dj_Jestar]

Again ... Unit tests are for designing software. The "test" part (regression and other) is a by-product, and a guarantee that when you refactor you haven't broken something.

e: This is in reply to billysielu

 

[D.P.]

I'd say thats an argument against TDD not unit testing, as long as you have some simple tests the infrastructure is there and you can add more later (which means "will never happen" in many companies).

That said, unit tests aren't really about helping you, they're about helping the guy who comes after you. Have you ever hit a new project you don't completely understand that has unit tests? I have, and I appreciated them.

We always write small example tests of functionality that show the code works in at least a couple of intended examples, showing new developers how the functions work etc, along with some comments and descriptions, but not a through set of unit tests. I am not sure unit tests really work well as a form of documentation for new developers because seeing thousands of lines of repeated code that have inputs often designed to break the function is not very intuitive.


The bottom line that we care about is if the intended behavior of the complete system being correct, not whether some low-level functionality is correct which is implied by the higher level goal. Secondly is regression testing so changes can be comfortably made in the future with confidence, which puts more weight on unit tests.

Third is performance tests which seems to be completely ignored by the unit test philosophy. I made some unit tests for some code earlier in the week and everything behaved exactly as expected giving all the correct results. However, one thing I did not was the functions were taking a little longer than expected to compute. Spend some time looking at the code, profiling individual components and all looked fine. Turns out the the sign was flipped in a priority queue so the heuristic search behavior was acting as a worst-first search instead of best-first. the upside was that the correct result are returned but only after extensively searching the graph through all the worst possible result before coming to the correct answer.
Unit testing completely failed to find that bug as it was performance related.

 

[aln]

We always write small example tests of functionality that show the code works in at least a couple of intended examples, showing new developers how the functions work etc, along with some comments and descriptions, but not a through set of unit tests. I am not sure unit tests really work well as a form of documentation for new developers because seeing thousands of lines of repeated code that have inputs often designed to break the function is not very intuitive.

It's not really a form of documentation, it's just saves a whole lot of bother if when I make a change on a new project, that I can run the tests and see if I'm wasting my time comitting my changes. This saves my time comitting something that is obviously broken and your time telling me this. Obviously I can still commit something that's not going to be accepted, but more than likely, it's going to be a lot closer overall.

The bottom line that we care about is if the intended behavior of the complete system being correct, not whether some low-level functionality is correct which is implied by the higher level goal. Secondly is regression testing so changes can be comfortably made in the future with confidence, which puts more weight on unit tests.

Of course. There are attempts at automated testing for non backend functionality in several languages, whether or not they're useful on your platform is something I don't know.

However, "this doesn't cover 100% of all potential problems" isn't actually a strong argument to write it off. You cover what you can and keep an eye on the rest.

Third is performance tests which seems to be completely ignored by the unit test philosophy. I made some unit tests for some code earlier in the week and everything behaved exactly as expected giving all the correct results. However, one thing I did not was the functions were taking a little longer than expected to compute. Spend some time looking at the code, profiling individual components and all looked fine. Turns out the the sign was flipped in a priority queue so the heuristic search behavior was acting as a worst-first search instead of best-first. the upside was that the correct result are returned but only after extensively searching the graph through all the worst possible result before coming to the correct answer.
Unit testing completely failed to find that bug as it was performance related.

What's to stop you doing that in a unit test?

If you're using a testing first strategy, what you should have done is wrote the test to check for performance the second you realise there wasn't one, so the problem doesn't crop up again.

I'm not sure how you don't get that? Are you arguing more along the lines that you think unit testing is a waste of time because it's not being done right? If so, who do you imagine should be writing these fictional tests? Someone other than yourself?

 

[Dj_Jestar]

D.P. you seem to be trying to redefine unit testing at every given opportunity. Unit testing is not performance testing. It is not integration testing. It is not acceptance testing. It is unit testing. It is the testing of a single unit of functionality or behaviour.