OSX is pretty clever at using all the memory it can get hold of to improve performance right now, but then releasing more instantly when it's needed by something else.
As mentioned, unless your system is slowing down, don't worry about it. If your system is slowing down, open activity monitor and see if anything is using excessive memory or CPU. I find that of the 5 occasions a year it happens, 4 are safari web apps or flash player. A quick force quit and all is well.