Nope you shouldn't be worried at all. I know some people who still don't know what they want to do. They're working and so on, but they don't know what they want career wise. Heck all throughout school I had no idea - I don't think many people honestly know what they want to do - bar a few who become doctors or something like that. To be honest I'm still not sure what I want - and unfortunately am just plodding along - then again I could change careers easily as I have quite a broad skillset owing to my work.
There are 2 different things, having a career, or just working. Personally I'm a work to live sort of person, I don't see myself living to work - ever. Yes you might get rich doing it, but to be honest I'd rather not have my life dictated by my work.
Go and seek some career advice, and be pro-active. Don't do "nothing" as that won't come across well for you when you have decided a path. Don't plod along being miserable either, if you don't enjoy what you are doing, find something you do enjoy.
Don't panic and don't get stressed - it's perfectly normal. Besides, I know some 40+ year olds who never really knew what they wanted to do, but they own a house a car, have a family and can afford to go on holidays - so a career or 1 path is not all that's open to you. However you are young enough to make useful/important decisions as to which path you want to head down.