It's your personal choice and value for your dollar. Some just have to get the newest, regardless of price or differences. Others wait for a few years just to get a break on the price. I just bought my 04 SSR just a few weeks ago and love it. Great exhaust sound and the performance seems fine given the weight of the truck. It is after all a sport truck, not a Vette or other pure performance car. I have an 03 BMW M5 and an 03 Ford Cobra Mustang that I bought for the pure performance. The SSR is mainly for how it has always just looked appealing to me and nothing is cooler than the power "hard" top convertible. I had an 00 Vette convertible and I actually think I'm going to enjoy the SSR more, certainly just as much.
I feel fortunate I got a fantastic deal, better than any other new vehicle I have ever purchased. For me saving $15,000 over an 05 made it an easy choice. I have 2 other manual gearboxes so having an automatic for my daily driver doesn't bother me, especially with a plug in tuner that firms up the shifts. If I find out I really need the extra HP I can always spend $2,000 to $6,000 and add NOS or a supercharger/turbo and blow any 05 away.
Go drive both & check out the differences (the new 05 Aqua Blue color is very neat) and THEN buy what you really want. I assume you can probably afford either year or you wouldn't be looking at all so get what will make you the most happy AND enjoy.