In the old days, when DirectSound3D and OpenAL were commonplace in games, the sound card DSP would take the load off the CPU. However, starting around 2007, FMOD Ex became the dominant middleware of choice, and it's all software-processed on the CPU. Same goes for XAudio2 + X3DAudio. Unfortunately, I find them inferior substitutes to the proper 3D sound APIs, mostly since I can't get proper 3D binaural sound out of my headphones and have to settle for virtual 7.1. (CMSS-3D Headphone on X-Fi cards mixes the 3D positional data from DS3D and OAL games when available, but for XAudio2 and FMOD Ex, it's limited to pre-mixed 7.1 channels.) Software-processed audio done right would basically be OpenAL + Rapture3D, which allows for your usual 7.1 mixing and also has no less than six different binaural HRTF modes to use with headphones. Then there's the use of EAX extensions that used to be dominant in the past. They're just reverb/chorus/occlusion/etc. effects (the sort of things done in software today), but the fact still stands that if you play a game that uses EAX with a primary audio device that doesn't support them, you lose the audio effects entirely and it's not as the game developer would have intended it to sound. Most audio devices (except audiophile USB DACs, since they're not meant for gaming) support EAX 1 and 2 since they were effectively made part of DirectSound3D itself, but EAX versions 3, 4 and 5 remain the sole domain of Creative hardware. As for whether that particular X-Fi is worth it...that depends on whether it's one of those XtremeAudio fakes, or one of the real cards with the EMU20k1 (PCI)/EMU20k2 (PCIe) DSP and all the gaming features you're looking for. If it's the latter, keep it.

I don't think it'll dramatically hinder game performance on a computer made within the last five years. When people buy sound cards for gaming these days, it's not for a framerate boost, but a sound quality boost. As for whether you'd prefer the X-Fi card and a nice set of speakers or headphones over the USB Bose setup for general music playback and things that are NOT gaming (where the choice is clear), it's hard to say. Sound is highly subjective like that, and people can have very polarizing opinions on what sounds "good" or "right". At the very least, the X-Fi cards have a nice EQ you can use to tailor the sound signature to your taste; just make sure to keep the sound card volume setting down to 25% or so if you want to EQ upwards to avoid clipping. (Obviously, this assumes you can do your actual volume adjustments on an external amplifier.)