Did you install the chipset drivers ?

9.0.0.1009 under Chipset, choose your OS:

http://support.asus.com/download/download.aspx?SLanguage=en-us&model=P5Q

Hard to say what the problem is. Run Memtest to check your RAM.

And maybe update your motherboard's bios.
I would try 0703.
http://support.asus.com/download/download.aspx?SLanguage=en-us&model=P5Q

-Download gpu-z and tell me what the Bus interface is, it should be x16.
-Remove the catalyst drivers, reboot, use Driver Sweeper to remove the leftover drivers. Install 8.7.
-Borrow your friend's card and try it on your pc.

It could be a defective card.

Lower AA to 4X edge-detect (4x edge-detect = 12x CFAA). Or use 4x Box.
Disable Adaptive AA or set it to Performance.
8x AF should be enough, Bioshock doesn't have very large areas.
More info on AA modes here.

Something is wrong. You should be getting much higher fps, average of 118 fps at 1680x1050 according to Anandtech.