Depends on YOU.

Rock Band if:

If you want to play different instruments or sing
If you like the track list
If you regularly have at least one other person to play with (World Tour mode)

GH3 if:

You JUST want to play guitar
You like the track list
Singing and drums do NOT appeal to you at all
You never plan to play multiplayer