@Dronz3rAs I see it, our Booker and Elisabeth is still the first to escape Columbia.

When they escape, they just trigger millions of alternate universes where they traverse the bridges and lighthouses in different ways. I think the purpose of the scene is just to illustrate the concept of a multiverse. Our Booker will still be the first to drown out of all of these other universes – otherwise the world would have “collapsed” before he got to that point.

@Topherfrom it is uncertain whether or not this is true, or simply fiction created by Comstock to magnify his power over Columbia

Great analysis, but I think you got something fundamentally wrong.

Anna would not have been born before Booker is baptized. Why do you think Comstock wants her in the first place? He most likely has no children. And as we learn in the game Comestock did not only get cancer from using Luteces' contraption, he also got sterile. To continue his legacy and plans for the future he had to get a child, and what better way to do this than to use Luteces' contraption to look into alternate universes, where he got a child, essentially adopting his own flesh and blood? That is, we have two outcomes that drives the story. One where Booker gets baptized and renames himself Comstock, and one where he does not get baptized and gets a child, with the ladder being the protagonist, that we are playing.

So yes, if Elisabeth kills Booker before the baptism she will in fact cease to exist, and that is what you are seeing in the very last seconds before the credits.

Or!... what if she only drowned the Bookers that where to accept the baptism? Elisabeth would still cease to exist, but the “original” version of her, Anna DeWitt, will not. She will be right back in her fathers cradle, since there never was no Comestock to take her away in the first place. Maybe that's what the post credits sequence is suggesting – an ending with Booker and Anna Dewitt together – a happy ending :)