Initial configuration plays a key part in your overall site security. The most obvious consideration is the passwords you choose for your site, but they shouldn't be the only thing you focus on when beginning the installation.
Database Security
Host name privilege
When creating a database, the database administrator has the option to restrict access to the database to certain hosts (local or remote systems). The lazy option would be to use the wildcard % (any host), but this creates a vulnerability. If open to any host request, an attacker might begin a bruteforce attack against the database server.
The safest option when creating database permissions is to specify the host by IP address (or localhost if appropriate). If the future requires you to grant access from another host, it's a quick configuration and much less painful than recovering a hacked database.
Usernames and Passwords
Be creative! Reusing the same username/password combination you use on the server is dangerous! Remember, the database username and password are stored in plain text (unencrypted) in the configuration.php file. If someone is able to exploit your site and retrieve that information, they could gain access to your site via other (more direct) means. Very dangerous! Pick a unique username and password to access your database.
Joomla Evolution
I started using Joomla back it was called Mambo (if that offends you, just let it go). Back then, the database tables were always prefixed "mos_". When the Joomla fork occurred the table prefix became "jos_". Many were unhappy about that, and they hacked their core files to accomplish a prefix change.
Fast forward 2 major revisions and today we enjoy an installation script that generates a random table prefix.
For those who don't see the significance of a random table prefix - imagine if a burglar knew half of the pattern for your house key. A random prefix makes certain SQL attack vectors useless because the table names are not known. If you can't see the target, you have no hope of hitting it.
Sample Data
Do not install sample data unless you have a good reason. Choose "None". Picking any of the other options enables modules that you would be disabling later.
Administrator Credentials
Change the admin username to something else. Under no circumstances should you leave the username as "admin".
A piece of advise about passwords:
Length and content are key. Longer is better and if you're going to use words - misspell them. An 8 character password can be cracked in a few seconds. A 10 character password takes days to crack. I don't use passwords with fewer than 12 characters, and my passwords do not contain any actual words.
I think Matt knows it only takes 31 seconds to crack the password "love".
Love is great, but not as a password.
Matt Mullenweg