With over 7000 extensions in the Joomla Extension Directory, it can be tempting to install a ton of extensions because you might want to use them. That is a very common mistake. Not all developers are created equal. Some are more security conscious than others, and that's something you can learn with research.
If you absolutely MUST install an extension to check it out - don't do it on a live server. I don't know any developers who don't have at least one development server. Spend a few minutes to install a local lamp server that you can sacrifice to potentially bad extensions. The last thing you want to do is try to recover your site from a backup - trust me on that.
Who is the Author?
The first thing you want to do when looking at an extension is to take a look at the author, their website, their other extensions and most importantly - the reviews on this and their other extensions. As a developer with many extensions in the JED, I can tell you that bad reviews stick. The JED moderators are not accommodating when it comes to reviews. In the years I've spent developing Joomla extensions and receiving JED reviews - only one was ever removed, and that was a review made by the author of a competing extension who attempted to get a better rank by breaking the rules and giving me a bad review. For his efforts - all of his extensions were removed from the JED and he is the proud recipient of a lifetime ban.
The VEL is Your Friend
Joomla publishes a "Vulnerable Extensions List", where any extension reported to have a vulnerability can be found. During its time on the VEL - an extension is removed from the JED until the author fixes the issue. Before installing an extension, search for the authors name and their other extensions on the VEL. Chances are good that the author of one vulnerable extension is the author of multiple vulnerable extensions.
Reviews on the Web
The JED isn't the only place to find reviews. Some people don't have accounts on the JED, they just download extensions. Reviews, both good and bad, exist on personal blogs - Twitter feeds and even Facebook.
An Exact Science
Unfortunately, there isn't any real way to know unless you read the code. I suspect that if you're reading this article, you are either a developer critiquing it, or you're reading it to learn. Developers, please give me some feedback on things I may have missed or explained badly. If you're here to learn, I hope this helps.
Isaac states the truth.
Learning to trust is one of life's most difficult tasks.
Isaac Watts