SSL Labs ScoreSecurityHeaders.io ScoreHSTS Preloaded

Administrators need to stick together

I had an interesting interaction this week that I thought might make a good article. It all started with a bounced email.

Since early 2000, I've been administering live servers on the internet. Web servers, streaming media servers, file servers, RADIUS, weird crap you've never heard of, and even some things I never want to see again. Among those myriad of different server types, mail servers.

If there's one thing that has been a constant for the nearly 20 years I've been doing this, it's that mail server administrators have a certain unspoken pact. I have received the call, and I have made the call - when there is that one problem that requires two email server administrators to work together. Unlike web hosting, email is a different kind of animal. Although I may want your 10,000 users - I don't necessarily want to do the work to get them. Certainly, everyone is protective of their users - but if there is a problem, it benefits everyone if the server admins work together to solve it. If the wrong email doesn't get delivered, it's going to cause a support call - and nobody wants that.

Most of the time, it's something mundane. My user is expecting an email and it's not arriving. I check the logs and find that the sender is misconfigured in some way, or is on a blacklist, or - whatever it is. The problem isn't going to fix itself, so I do the research and make the call. After a short song and dance where we get acquainted and describe the problem - the collaboration begins. Maybe a few test messages go back and forth, then the issue is resolved. Of course, sometimes it's more complicated than that - but in a nutshell, it's the mutual respect and collaboration for the common good that wins the day.

That has been my experience for nearly 20 years. It doesn't happen often, and it usually is resolved quickly (within a day, maybe two or three if there is a weekend or holiday involved).

This was not the case on Friday. My goodness! I couldn't believe what I was being told by a BT Internet (British Telecommunications).

It started, as I described above, with a bounced email. The user #####@btinternet.com could not receive our message, and the reason given by the receiving server was that it was being rejected as possible SPAM. I read it, it isn't spam - it was a harmless informational email regarding account info and login credentials for a product that was purchased. No spam filter in the world would classify this as spam. It didn't even have any links in it.

So, I spent a half hour on hold while calling BT Internet. The email help desk agent asked what my BT phone number was, and when I said that I didn't have one - I'm an ISP from America reporting an email problem, he said that he couldn't talk to me unless I was a BT Internet customer. I argued that I wasn't trying to make a change or access an account - I was reporting a communication problem to BT servers, but he was adamant that he could not speak to me. He told me that I needed to have the customer call to report the issue, and then he hung up on me.

OK... I get that they don't want to divulge customer information, but aside from it being their email address - I wasn't asking for anything to do with the account.

I hatched a plan. They won't talk to me about a customer account - what happens if I'm calling about a problem emailing a BT Internet corporate email address! It took a few minutes to dig up a valid email address (they're surprisingly stingy about it on their website), but I found one eventually - and it was perfect: This email address is being protected from spambots. You need JavaScript enabled to view it.

I copied and pasted the customer email (removing the credentials) and sent it to customer care with a message that I was unable to send it to their customer and that I expected it to bounce. My expectations were correct - the message bounced, and now I had the ammo I needed to talk to them about their problem.

Or so I thought.

I spent another half hour on hold, and this time I explained that I was unable to send an email to the BT corporate address. To my utter amazement - the rep told me that he couldn't speak to me about the issue because I was not a BT customer. I explained that I was trying to email a BT corporate account and that HE was the customer. Surely they would be concerned that outside entities are not able to send email to a BT corporate email address - but nope, he was not concerned one bit. I was informed that if customer care was not receiving some of their emails, they would need to call to report it.

I offered, the only way they would know if they weren't receiving some emails is if someone called to tell them. The rep agreed. Then I dropped the bomb on him: "THAT'S WHAT I'M DOING - I'M CALLING TO TELL YOU THAT THEY'RE NOT RECEIVING SOME OF THEIR EMAILS". The rep could only offer the same line as the other guy - I cannot speak to you unless you are a BT customer.

So, I gave up. I have no way to combat that kind of lunacy. It blows my mind that the rep had no idea what I was talking about, and was so quick to dismiss an issue like this.

I can only offer advise to anyone with BT. If you're having email troubles, they'll probably continue until you find an ISP that understands technology.

What was he thinking?

Anyone paying attention to my AdminExile graph on the homepage will notice that for the past few days there is a ~200 attempt spike in the attempts to access /administrator on this site.

A user at 178.128.110.181 has been doing the same thing for 3 days, over and over, trying to hack this website.

After reporting them to their ISP (DIGITALOCEAN), I blocked them in the firewall.

AdminExile + Fail2Ban

It finally happened. Although I've been running AdminExile with Fail2Ban for a long time, nobody has asked how to do it. Certainly, there are some admins out there who didn't need help. Someone finally asked, and that prompted me to write this document.

This document is for administrators who operate their own servers, and are capable of installing and configuring Fail2Ban. Users who are on shared servers, or use commodity services to host their sites will probably not have access to install or configure Fail2Ban.

Any current (and future) version of AdminExile is capable of this configuration.

AdminExile configuration:

In /administrator, view the AdminExile configuration within the Plugin Manager. The only necessary configuration is to set "Enable Failure Logs" to "Yes"

Fail2Ban configuration:

First, you need to create a filter. This template is a very simple example, because the goal is only to identify the specific lines in the error log which indicate an AdminExile failure.

# Fail2Ban filter to block failed AdminExile authentication 
#

[Definition]

failregex = ^.*?\(<HOST>\) failed to authenticate via AdminExile
ignoreregex =

Second, you must configure Fail2Ban to utilize this filter. There are many ways to do it, but I like to maintain a jail.local file. This file defines which filter is to be used, which file the filter is supposed to monitor, the maximum number of tolerated failures, and the penalty for exceeding that maximum.

[adminexile] 
enabled = true
port = http,https
filter = adminexile
logpath = /var/log/apache2/error.log
maxretry = 2
findtime = 600

Of course, you'll need to adjust this to the location of your server error log.

Once configured, restart Fail2Ban and you should be in business.

Joomla in the Cloud

It's been a long time coming. Some of my customers are seeing growth rates that will soon outpace the ability to do any more vertical scaling of their websites. My own site was feeling the strain of several hundred thousand websites receiving XML update files daily. It was long past time to provide a solution for my customers (and my own sites) that could scale with the expected growth.

This weekend, I turned the keys on a new cluster hosting RicheyWeb.com. This development allows horizontal scaling as my traffic increases. When I reach a level of utilization that means customers and visitors see degraded performance - I can simply add another worker node to the cluster and increase the site capacity.

For my customers, it means that the Joomla sites they've been building upon for years will remain viable for years to come. This type of scaling allows for gradual growth, and even immediate / rapid expansion.

Are you subject to GDPR regulations?

There is an excellent article on Forbes titled "US Businesses Can't Hide From GDPR" and that led me to the question - How many companies are unaware of their exposure?

The answer is actually very easy to determine. If you answer yes to both of these questions, then you are subject to the GDPR regulations.

  1. Are you running a business and engaged in economic activity, and does you business collect or process personal data obtained from EU residents? (Article 4(18))
  2. Does your business have more than 250 employees? (Article 30)

If you answer no to question 2, you might still be subject to GDPR but it would benefit you to consult with legal council.