Web Hosting

There are 10 posts tagged Web Hosting (this is page 1 of 1).

How I Revamped My Web Hosting Business

During the past two months or so, I’ve pushed really hard to finish a project that I’ve been “working on” for the past year. The project was to rename, streamline, and solidify my web hosting business. This included registering the business as a trade name with the state, opening a business bank account, opening a post office box, designing a new website, and setting up and integrating the billing system into the website.

The hosting business originally started as a side-project — something I set up to host my own websites and those of friends and colleagues. I hadn’t anticipated the business growing and not putting much thought into setting it up properly from the start was a mistake.

As the number of hosting clients grew, I found it increasingly difficult to keep track of account balances, payments, contact information, account statuses, etc. I realized that I needed something that would automatically generate invoices and send them out, accept payments through my website, and handle new account signups.

I researched various billing systems and eventually settled with WHM.AutoPilot: It was relatively inexpensive and appeared to have everything I needed. However, after adding all my clients and reconciling the accounts, I struggled to get the system working just the way I wanted. There were many bugs and newer versions weren’t being released in a timely manner. It felt as though the billing software just made things more complicated rather than improving them.

All the while, my hosting business continued to grow. Despite bad accounting and being very unorganized, clients were continuing to roll in by word-of-mouth advertising. Every time a new client would sign up, I felt embarrassed that I didn’t have something more professional to present to them. I was beginning to dislike the thought of setting up new accounts!

That’s when I realized I really needed to streamline everything and make growth less of a burden. (Growth should be easy, not something you dread!) So almost exactly a year ago, I decided to change the name of the business to ActualWebSpace, open a bank account so clients could make checks payable to the business instead of my personal name, and get everything set up properly once and for all.

After researching billing systems once again, I decided on and purchased WHMCS. The developers seemed more “on top of things” and the community around the software appeared very active. It used a very simple PHP templating system that would make integrating the billing system into the website very simple.

So that I could offer domain registration and SSL certificates, I signed up for an eNom reseller account. WHMCS included full integration with eNom, so automating domains and SSL certificates was easy.

For the website, I decided to use WordPress as a back-end and design a WordPress theme from scratch. It was both the first time I had set up WordPress as a CMS and the first time I had designed a WordPress theme from scratch. While it extended the amount of time it took the launch the site, it was an invaluable learning experience.

Setting up the website was probably the most difficult part of the entire process for me. I’m a perfectionist and it’s difficult for me to create something and be happy with it. But, after much persistence, I discovered ways of getting out of my perfectionist state of mind and I made enormous progress in the past few weeks.

I launched the new website for ActualWebSpace yesterday. It has been almost a year in the making and it feels great to finally follow through with a project to the end. I’m going to use the lessons I learned to start, and finish, my next project (still deciding what that will be).

The biggest lesson I learned from this year-long project: When starting any kind of venture that has the potential to grow, set things up properly from the beginning. Plan for and anticipate growth instead of waiting until the growth begins to happen.

Bounce-back Spam (Backscatter)

I really hate bounce-back spam! (I call it bounce-back spam, but the official name for it is Backscatter.) I've read, and been told by sysadmins, that there is not much that can be done about it. The Wikipedia page on bounce messages has a little section that explains why:

Excluding MDAs, all MTAs forward mails to another MTA. This next MTA is free to reject the mail with an SMTP error message like user unknown, over quota, etc. At this point the sending MTA has to inform the originator, or as RFC 5321 puts it:

If an SMTP server has accepted the task of relaying the mail and later finds that the destination is incorrect or that the mail cannot be delivered for some other reason, then it MUST construct an "undeliverable mail" notification message and send it to the originator of the undeliverable mail (as indicated by the reverse-path).

This rule is essential for SMTP: as the name says, it's a simple protocol, it cannot reliably work if mail silently vanishes in black holes, so bounces are required to spot and fix problems.

Today, however, most email is spam, which usually utilizes forged Return-Paths. It is then often impossible for the MTA to inform the originator, and sending a bounce to the forged Return-Path would hit an innocent third party. This inherent flaw in today's SMTP (without the deprecated source routes) is addressed by various proposals, most directly by BATV and SPF.

It looks like I'll have to just deal with it. (I could set up filters and such, but then I might miss a real bounce-back and not know that my message didn't go through!) I'm just grateful it comes in waves of a few hours every few weeks instead of non-stop! Has anyone else had to deal with this? If so, what did you do about it?

Backscatter Spam

WHM Whitelist to Exclude from Exim Sender Verify Callbacks

Sender verification is an important feature used by email servers to help prevent spam. When sender verification is enabled, the receiving email server checks to make sure the sender exists. Various email servers have different ways of handling this feature. Exim, for example, uses a mechanism called 'sender callouts' or 'callbacks'. (When the sending server does not accept a verification request, it does not comply with RFC 2821.)

However, in the event that the network route from the receiving email server to the originating email server is broken (or a firewall blocks the connection), the result can be a bit confusing. The receiving email server treats a failed verification as a failed verification, regardless of whether or not it could even connect to the originating server. This means the email never comes through to the recipient. After all, as far as the email server knows, it's spam.

One of my hosting clients was experiencing this "lost email" problem and a quick grep at /var/log/exim_mainlog confirmed the problem (hosts and IPs changed for obvious reasons):


2008-11-17 15:02:27 [30121] H=relay1.example.com (qsv-spam1.example.com) [67.26.151.59]:36752 I=[69.161.211.25]:25 sender verify defer for : could not connect to customer.example.com [163.112.75.15]: Connection timed out
2008-11-17 15:02:27 [30121] H=relay1.example.com (qsv-spam1.example.com) [67.26.151.59]:36752 I=[69.161.211.25]:25 F=<[email protected]> temporarily rejected RCPT <[email protected]>: Could not complete sender verify callout
2008-11-17 15:02:27 [30120] H=relay1.example.com (qsv-spam1.example.com) [67.26.151.59]:36751 I=[69.161.211.25]:25 incomplete transaction (RSET) from <[email protected]>

As you can see, the email server was unable to connect to customer.example.com to verify the existence of the sender ([email protected]). This doesn't mean the sending server doesn't verify callbacks, but rather that the network connection from my server to the sending server could not be established.

Most of the stuff I found online related to solving this problem on a server running WHM (here and here) explain how to modify exim.conf to add special whitelist rules. Luckily, my server is running WHM 11.23.2 and has a whitelist option that makes it really easy to exclude a particular IP address from sender verification without any manual changes to exim.conf:

1. Click Service Configuration -> Exim Configuration Editor
2. Under Access Lists, find "Whitelist: Bypass all SMTP time recipient/sender/spam/relay checks" and click [EDIT]
3. Add the IP address for the sending server for which you wish to skip sender verification (as the note at the bottom explains, hosts cannot be used in this list)
4. Click Save
5. Click Save again near the bottom of the Exim Configuration Editor page

That's it! Now any emails from that IP that were failing to come through because of a sender verification failure will come through without a problem (again, you can watch /var/log/exim_mainlog to confirm).

Switching to suPHP; What a Mess!

When one of my users reported problems deleting files he had uploaded using a PHP script, I quickly discovered all the files being uploaded were owned by the user running the web server: nobody. This meant only the root user could delete those files.

Apache suEXEC is commonly used to resolve this problem. It allows Apache to run as the user who owns the domain being accessed. This way, files created by PHP would be owned by the user owning the site instead of the default nobody user.

However, Apache suEXEC only works if you're using CGI as the PHP handler. The PHP5 handler on my server was set to use CGI, but I have PHP4 configured as the default PHP version and it was configured to use DSO. When I tried changing PHP4 to use CGI as the handler, most of the domains on my server displayed this:

Warning: Unexpected character in input: '' (ASCII=15) state=1 in /usr/local/cpanel/cgi-sys/php4 on line 772
Warning: Unexpected character in input: ' in /usr/local/cpanel/cgi-sys/php4 on line 772
Warning: Unexpected character in input: ' in /usr/local/cpanel/cgi-sys/php4 on line 772
Warning: Unexpected character in input: ' in /usr/local/cpanel/cgi-sys/php4 on line 772
Parse error: syntax error, unexpected T_STRING in /usr/local/cpanel/cgi-sys/php4 on line 772

OK, that looks like a problem with cPanel. I don't have time to debug cPanel's problems.

suPHP, like suEXEC, is used to run Apache as the user who owns the domain. I decided to try recompiling Apache and PHP with suPHP enabled to see if that would fix the problem.

File Ownership Hell

suPHP worked, except now the sites using PHP sessions were trying to access stored session data in /tmp/ that was owned by the user nobody! So I deleted all the session data and that allowed the PHP sites to create new session data with file ownership of the user owning the domain.

But then I tried accessing my WordPress admin page and started getting permission denied errors in /wp-content/cache/. Same problem: the cache files that had been created before I enabled suPHP were owned by the user nobody and now the user who owns my domain couldn't access them. A quick chown -R raamdev:raamdev /wp-content/cache/ fixed that problem.

Yeah, I could simply chown -R [user]:[user] /home/[user] for each of the users on the server, but there's something about running a recursive command on files I've never seen, and know nothing about, that makes me uncomfortable.

More suPHP Limitations

I was beginning to worry that this was going to be more difficult than simply enabling suPHP and I wondered how many other sites I'm hosting could have similar problems. I tried accessing one of the high priority sites I'm hosting and discovered it was broken and displaying an "Internal Server Error".

After a little research, I discovered that you cannot use php_value directives in .htaccess files with suPHP. The .htaccess file included with (created by?) Joomla! contained this at the bottom:

#Fix Register Globals
php_flag register_globals off

I already knew register_globals was turned off in the global PHP configuration, so I simply commented out that line and the site started working again.

Conclusion

It was at this point that I concluded it was too risky to just blindly enable suPHP while hosting over 50 domains, many of which I am not at all familiar with what's being used or hosted. I will need to take the time to carefully crawl through all the sites making sure their .htaccess files don't contain anything that might disrupt suPHP and then confirm all the sites are still working properly.

Lesson learned: Setup suPHP before you're hosting 50+ domains.

LiquidWeb Dedicated Server Support Rocks!

A rouge mail server started using my server to send tons of spam messages. After checking a few obvious things (rouge scripts on my server sending spam, users sending the spam, etc), I immediately submitted a ticket to the data center, LiquidWeb. Within 30 seconds, D. Walters responded to the ticket:

Hey Raam,

I'm logging into your server right now to check things out. If you're
still logged in, feel free to follow my work with this command:

screen -x dwalters

Regards,
D. Walters

I was already logged into the server, so I typed screen -x dwalters and sure enough, I instantly saw what he was doing on the server (I never knew this was possible with screen). I watched as he did various things, including installing a custom script that he wrote to help handle spam on server.

After a minute or two, he started typing something in the terminal that didn't look like any command I had ever seen:

web:~ root# Hey Raam, are you there?

Since I was attached to the same terminal as him, I was able to type back. Over the next 20 minutes we spoke back and forth several times and exchanged geeky jokes. I asked him a couple of questions and he showed me example commands that I could run to glean useful information from the exim logs.

The name D. Walters was familiar to me, as he had responded to several of my LiquidWeb tickets in the past. He was familiar with my technical abilities and knew that I was Linux savvy, so he invited me to watch what he was doing. He trusted that I wouldn't foolishly try typing in the terminal thereby interrupting what he was doing.

Dealing with someone familiar is important as it brings a feeling of trust to the situation. I compare the feeling to that of my auto mechanic; he lets me stay in the garage while he works on my truck. I stay out of the way and in return I get to stand under the truck, walk around the garage, and watch as he goes about his work. I simply wouldn't be able to do that if I went to a Chevy dealership.

The response time at LiquidWeb is second to none. I can always get someone technically able on the phone, 24/7. That person is always within walking distance to my physical server on the rack. Add those things to my recent experience and I can honestly say, LiquidWeb dedicated server support rocks!

Installed DenyHosts to Help Prevent SSH Attacks

When the LogWatch report from yesterday (for web.akmai.net) arrived in my Inbox, it had over 20,000 failed SSH login attempts. Today I decided it was finally time to do something about all those attacks.

After looking around a bit, I found several different solutions. Some solutions utilized firewall rules and others monitored your /var/log/secure (or /var/log/auth.log) log files for multiple failed login attempts and then added those IPs/Hosts to the /etc/hosts.deny file.

I decided to go with the latter method and quickly found a nice tutorial for setting up DenyHosts (be sure to download the latest version (2.6 as of this writing) instead of the older version 2.0). Rather than reinvent the wheel, here is what the DenyHosts website says about itself:

What is DenyHosts?

DenyHosts is a Python script that analyzes the sshd server log messages to determine what hosts are attempting to hack into your system. It also determines what user accounts are being targeted. It keeps track of the frequency of attempts from each host.

Additionally, upon discovering a repeated attack host, the /etc/hosts.deny file is updated to prevent future break-in attempts from that host.

An email report can be sent to a system admin.

Since I was setting up DenyHosts on a RedHat-based machine (CentOS) and not a Debian-based machine, I needed to change this line:

update-rc.d denyhosts defaults

to this:

chkconfig denyhosts --add

Other than that, the installation steps were just as the tutorial described. I decided to enable the ADMIN_EMAIL option so that I would receive an email every time something was added to hosts.deny, but within minutes of starting DenyHosts I had a dozen attacks with a dozen emails on my BlackBerry. I had to disable ADMIN_EMAIL to stop the spamming!

To make sure DenyHosts was working properly I tried logging in with the wrong password three times. When I tried to connect again, here is what I received:

ssh [email protected]
ssh_exchange_identification: Connection closed by remote host

DenyHosts also has the ability to report to a central server the hosts that are trying to break in and you can also download a list of hosts that have been reported by others. I choose to opt out of doing this for now. The DenyHosts statistics page is pretty cool. Notice how the majority of the hosts come from China? Hmm.

UPDATE:
I quickly discovered that DenyHosts was adding my IP address to the hosts.deny file. When I watched /var/log/secure I discovered the problem:

Jun 13 20:18:46 web sshd[5959]: reverse mapping checking getaddrinfo for 75-147-49-211-newengland.hfc.comcastbusiness.net failed - POSSIBLE BREAKIN ATTEMPT!
Jun 13 20:18:46 web sshd[5959]: Accepted publickey for fooUser from ::ffff:75.147.49.211 port 57926 ssh2
Jun 13 20:18:48 web sshd[5994]: Did not receive identification string from ::ffff:75.147.49.211

I'm not entirely sure how to fix this, but for now I added my IP address to /usr/share/denyhosts/data/allowed-hosts (I had to create this file) which prevents DenyHosts from blocking my IP no matter what (see this FAQ for more info). Also, I had to restart DenyHosts (/etc/init.d/denyhosts restart) before the change to allowed-hosts took effect.

Research Before Committing

For the past few months, I've been meaning to purchase the latest version of WHM.AutoPilot, the web host billing software I use for Akmai.net. There are lots of bugs in the current version and many of my clients have complained about the problems with the billing system (it doesn't work). I finally put it on my calendar and told myself I would purchase the $180 upgrade and get it installed this week.

I was on the WHM.AutoPilot website with the V3 Owned License in my shopping cart ready to checkout when I realized that I should probably do some research to see what other web host billing software was out there. I wasn't even aware of any other options, but with the size of the web hosting industry I knew there had to be competing products.

I quickly came across a couple of different ones, namely ModernBill, ClientExec, and WHMCS. After spending about half an hour searching Google and reading forum posts, I concluded that ModernBill was definitely the most popular and most recommended. But realizing that I had only known of the existence of these products for thirty minutes, I decided to spend some more time researching.

All of these applications have demo areas where I can try the admin and client areas, so I spent some time playing around with them. I then decided to do in-depth searches for recent forum discussions on these products. The software world changes so quickly.

What did I learn? I learned that ModernBill was recently bought by a bigger company (Parallels) and that the ModernBill software has been very buggy since the release of V5. Wow, and a few minutes ago I was thinking that it was the best option for my money! To add to my shock, almost everyone on the Web Hosting Talk forums seemed to think of WHM.AutoPilot as the "beginners web host billing software".

It's looking like I will probably go with WHMCS for my new billing software, but not before I do more research!

Web hosting is not for everyone

As you may know, I run my own web hosting business called Akmai.net Web Hosting (soon to be CORBAWeb) and I host about 45 active domains for a small but dedicated base of 15-20 clients. Running a web hosting business is not particularly difficult, especially with software like CPanel (to give the customer easy access to common domain related functions like email, subdomains, etc), WHM (to allow the administrator to control nearly all aspects of running a web server, including DNS, shell access, etc) and WHM.AutoPilot (to assist with billing, invoicing and automatic account creation).

You might be thinking, "if running a web hosting business was so easy why wouldn't everyone be doing it?". Everyone is doing it and that is the reason 90% of the email on the Internet is spam! There are so many inexperienced web host administrators who don't understand the technology behind the software they're using because wonderful applications like CPanel and WHM remove that requirement (don't get me wrong, I love CPanel and WHM). All the people who jump at the chance to run their own web hosting business need to understand there is more to it than just creating accounts and watching your Paypal balance increase -- there is great responsibility that comes with running a web hosting business and there is no room for incompetence.

Let me give you an example. Late this morning my Blackberry beeped to indicate an incoming email. No big deal -- I hear that beep dozens of times throughout the day. But the beeping didn't stop -- it kept beeping as if it was an alarm. Sure enough, I had 12 "Mail Delivery Failed" messages. Then 13. Then 14. After about 40 seconds it was up to over 100 messages. I instantly knew what this meant. Someone, or some thing, was sending a huge number of emails from my web server and the vast majority of those were bouncing back because the recipient email address was invalid. A quick check of the server showed over 20,000 emails had already been sent.

With the help of an on-site engineer, at the data center where my server is located, I was able to track down the origin of the email spamming. It was coming from a mail form installed on one of the domain on my server. The form wasn't anything harmful, and neither was the domain (nor the person who owned the domain), but the mail form wasn't secure. It didn't have any type of captcha installed to prevent a spam bot from submitting endless requests to the script. A spam bot crawling the web for insecure forms found the script hosted on my server and started using it to send a 'Paypal Account Notice' email designed to phish account details from the recipient. I quickly deleted the script from my server and had any remaining messages purged from the mail queue.

This is a perfect example of how incompetent web host server administrators are to blame for all the Internet's spam. If I didn't allow myself to be bothered on my Blackberry with all the "Mail Delivery Failed" messages for my server (including legitimate ones), I wouldn't have discovered this was happening as quickly as I did. Most people simply let those emails drop into an Inbox somewhere and forget about them. If 20,000+ messages were sent out in the 5 minutes it took me to discover and fix the problem, how many messages would have been sent out if I didn't discover the problem for a few hours? Or a few days?

You cannot blame the creator of the mail script, because while the programmer might understand that his script needs additional security before being used in the real world, a web designer will simply upload the script to a web server and expect it to work. This means that there will always be instances where a faulty script is utilized in a malicious way by someone with bad intentions. So who is responsible? The system administrator is responsible. It's his job to make sure everything runs smoothly and there is no room for incompetence. How many web host administrators regularly read their logs for suspicious activity or broken software? I read akmai.net's logs on a daily basis.

Web hosting is not for everyone because many people lack the technical understanding, the competence, and the time required to properly manage a web server. If you're running your own personal web server at home, fine. If you're running your own mail server, I hope you know what you're doing. If you're running a web server that's located in a data center with lots of bandwidth and you're hosting domains, email, and DNS for people you don't know very well, then you'd damn well better know what you're doing and understand the nasty things that exist out there on the Internet. You will be attacked. Be prepared.

A missing procedure for DNS changes

If you noticed the error message on my blog during the past few hours, you may be wondering what that's all about. Well, I'm moving my web hosting company, akmai.net to it's own dedicated server. I have been coding a support site for Aerva and they agreed to host it with my hosting company. We worked out a deal so that I can purchase a dedicated server and Aerva would pick up the tab on the extra expense. In return, I provide Aerva's support site with all the bandwidth and space they need. My web hosting business went from making a profit to just breaking even, but now that I have a dedicated server, I can offer more hosting packages at better prices and hopefully pick up a few more clients. Here's the specs on the new server for anyone who's interested:

    Pentium 4 3.0Ghz DUAL CORE
    2GB RAM
    Two 120 GB SATA hard drives (one for backup)
    1600 GB of monthly bandwidth

Oh, back to why my blog has been out of service. I figured since I was only moving from one account type (VPS1) to another (dedicated server) within the same web host (LiquidWeb) that I wouldn't need to change any DNS or nameserver addresses. Well, come to find out, I needed to update the IP addresses for my two nameservers, ns1.akmai.net and ns2.akmai.net. So I put in a request with my registrar to have the IPs updated, and a few hours later received a reply informing me the update had been submitted. Due to the nature of the Domain Name System (DNS), updates across the entire Internet run rather slow... sometimes a few days slow. You would think with all the incredibly fast Internet connections available around the world that DNS updates would be nearly instantaneous!

Now I wouldn't really mind the DNS updates running slow if it didn't cause major problems with web hosts. I host over 20 domains, and all of those domains are set to use my two nameservers (ns1 and ns2.akmai.net). When I change my nameserver IP addresses to point to a new server, I have no way of knowing when all of the computers around the world are going to get the new information, and therefore point to the new server. So, on Sunday, the tech at LiquidWeb copied over all the accounts from my old VPS1 server to the new dedicated server. That's great, but for now anyone uploading new files to their domain, or making changes to their databases (as is the case for a forum, or blog) will be making those changes on the old server. Suddenly, a few days later, their computer will recognize the new nameserver IP address and start pointing their domain to the new server... which contains all the data as of Sunday! That means everything that changed between Sunday and whenever the nameserver change comes into effect, is lost! They won't see any emails, blog posts, forum posts, or files that were changed since Sunday. That is unacceptable! To make things worse, if they make a change to their domain after their domain is pointing to the new server, but before I copy over their account again to restore the missing data, then there wont be any easy way to restore the missing data from the days in between. There isn't any easy way to simply "merge" the missing data with the current data; I need to actually replace the data.

Lucky for me, most of my clients are people with personal domains, not business crucial, high-traffic domains (although I do have several important business domains, so it's still an issue).

So what do I think should be done about this? Well, something on the web host level could be done. The web host could set their internal DNS to point the new IP addresses to the old server. Then after a few days, when we're sure the DNS has propagated through the system and everyone's computers have the newly updated nameserver info, we pick a good time, late at night for example, to lock all the domains from having any changes made to them. Then we copy over all accounts, databases, and data over to the new server, have the web host switch their DNS to point those new nameserver addresses to the new server, and unlock all the domains. Instantly, everyone will be accessing the domains on the new server. Problem solved.

I don't pretend to be an expert on DNS, or the Internet for that matter, but if anyone sees a problem with this solution I'd be interested to hear your argument. In fact, now that I look back, I realize this really isn't a problem with the Domain Name System, but rather with the procedures of web hosts. Don't get me wrong, LiquidWeb is an awesome web hosting company and their technical support staff is very helpful and knowledgeable. But of all the web host's I've been with, this problem with DNS has never been addressed by any of them. It's always "Wait 48 hours for the changes to take effect. Until then, all you can do is wait.". Even if this solution cannot be easily applied in the case of moving a domain from one web host to another, it should be easily addressed when moving a domain from one server to another inside the same web host.