Yahoo DNS Issues Cause Problems in the United States

Yahoo! appears to be inaccessible to people in the US. Visiting yahoo.com redirects to www.yahoo.com and fails to load. I confirmed it was at least somewhat limited to the US by trying the connection from a shell account on a server in Europe.

Using dig (a Unix DNS lookup utility), we can see from within the United States that there is a problem with DNS. There is no A record with an IP address listed in the ANSWERS section:

;; QUESTION SECTION: ;www.yahoo.com. IN A

;; ANSWER SECTION: www.yahoo.com. 129 IN CNAME www.wa1.b.yahoo.com.

And from the server in Europe:

;; QUESTION SECTION: ;www.yahoo.com. IN A

;; ANSWER SECTION: www.yahoo.com. 272 IN CNAME www.wa1.b.yahoo.com. www.wa1.b.yahoo.com. 33 IN CNAME www-real.wa1.b.yahoo.com. www-real.wa1.b.yahoo.com. 33 IN A 209.191.93.52
;; AUTHORITY SECTION: wa1.b.yahoo.com. 273 IN NS yf2.yahoo.com. wa1.b.yahoo.com. 273 IN NS yf1.yahoo.com.

If you try connecting directly to the missing IP address, you should at least be able to get the main Yahoo page: http://209.191.93.52. You might also try temporarily adding an entry to your /etc/hosts or C:Windowssystem32driversetchosts if you want to continue being able to use the FQDN.

UPDATE: As of 15:50 EST, Yahoo appears to be working again. The outage appeared to start around 15:11 EST, so that's a good 40 minutes of downtime.

Reverse DNS: That's not me!

I have Speakeasy DSL at home with a static IP address (I'm boycotting Comcast). I run a Linux server on a Mac Mini and I use it for all my messaging (using naim IRC/AIM and Jabber via Bitlbee, but that's for another post).

Since I SSH into my Linux box several times a day, it would be nice to avoid typing the full IP address each time. So I decided to setup an A Record on one of the domains I own (we'll use dev82.org as an example) so that dev82.org points to the IP address of my home DSL connection (66.92.25.92 in this example).

After transferring the Speakeasy DSL to my new apartment in Cambridge, I had a new IP address. No problem -- I simply updated the A Record and dev82.org worked again. However, this time I noticed something funky. Take a look at what hostname my IP address resolved to when I pinged dev82.org:

raam@wfc-main2:~$ ping dev82.org PING dev82.org (66.92.25.92) 56(84) bytes of data. 64 bytes from host103-spk.online-buddies.com (66.92.25.92): icmp_seq=1 ttl=53 time=38.3 ms

That's weird. What the hell is host103-spk.online-buddies.com? A little Googling tells me:

"Online Buddies, Inc., developer of MANHUNT.net is one of largest developers of web-oriented services serving the gay community."

Uh, I'm not gay. Besides, why the hell is my home DSL IP address responding to a domain I've never heard of? I ran a few more tests, including tests from different ISPs to rule out a local DNS issue. Each time, my home IP address resolved the same:

raam@wfc-main2:~$ nslookup 66.92.25.92 Non-authoritative answer: 92.25.92.66.in-addr.arpa name = host92-spk.online-buddies.com.

So I decided to pick up the phone and call Speakeasy. They have always been helpful and I figured worst-case scenario is that I'll have to request a new IP address and re-point dev82.org. At least that way my IP address won't resolve to some gay site.

I called Speakeasy at 3:30am and had a tech on the phone within 3 minutes (Speakeasy rocks). I explained to the tech my situation and he quickly had an explanation: Whoever was assigned my IP address before I was must have had it set up to resolve to that hostname.

I told the tech I had setup an A Record to point the IP address to dev82.org. After confirming that was true with a ping test, the tech said he would update the Reverse DNS record so that 66.92.25.92 resolves to dev82.org.

This is awesome. I had no idea I could request an update to the Reverse DNS record for my static DSL connection! I wonder how easy that process is with a Comcast connection, or if it's even possible. 😕

A missing procedure for DNS changes

If you noticed the error message on my blog during the past few hours, you may be wondering what that's all about. Well, I'm moving my web hosting company, akmai.net to it's own dedicated server. I have been coding a support site for Aerva and they agreed to host it with my hosting company. We worked out a deal so that I can purchase a dedicated server and Aerva would pick up the tab on the extra expense. In return, I provide Aerva's support site with all the bandwidth and space they need. My web hosting business went from making a profit to just breaking even, but now that I have a dedicated server, I can offer more hosting packages at better prices and hopefully pick up a few more clients. Here's the specs on the new server for anyone who's interested:

Oh, back to why my blog has been out of service. I figured since I was only moving from one account type (VPS1) to another (dedicated server) within the same web host (LiquidWeb) that I wouldn't need to change any DNS or nameserver addresses. Well, come to find out, I needed to update the IP addresses for my two nameservers, ns1.akmai.net and ns2.akmai.net. So I put in a request with my registrar to have the IPs updated, and a few hours later received a reply informing me the update had been submitted. Due to the nature of the Domain Name System (DNS), updates across the entire Internet run rather slow... sometimes a few days slow. You would think with all the incredibly fast Internet connections available around the world that DNS updates would be nearly instantaneous!

Now I wouldn't really mind the DNS updates running slow if it didn't cause major problems with web hosts. I host over 20 domains, and all of those domains are set to use my two nameservers (ns1 and ns2.akmai.net). When I change my nameserver IP addresses to point to a new server, I have no way of knowing when all of the computers around the world are going to get the new information, and therefore point to the new server. So, on Sunday, the tech at LiquidWeb copied over all the accounts from my old VPS1 server to the new dedicated server. That's great, but for now anyone uploading new files to their domain, or making changes to their databases (as is the case for a forum, or blog) will be making those changes on the old server. Suddenly, a few days later, their computer will recognize the new nameserver IP address and start pointing their domain to the new server... which contains all the data as of Sunday! That means everything that changed between Sunday and whenever the nameserver change comes into effect, is lost! They won't see any emails, blog posts, forum posts, or files that were changed since Sunday. That is unacceptable! To make things worse, if they make a change to their domain after their domain is pointing to the new server, but before I copy over their account again to restore the missing data, then there wont be any easy way to restore the missing data from the days in between. There isn't any easy way to simply "merge" the missing data with the current data; I need to actually replace the data.

Lucky for me, most of my clients are people with personal domains, not business crucial, high-traffic domains (although I do have several important business domains, so it's still an issue).

So what do I think should be done about this? Well, something on the web host level could be done. The web host could set their internal DNS to point the new IP addresses to the old server. Then after a few days, when we're sure the DNS has propagated through the system and everyone's computers have the newly updated nameserver info, we pick a good time, late at night for example, to lock all the domains from having any changes made to them. Then we copy over all accounts, databases, and data over to the new server, have the web host switch their DNS to point those new nameserver addresses to the new server, and unlock all the domains. Instantly, everyone will be accessing the domains on the new server. Problem solved.

I don't pretend to be an expert on DNS, or the Internet for that matter, but if anyone sees a problem with this solution I'd be interested to hear your argument. In fact, now that I look back, I realize this really isn't a problem with the Domain Name System, but rather with the procedures of web hosts. Don't get me wrong, LiquidWeb is an awesome web hosting company and their technical support staff is very helpful and knowledgeable. But of all the web host's I've been with, this problem with DNS has never been addressed by any of them. It's always "Wait 48 hours for the changes to take effect. Until then, all you can do is wait.". Even if this solution cannot be easily applied in the case of moving a domain from one web host to another, it should be easily addressed when moving a domain from one server to another inside the same web host.