A data center migration from eNom web hosting provider caused unexpected domain resolution problems that are expected to last for a few hours.
Customers started to complain that they could no longer access their websites and emails due to Domain Name System (DNS) issues.
Enom services are down impacting access to websites and email.
— Ken Westin (@kwestin) January 16, 2022
My google apps gmail is not getting email, turns out DNS is not working because @enom is doing "a datacenter move" that ran into problems. What medieval times are these when a datacenter move brings down DNS for organizations? Advance warning would have been nice @enomsupport.
— Pieter Viljoen (@pieter727) January 16, 2022
The company said that it received reports of domains using eNom nameservers that were failing to resolve and acknowledged the problem.
"We are receiving some reports of domains using our nameservers which are failing to resolve. Owing to the migration we are unable to research and fully address the issue until the migration is complete. This is not an expected outcome from the migration, and we are working to address it as a priority" - eNom
After looking into the issue, the company has found that the domain resolution problems are affecting a few hundred domains. eNom estimates that the outage would last for a few hours.
eNom customers affected by the problem can't change the nameservers because eNom is down and all they can do is wait for the migration to complete:
"What's absolutely nuts about this is we can not change our NameServers due to eNom itself is down (going on 24 hours now). Never in my life have I been in a situation where I can't route around network issue at the root domain records level. We are totally trapped!" - eNom customer
An initial maintenance notice from the company said that the data center migration would take 12 hours, between 6 AM PST and 6 PM PST on Saturday, January 15.
During this period, enom.com and enomcentral.com would be unavailable and customers would not be able to log into their account from the web interface and the API would be down.
While some management actions would not be possible, the company said that DNS resolution and email services would remain operational.
In subsequent updates on the progress of the data center migration progress, the company announced multiple extensions for completing the process, the latest one being for January 16, 12:00 PM PST.
Comments
genesalvatore - 2 years ago
Hi @IONUT ILASCU I can assure you that there are more than "a few hundred" domains that are effected by this outage. ENOM is heavily downplaying this outage, providing no updates of any significance to their direct customers, or resellers like myself.
We were NOT informed of this outage in advance. Had we been informed, we would have had the opportunity to put mitigation plans into place.
As of the time of this reply to your post, we have had hundreds of customers websites and thousands of email users offline for 36 hours and counting.
This is a terribly planned and implemented migration and is harming customers financially and reputationally.
forenzic - 2 years ago
There WAS a notification sent out by eNom via email on 12 January. I can't say how many received it, though. I certainly did. Notification mentioned a 12-hour maintenance window from 6am-6pm PST on January 15.
AND THIS IS IMPORTANT - The notification also said this:
DNS will continue to resolve normally
Email will remain fully operational; your customers will see no impact to their service
But NO, that's not what happened. Things are still substantially down 36 hours and counting since the beginning of the maintenance window.
This has harmed many website operators and entities. And I agree, the number harmed is more like millions than hundreds.
genesalvatore - 2 years ago
As a reseller we 100% did not receive it. We archive all inbound and outbound email and searched our archives and spam traps and never received it - otherwise we would have put mitigation policies into place. I can say that in searching the email archives 1 of our customers that has their own ENOM account received the email. We are not the only resellers and/or customers that did not receive the email, just take a look at twitter and search for @enom and @enomsupport and you will see this.
forenzic - 2 years ago
Now 39 hours since the beginning of their maintenance window....
I have no doubt that many didn't receive eNom's email. Since I've been with eNom approximately forever in Internet time, with relatively minor issues, I (and many others) never expected this outage to be much more than a short-term annoyance once the trouble began, whether they received the maintenance notice or not. Hope this doesn't prove to be the 737-800 Max of Internet outages. But there is a chance it will be the safest of registrars after they fully come back up. Except at this point it feels more like "IF they fully come back up" than After...
FUUCK-ENOM - 2 years ago
Incompetence on the highest level, who migrates to new data center without knowing the issues they'll face. Enom are holding our domains hostage, many e-commerce site are down and not processing orders because of this stupidity. This is not to be taken lightly, I see a class action lawsuit coming.
flubadub - 2 years ago
I work for a website provider with a few hundred clients. I came to this article when I got alerts that all of my Enom customers' sites were down, and I started Googling for answers. It's definitely widespread.
forenzic - 2 years ago
And I still can't login to the eNom portal.
webhoo - 2 years ago
Ionut is just the messenger and the first to report but NO enom did not send a warning and now that its WAAAAAY over 12 hours so it's clearly a phuqup. Doing this since 1994 I have seen this at least 12 times and godaddy had this twice when they started. It really sucks that its taking so long but I'm grateful this article was written and on google so fast so my clients can see it's not the website or our hosting but their choice of domain provider.
arifsaha - 2 years ago
Still not working.
Actually, only my domains that use eNom's DNS (dns*.name-service.com) don't work. Domains that use different DNS are working normally.
Indeed, seems the issue is with their DNS:
$ for n in DNS{1,2,3,4,5}.NAME-SERVICES.COM ; do echo == $n == ; host example.com $n ; done
== DNS1.NAME-SERVICES.COM ==
;; connection timed out; no servers could be reached
== DNS2.NAME-SERVICES.COM ==
;; connection timed out; no servers could be reached
== DNS3.NAME-SERVICES.COM ==
;; connection timed out; no servers could be reached
== DNS4.NAME-SERVICES.COM ==
;; connection timed out; no servers could be reached
== DNS5.NAME-SERVICES.COM ==
;; connection timed out; no servers could be reached
arifsaha - 2 years ago
Seems like the DNS issue resolved now.
The management page is still broken, though.
Truthinshredding - 2 years ago
I have two websites with enom. One has remained online, whilst the other continues to be offline.
I have experience in the domain, you don't treat DNS lightly, particularly if you are moving it. Seems to me the project seems to have been poorly run. I don't see a roll back or fail over being enacted. In addition, I didn't get an email.
When these finally come back, I will be moving my names to a professional service.. I would expect management heads to roll for a failure of this magnitude.
amaxi - 2 years ago
The DNS login page and even the DNS resolution is still not working. It would be nice to restore back the DNS service.
edrokov - 2 years ago
DNS issue is not resolved for now.
Domain management page reports 'Failed to retrieve domain list' to me, so I can't change NS record to make my domains work!
Enom is lying on https://enomstatus.com/ where all services are green.
They are to be punished for the lie. I'll close my account and move domains to another registrar.
dialsoft - 2 years ago
This DNS outage is still widespread and none of my accounts are accessible in my account. I can finagle the ability to login but it says Invalid client IP: 64.98.133.66 I dont know where that is coming from. no domains are accessible to change anything.
WorryboutEnom - 2 years ago
I doubt it is limited to a few hundred. I am a random dude in Minnesota and my website and email have essentially vanished, leaving customers and partners with this response: "Address not found Your message wasn't delivered to ______ because the domain __________couldn't be found."
amaxi - 2 years ago
It is horrible. I don't understand why don't they load back the backup?????
user0011222 - 2 years ago
eNom is making it so my website is down completely, this is so totally unacceptable. I will be transferring my domain out of eNom as soon as I can and filing a complaint with the BBB. I think everyone else should as well!
internetnews - 2 years ago
Still down but you can log into their site now and change the root DNS Servers to another company and then copy your host records and email (MX) records to the other company. But, even that root server change will take a while to propagate so consider putting in enom's server as your first server and then the second company's servers after that, so you'll get a combination of the two in case enom fixes on their side before the root propagation resolves.
Also, anyone else wondering why they seemingly migrated of all 5 of their DNS Servers at the same time? Why not do the first one and leave the other four? It defeats the idea of redundancy to make a datacenter change to all five at once! Who thought that was a good idea?
And, we also did not receive advance notice of this migration, especially for DNS, else we would have already put in another company as the DNS provider. This was another huge management mistake to do a DNS migration without telling anyone in advance so they could prepare.
And, we cannot reach anyone there by phone. Phones keep hanging up on the first ring. We got through twice but then waited on hold endlessly without a human. Shouldn't they try to have their phones working when their DNS is not?
And, why did they clear their status page as if this is just some new issue instead of some multi-day issue?
I am shocked by the decisions they've made. We are suffering huge losses as a result.
story9797 - 2 years ago
"I doubt it is limited to a few hundred. I am a random dude in Minnesota and my website and email have essentially vanished, leaving customers and partners with this response: "Address not found Your message wasn't delivered to ______ because the domain __________couldn't be found.""
story9797 - 2 years ago
Damnit this board just wiped out my entire reply and it was failry long. you can't even edit your post?
genesalvatore - 2 years ago
I usually copy and paste into notepad before adding a long comment, or write it in notepad 1st. So many web forms act that way
genesalvatore - 2 years ago
So here is a ridiculous update on how to get notified by Enom for outages, maintenance, etc You have to SIGN UP for notices from the Subscribe link on enomstatus.com -- what another ridiculous, massive FAIL.
That's why none of us got the emails about the scheduled maintenance window.
Another massive FAIL for the architects, admins, devops and management that oversaw this migration. How could you NOT send notices to ALL your customers, ALL your resellers. How could you NOT post banner messages and/or pop up in the reseller and customer control panels.
This is such a failure on so many levels and people should be fired over such a massive screw up.
Who thinks we're going to get a full public post mortem on this? Not that it will help us, anyway
edrokov - 2 years ago
On the third day of migration they have finally managed to make Dashboard page to show domain list, however clicking domain name results in the error popup 'XMLHttpTransport Error: 500 Internal Server Error'. This is insane. I really hope that someday we will get a working system (in a week time from now considering how things develop?) and migrate to some reliable registrar.
internetnews - 2 years ago
Their status page now says "Any DNS update requests have been backlogged and will be provisioned as soon as we’ve implemented our system-wide solution." Does this mean they are also not processing root DNS server updates to change to another company?
internetnews - 2 years ago
They're not responding to any emails and when you try to open a ticket on their site, it just stays on the same page with "The remote server returned an error: (500) Internal Server Error." Are we all just being held hostage here? Phones and email and support ticketing not working. Does someone have a name and phone number to a live human there? Having a problem is one thing but how they handle us throughout is another and right now both are a big fail.