On the importance of the user experience

One man's opinion on why CPanel is clearly broken

Posted by Trevor Pott on Aug 13, 2016 in Blog | Comments Off on On the importance of the user experience

Yesterday, after having spent many frustrating hours trying to resolve a problem with a VM I have had handed off to me, I tweeted “Wow. Cpanel is awful. O_O”. The developers, rightly, were not pleased to hear this and invited me to share with them my issues.

As a strong – some might say annoyingly persistent – advocate of direct engineering/developer engagement with end users, and of the customer advocacy approach to community engagement, it would be hypocritical of me to ignore this request. Whatever I may think of Cpanel the application, developers reaching out to people they don’t follow on Twitter who evidence dissatisfaction with their product is exactly the kind of community engagement I believe in.

So, rather than simply provide an angry throwaway Tweet, I am going to do something of a teardown of my encounter with Cpanel. What went wrong, what went right, what is a bias on my part and what is clearly broken.

The problem environment

The VM in question is a CentOS 6.x VM running CPanel 56.0.32. The VM is running on on-premises hardware, so I have console access and thus the ability to effect maintenance outside of the CPanel interface.

The VM is running a VMware infrastructure, and did not appear to have VMware tools installed when I got hold of it. (This has now been remedied.) The VM has 4GB of RAM, a pair of vCPUs, and a 40GB drive that is roughly 60% full. The system is used primarily for hosting a small ecommerce site and its associated database.

The system was configured by a third party and handed off to a client of mine. The client configured basic items, such as e-mail addresses, in case something goes sideways on the system. The system barks about a DNS error once a week, (I have never figured out why,) but the error seems transient and so the system has been relatively low priority since the client took control.

Two days ago, the system started e-mailing me every few minutes that the “spamd” service had stopped and could not be restarted. After a few hours of this, I had put out the other fires I was fighting and decided to dive into the CPanel box and right the ship.

What went wrong

While it only took a cumulative three or four hours of my time to isolate the problem and repair it, it took the better part of a day to solve the CPanel thing. This is mostly because other things kept interrupting. The time delta is worth bearing in mind because frustration is an important part of the discussion behind my issues with CPanel.

First things first: “spamd” is, as most Linux administrators will have guessed, CPanel’s name for SpamAssassin. By saying the “service” spamd couldn’t start, CPanel was saying SpamAssassin had fallen and couldn’t get up. I use the air quotes around service because neither spamd nor SpamAssassin are services as far as CentOS is concerned.

The SpamAssassin version installed is not YUMs doing. It doesn’t come from the standard CentOS repository or even from EPEL. You see, CPanel has its own package manager; some Perl monstrosity made out of sadness and horror. It looks like it downloads source for the packages it wants to install and compiles them during install. I’m not 100% sure on that, I haven’t had time to go over the CPanel documentation in full yet.

What is important here is that “services” installed by CPanel (mostly) don’t have config files where veteran users of a given distribution would expect to find them, and the init scripts for those services aren’t in /etc/init.d. You also shouldn’t expect that these services will write error logs where you would expect them to, or at all, really.

If you have issues with systemd, you’re going to love CPanel.

Eventually, I found out where CPanel hid its version of /etc/init.d. (It’s in /scripts/, for those curious.) You can restart spamd by using the /scripts/restartsrv_spamd script. Fun fact: the scripts in /scripts/ are binary blobs, so good luck using them to figure out where the bits you need to actually solve problems are.

Teriffic.

On a lark, I restarted spamd. Lo and behold, it barked an error!  A kind of useless error, but it was enough to set me on my journey: SpamAssassin couldn’t find any signatures!

Running sa-update (the SpamAssassin signature update process) showed me that it couldn’t contact mirrors.updates.spamassassin.org, which is fair; mirrors.updates.spamassassin.org genuinely appears to not be there. Something is broken with SpamAssassin’s update infrastructure at the cloud level.

When CPanel started having a conniption was when it had recent updated itself. In doing so it compiled and installed a completely fresh copy of SpamAssassin and then called sa-update to install signatures. Sa-update couldn’t find signatures – because mirrors.updates.spamassassin.org is down – and so the spamd “service” couldn’t start; it had no signatures to test against.

After picking through clues, I figured out that /var/lib/spamassassin is where the various versions of the signatures ended up. I simply copied the signatures from a previous version into the directory for the latest version, restarted spamd and Bob is your mother’s brother: no more e-mails!

What went right

If that seems like a lot of griping, it’s worth counterbalancing the negativity with some positive comments about CPanel. The first: it did its job. When a service went down, it e-mailed me. And kept e-mailing me. And was utterly relentless in it’s e-mailing of me.

But the e-mails contain a link to turn off notifications, if I so chose. So CPanel’s devs cannot be faulted either for their monitoring, their notification mechanism or the construction of their e-mails. They got this aspect of the application absolutely, 100%, dead on right.

I want my servers to let me know when something’s gone TITSUP (Total Inability To Support Usual Performance), and it’s great that it includes a “bugger off” button. So cheers for that.

Similarly, it’s worth mentioning that CPanel did, in fact, update SpamAssassin as it was supposed to. It isn’t CPanel’s fault that there’s something broken upstream from them. They got the latest code, installed the latest version; the auto-updater does its thing and the system has been kept secure and up-to-date for over a year.

In this regard, CPanel meets or exceeds all expectations I have of a Linux management interface. They should be celebrated and applauded for this.

Personal biases

A discussion of my personal biases becomes important here before we continue on to what’s broken about CPanel’s implementation. What I consider broken is deeply intertwined with these personal biases, and these biases are driven in large part by my experience.

To understand my biases, you must understand a little bit about me. I am a systems administrator and technology writer who has been fixing computer for over twenty years. I tend to work with small businesses and midmarket companies with a particular specialty in making heterogeneous environments work.

I am at my best when working with a mix of operating systems, hypervisors, infrastructure, applications and management platforms. I make A talk to B and generally enjoy troubleshooting technology I’ve never seen before and that nobody else understands. These are the puzzles that make me happy.

I’m also incredibly, incredibly lazy.

Where the biases come in is that I like things that are easy to understand. A well designed user interface shouldn’t require a 400 page manual and $30k USD worth of specialist vendor training to understand.

If you’ve worked with 6 other flavours of layer 3 switch then the seventh should be expected to use industry standard terminology, have comprehensible help documentation available using standard (-h) triggers and otherwise be something whose syntax and/or UI can be worked out by a professional in about an hour.

Similarly, if I know the nerdy details of storage, I should be able to create a LUN on any storage array I happen across, or serve up an NFS share. I should not need the equivalent of a doctorate in that array’s UI.

This is my bias. I have a great big drum labelled “ease of use” that I bang on all day long. I believe that it is just as important to make technology intuitive to use as it is to make a technical somethingorother work.

Many people don’t share this view point. It is a long running debate in our industry.

Clearly broken

Bias or no bias, some things are very clearly broken about CPanel. The first and most important thing to note is: if log files can be perused in the CPanel UI, I was unable to find them in 4 hours of banging my head against the monitor. Considering that CPanel was very clearly designed to be a colour-by-numbers UI for people who don’t actually know anything about Linux, this is very, very bad.

Not everyone has access to the console or to SSH in order to go rooting about in /var/log. Yet, without the ability to punch the systems logs around a bit, I never would have solved that spamd problem. Access to logs – all the logs – needs to be available, and it need to be prominent.

This leads me into my next issue: organization. Or, more accurately, the lack thereof.

I like hierarchical design. It’s that personal bias thing again. Webmin/Virtualmin is my Linux management interface of choice, not because it is without flaw (don’t get me started), but because all the bits of “how to control your computer” are nested hierarchically, and – for the most part – in rational places.

In Webmin, the categories are:

Webmin (contains things dealing with Webmin itself),
System (contains log files, the crontab, authentication, and service management),
Servers (should be called “apps”, but contains things like MySQL, Apache/httpd, etc),
Networking (does what it says on the tin)
Hardware (also does what it says on the tin),
Cluster (never works, so do not use), and
Others (contains things like a shell file manger, should rightly be called “admin tools”).

I’m not writing to advertise Webmin. I offer up it’s top level category structure only as a comparison. It is a competing product to CPanel and thus seems reasonable to compare.

It is worth noting that for the full multitenant and colour-by-numbers approach to webhosting one would have to get Virtualmin, an extension to Webmin that offers us a few more (reasonably) well organized categories that are mostly wizards pertaining to setting up web space, mail, DNS and other services for tenants. In Virtualmin, this is in a separate tab from the core Webmin server administration section.

Personally, I don’t really care what categories an admin interface use. What I care about is that they’re reasonably clear. My wife is a junior windows administrator who dabbles in virtualization. She was able to figure out how to administer a Webmin-enabled Linux box in about 15 minutes.

At one point trying to solve the spamd thing, I gave up in frustration to take a half an hour off to pet a cat. My wife took a boo at the CPanel UI to see if she could figure out what was wrong. 15 minutes in she poured me a glass of Scotch and made a fresh pot of coffee. Her only comment: “I think it was designed by international committee”.

Now I understand so far this article has been a throwing a lot of shade with only a handful of specifics. Let’s talk specifics.

CPanel top level categories

CPanel has no less than 30 top level categories in its UI design. If you’re seeing things that should be grouped together, good. That means I’m not alone. But that isn’t the worst of it. Oh no, not by a long shot. I’ll provide running commentary as I go.

Server Configuration

Believe it or not, here is where you enable or disable spamd. Nestled under “email” inside “tweak settings” is a radio button that enables or disables the service. The most logical place for this would be in “service configuration”, but I would have accepted “packages”, “software” or “email” as places you might find that control. Note that the radio button here disables the use of spamd; it doesn’t provide a mechanism to start, stop or restart the service.

Aside from this, it’s worth noting that other than “change root password” and “server time” nothing in the “server configuration” top level category has anything to do with your server. It’s all about fiddling with CPanel settings and configuration. Which, you know, you’d sort of expect to be in the “CPanel” top level category. We’re off to an awesome start.

Support

Does what it says on the tin. Gives access to support options for CPanel.

Networking Setup

Offers “change hostname” and a wizard to change resolv.conf. Not helpful.

Security Center

Does what is says on the tin. Why isn’t “change root password” in here?

Server Contacts

Contains “Contact Manger” and “Edit System Mail Preferences”. While “Server Contacts” does what it says on the tin, this whole thing should really be under “Server Configuration”, or rolled up into any of many of the other categories.

Personally, I’d like to see “Server Configuration” used for things related to the underlying hardware and operating system, and “CPanel” used to contain “Server Contacts”, “Backups” and other things that are largely about CPanel’s functionality and maintenance. “Server Contacts” is basically a “set it and forget it” section that should be configured during initial system deployment and is unlikely to be changed much afterwards. It has no business being a top level category.

Resellers

Reseller account management. Does what it says on the tin. Should be under a more generic “accounts”.

Service Configuration

This section contains the configuration information for some services. Apache, Bandmin, PHP, Exim, FTP, DNS, etc. There are options to choose different applications for some functions. DNS and FTP in particular offer choices. CPanel says there are mail choices, but that’s a lie. The choice is “enable Dovecot or don’t”. You’re stuck with Exim for the MTA.

Under “Exim configuration manager” there are more SpamAssassin options. This isn’t about enabling or disabling the service (despite the option that says “Forced Global ON”). Nor is there any mention that “spamd” is SpamAssassin. It’s just some tweaks to SpamAssassin’s parameters. There is no info on where SpamAssassin is installed, where it’s config files are, no ability to pry open sa-update or it’s config files, etc.

Locales

Does what it says on the tin. Here you configure regional settings for CPanel. Again we’re confronted with a “set it and forget it” configuration item that really applies to CPanel itself, not the underlying server/OS or any of the applications. Why this isn’t under “CPanel” only Jibbers knows.

Backup

Ice cream is dispensed here (not really). See comments under “Server Contacts”.

Clusters

Here is where you can break everything badly, very easily. Yet for some reason it’s a top level category in a colour by numbers Linux management solution. This really should be somewhere under a box that only real geeks touch.

System Reboot

This is a top level category. A category. This isn’t, oh say, an icon in the top bar. Or under “Server Configuration”, or anywhere else rational. It’s a top level category that offers two options: graceful reboot and forced reboot. Stellar.

Server Status

Here you can see CPanel’s limited service monitoring capabilities, some basic server hardware information, the “Daily Process Log” (how much CPU and RAM various threads consume on a daily average), and Apache Status. That’s it. No log files to peruse. No debug information. Nothing that lets you restart services, say from the useful screen that shows you they’re down. Just the basic written-in-crayon stats about your install.

Account Information
Account Functions
Multi-Account Functions
Transfers

4 top level categories that deal with account information.  See what I said about “Resellers”. Also: for the love of Jibbers why?

Themes

Seriously? Get thee under “CPanel”. This shouldn’t be a top level category. This is just ridiculous.

Packages

This is a GUI front end to YUM. At least I think it’s YUM. It could be that hideous PERL package manger abomination. It doesn’t say. Either way, why is this a top level category?

DNS Functions

This does not let you control DNS functions for the underlying OS. This is about controlling DNS if you have a DNS server installed. It would be groovy if this were under an “applications” or “services” top level category.

SQL Services

See: “DNS Functions”

IP Functions

IP setup for the host OS, but also assignation of those addresses to tenants. Really not very clear. Personally, I’d break it in two, with the assignation of IPs to tenants being under accounts and the bits that give IPs to the OS under something nerdy to do with configuring the server itself. Mostly because this would help with moving towards role-based administration of CPanel clusters for larger environments.

Software

Somehow this is different from packages. There are two separate options for installing RPMs and for installing Perl modules. There is also something called “EasyApache” that gives me the fear and a MySQL/MariaDB upgrade. What?

I’m super confused. So there’s “Packages”, “Software” and “Plugins”. These are all different enough to warrant their own top-level categories, but there isn’t much to tell me why. Individual applications like Apache and MySQL are configured here, but also in “service configuration” and they have their own top level categories.

BRB, getting more Scotch.

Email

This is neither configuration of e-mail for CPanel itself (which is in “server contacts”) nor is this configuration of the email MTA (which is found partly in “Tweak Settings” under “Server Configuration” and partly under “Exim” in “Service Configuration”). This is mostly tweaking parameters for SpamAssassin, Greylisting (I don’t know what service they are using for that), and mail queue management.

*drinks Scotch*

System Health

Process management and free disk space. Give me one good reason this isn’t merged with “Server Status”.

CPanel

Update CPanel, install addons, and brand your CPanel install. You can’t actually configure anything about CPanel here. That’s sprinkled literally everywhere else it shouldn’t be.

*drinks Scotch*

SSL/TLS

Does what it says on the tin.

Market

Has exactly one item in it: “Market Provider Manager”. *sigh*

This isn’t an ecommerce package, but rather the “CPanel market” where you can buy plugins/addons/whatever you want to call them. I will point you at the “CPanel” section again, and at the “plugins” section, and then I’m going to refill my Scotch.

Restart Services

Sort of does what it says on the tin. Instead of a list of all the “services” you have installed, you get general categories of services. You can, for example, restart “Mail Server (Exim)”. This will restart all the slave services (including spamd). You cannot restart individual services (such as spamd) that have been slaved to a “category”, nor get useful or complete diagnostic output from those individual failed services.

Development

Developers developing plugins for CPanel will find most of what they’re looking for here. Including “Apps Managed by AppConfig”, which seems to be separate from YUM, the Perl package manager, the CPanel store, CPanel modules, plugins or whatever it is that was under the “packages” top level category.

*finishes Scotch*

Plugins

Finally, we get to “Plugins”, which isn’t plugins at all! It contains “ConfigServer Security&Firewall” and ClamAV!

Conclusion

So that’s my story and I’m sticking to it. Cpanel is awful.

Cpanel is awful not because the underlying technology is broken – it seems to work mostly okay, thanks – but because it lacks any semblance of an organizing principle to the UI, doesn’t expose critical functionality to administrators, and doesn’t give access to important diagnostic information for troubleshooting purposes.

I’m sure CPanel makes perfect sense to its developers. These sorts of UI clusterfucks always make sense to the people who dreamed them up. But the entire thing is as counterintuitive a UI as I have ever seen. To me, that means it’s very, very broken.

If CPanel wants to maintain a UI that needs a 400 page manual then it needs to start offering an expensive certification process so that HR people can identify the few humans who have put in the hours to understand what goes where, how and why.

CPanel isn’t the sort of thing that a systems administrator can just pick up and go. It isn’t the sort of thing you drop into a busy SMB/midmarket operation where all the sysadmins’ hair is on fire all the time.  The sort of place where they only get a chance to pop into a given UI once or twice a year, when the thing decides it’s going to throw a wobbly.

CPanel is the sort of thing you need a specialist for.  Despite this, the fact that it doesn’t expose a lot of the nerd knobs, logs and other critical techie bits seems to indicate it’s aimed at colour-by-numbers types.  There’s a fundamental dichotomy of design philosophy that I simply can’t reconcile.

This whole thing is frustrating not because I think CPanel is crap and should be abandoned, or because I think the developers are awful.  It is frustrating because CPanel gets (most of) the hard technical bits right, and seems to be a decent management platform.  It deserves to be at a state that it can be championed, not berated.

So pick a path, CPanel devs, and commit to it.  Either you head deep into crayon territory, which means fixing your “organization” structure, making trouble shooting super easy and generally solving the problems I’ve griped about…or you abandon the pretense and go the “certified specialist” route.

Either is fine, but you can’t straddle both horses like this.  And with that, I need to go buy more Scotch.