Recovering from prolonged outage

tny.im and this website were down for about 42 hours, starting on June 29 at 03:16 UTC.

The problem? BlueVM’s S19-NY server went down, taking with it the server I have/had there (and which I paid for a full year!). Other than this outage, the service had worked fine for three months, – fast network, full resources availability – since I bought it.

S19-NY is still down, without any ETA for it to come back. There’s no information in what conditions it will come back, or *if* it will come back (with the previous contents, at least). BlueVM staff is pretty much unresponsive, other than a guy who sometimes hangs on IRC and says he can’t fix the KVM instance because he doesn’t have access to it.

Of course, I no longer recommend BlueVM and I don’t plan on renewing the server I have with them.

The “solution” to put an end to two days of downtime, was to buy the cheapest SecureDragon OpenVZ server, (OpenVZ! so hard to live without my beloved KVM, I can’t even use davfs2 because there’s no fuse module!) and restore the backups I had (from four hours before the BlueVM instance went down). This has been done, except for the HTTPS certificate of tny.im… that alone is another story:

As I tried to retrieve the existing cert from StartSSL (because, stupid me, automated backups were not copying everything SSL-related, and I didn’t save it locally), I found my authentication certificate had expired. This basically means my StartSSL account is lost, unless I create a new account and ask their staff to link it to the old one. They probably won’t do that without a payment and some ID checks so… out of question. I guess, if the BlueVM server doesn’t come back, that I’ll just create a new StartSSL account and generate a new cert for tny.im. There’s no security issue with this, as the previous certificate has not been compromised (unless BlueVM is collecting certificate private keys from inside their clients’ machines…) and so it doesn’t need to be revoked.

To conclude the HTTPS point: tny.im has the HTTPS service unavailable for now, until I can retrieve the existing certificate from the previous server, or until I get a new one.

Is all the fault in BlueVM’s side? Of course not… I could have lost my love for the money earlier, bought the SecureDragon VPS yesterday already and reduce the downtime by 24 hours. But since I had hope the problems on BlueVM support were just Sunday-related, I thought that by Monday they would have it fixed. They didn’t.

On related news, I’ll take this downtime and new server acquisition as the motivation for setting up a new advanced and redundant system, so that if one server goes down, tny.im (and possibly this blog too) will continue to operate as normal. I have two servers already (assuming the BlueVM one comes up), and I plan on developing a system where firing up new instances of tny.im on any empty server will be really easy. The system will be always prepared to lose any server at any point, without data loss, and restore full service within five minutes. That way, I can add less reliable hosts, perhaps even VPS trials, to the redundant system. This also allows for scaling the service as needed. Sounds ambitious? Of course it won’t happen in a week, but I have the full summer to develop and test the system…

Why don’t I just go with some SaaS that supports scaling? Two reasons: the price is too high, and the tny.im software is not coded in a way that’s compatible with these services. Let me remember you, that while not exactly being a CGI script, tny.im is not coded in one of those fancy modern languages, and even though PHP is not exactly outdated or unmaintained, the quality of the code can make it pretty horrible or pretty good. And the code is… not perfect – it doesn’t use any popular framework, it is based on YOURLS and has many, many hacks feature additions, plus a… very close relationship with the database.

Let me finish by saying that downtime of this kind is something to be expected if I were still hosted by FreeVPS and the like. But believe it or not, on FreeVPS and other sponsorships I’ve never seen a customer service as bad as the one of BlueVM (and it’s hard to remember an outage as big as this one). It is definitely not adequate for a paid service. In addition to S19-NY, they have many other instances down, with similar or worse downtime. The admins don’t appear to be online or reachable in any way, even by other staff members. The latest news/excuse on the S19-NY situation is that IPMI is broken and they are waiting for the provider to fix it… now tell me, does this look like a serious company, or some poor-man’s sponsorship?

EDIT: The BlueVM server is still down. tny.im is now hosted by three servers with round-robin balancing. HTTPS service was restored with a new certificate.

EDIT 7/7/2014: I forgot to update this post in time, but the BlueVM server has been up since three days ago. But I only got to know that the service was restored thanks to a friend of mine, because they didn’t reply to my ticket to inform me about it. Anyway, I don’t plan to renew this server, and BlueVM lost me as a customer (except for some really cheap deal which I’ll use as a personal/development box, and never in production).

On related news, Mirasm – the Tiny Server Redundancy Manager – is mostly finished, only needs some more testing to be put on production servers, managing the new tny.im redundancy system.

Alternative IM/VoIP solution?

If anyone knows of a instant messaging and VoIP solution that’s free, privacy friendly (end-to-end encryption, preferably) and, most importantly, easy to get the “average Joe” to switch to (i.e. my non-technical friends), please let me know. Bonus points if it has interoperability to work with other systems (at the eventual cost of losing some of the privacy) – that way, I wouldn’t need to get all my friends to switch.

(Of course, Google Talks/Hangouts, Skype and anything Facebook-related is out of the equation.)

Another thing I’m looking forward to is having my computers (Linux and Windows), old Windows Mobile phones and my Android phone and tablet synchronized (contacts, calendar, email) through the Internet without Google’s help, but I realize this is even harder find than the IM solution I specified above.

I don’t appreciate the “Like” button

Of all the ways to express your opinion on some subject, I believe the “Like”, “+1” and similar buttons are some of the worst. Why? Well, nowadays “liking” something on the internet means little to nothing. People are asked to “like” things, “likes” are sold and bought as a product and not actually as a consequence on someone’s feelings on what one has seen/read/experienced, and now the quality of things seems to have become measured in the number of “likes”.

I usually say the “Like” button was the best invention for those that are so lazy that don’t want to write anything, or those so lazy that don’t want to create an opinion on a certain subject. It is also a great thing for those who don’t care about explaining why they “like”. The same argument is also true for “disliking”, on the places where that’s permitted. Those who have something to say will comment or reply, but “liking” is something so vague that adds little value.

It’s important to let people express their opinion on other Internet content in a meaningful way. Allowing users to comment and reply in an Internet that’s more and more made by its daily users is a good thing (that is, if you really promote freedom of speech). It perhaps even motivates people to think about things and form their own view on the subject, instead of just “liking” a view that’s being forced into their minds.

Imagine someone on the Internet says “WordPress is a really cool blogging tool”. You have the following options: you can either “Like” this statement, comment on it, or don’t give a s*** about it and move on. If you agree with the point of view stated, but have nothing to say on it, you’ll probably click the “Like” button. If you don’t agree, you’ll move on, or eventually post a short comment stating that you don’t agree. And if you are of those that actually wants to express an opinion and cares to write trying to use the language properly, you’ll comment. Now imagine you can’t comment… probably you’ll just move on.

If you comment and your comment is insightful, it will add value to an existing discussion or perhaps even start a new one. But those who “like”… what will happen? When you see “34 people like this”, do you have any idea of what those 34 people think? Did they “like” because they found it funny? Because that content was interesting? Because it was so wrong that it made one laugh? And who knows how many people didn’t like that content, specially when compared to something else? I think this need for comparison and ranking caused “likes” to be used as if they were a measurement unit, as I’ll explain later.

I even fear one day people living in a democracy will vote for their representatives by “liking” them. Knowing how many didn’t “like” any of the options is going to be hard. And you won’t know it was because none of the options suited them, or because they were ill in the elections day, or because they preferred going to the beach instead of voting, errm, “liking”. Knowing how many people “liked” twice can get hard too, but that’s easily fixed.

One more thing that illustrates the stupidity of the “Like” (or similar) button: it doesn’t exist in natural human communication. Well, it does exist, but it’s way more elaborated than a “Like”. Imagine you’re hanging out with your friends, in the pre-“Like”-button era, and one of them tells a joke. Nobody’s going to say “I like” without saying anything more. Since it was a joke, if one has found it funny, laughs will follow. And if it was really funny, one will laugh a lot (I also have my opinion on the LOL thing, but that’s for another post). And if the joke wasn’t funny at all, or the way it was told wasn’t good enough, one will at least smile, or say “Man, you’re not good at telling jokes”.

And another example: if you go to a restaurant and you enjoy the meal you ordered, it’s unlikely that you just say “Like”. Even if you only want to say you liked what you ate, there are many, many ways to say “Like”. Now if I want to be ultra-nerd, I can even say the “Like” button impoverishes people’s vocabulary. 🙂 So to conclude this point: at most, people have brought “I like this” into real-life communication after it became popular in the web – it didn’t exist in such a monotone and endlessly overused way before that.

I’m not saying the “Like” button isn’t useful – for the times when you actually like and there’s nothing else to say. The problem is, people became lazy and now they prefer to click a button than to write their opinion – sometimes because they don’t have any opinion, other times because it’s just easier to “Like”. Again, if I jump to extreme cases, the web might become something where some party says “1+2=5” and all there is to say is that “56,322,943 people like this”.

Now about the “Like” button as a measure of quality of things. If for a given “product X” there are 60000 likes on some social network and for another “product Y” there are only 2000 likes, people will often think “product X” is better than “product Y”. But those who will care about doing some research will find that “product Y” doesn’t contain “substance N”, which is really bad for health, while “product X” does contain it. “Product X” has more likes because it appeared first on that social network as part of an advertising campaign that costed millions. Conclusion: the number of people that “Like” something is worth nothing, even though at first it might look like so. Even because “likes” can often be bought: imagine that millionaire advertising campaign included buying 10000 “likes” to bootstrap it, and “liking” things becomes even more meaningless.

But the example doesn’t need to be about evil companies and products that are bad for health being advertised in a giant scale. You certainly know those people that ask for likes on their content. And those annoying “If you are happy, like this”-style messages. This happens in social networks in each other’s friends circles.

Oh, and another thing: “Like” buttons are used for tracking people whenever they go on the web. You can leave the “website X” that hosts a “Like” button, that as long as there is a “Like” button of that “website X” in any other page, the owners of that website can know you’re at that page. And I’m not dreaming, as you know, Facebook and other social networks do this.

This stupid “Like”/”+1” button is one of the many reasons why I deactivated my Facebook account some days ago. But this isn’t only about Facebook, it’s about everything sponsoring a “Like” button. (At least Twitter doesn’t have such a “feature”, hooray! 🙂 )

Putting short: yes, you can keep the “Like” button, but make sure people can comment – and I’d encourage them to comment and show their views on things whenever possible: I think it adds a lot more value to the Internet.

EDIT: looks like Facebook “Likes” aren’t speech protected by the US First Amendment.