HTTPS on this blog

Because, why not? Let’s Encrypt makes it so easy…

Let’s Encrypt certificates are now used on all the websites maintained by Segvault, but not all of the websites of the TNY Network – the CPUVInf website, for example, seems to be using CloudFlare-provided TLS.

Recovering from prolonged outage

tny.im and this website were down for about 42 hours, starting on June 29 at 03:16 UTC.

The problem? BlueVM’s S19-NY server went down, taking with it the server I have/had there (and which I paid for a full year!). Other than this outage, the service had worked fine for three months, – fast network, full resources availability – since I bought it.

S19-NY is still down, without any ETA for it to come back. There’s no information in what conditions it will come back, or *if* it will come back (with the previous contents, at least). BlueVM staff is pretty much unresponsive, other than a guy who sometimes hangs on IRC and says he can’t fix the KVM instance because he doesn’t have access to it.

Of course, I no longer recommend BlueVM and I don’t plan on renewing the server I have with them.

The “solution” to put an end to two days of downtime, was to buy the cheapest SecureDragon OpenVZ server, (OpenVZ! so hard to live without my beloved KVM, I can’t even use davfs2 because there’s no fuse module!) and restore the backups I had (from four hours before the BlueVM instance went down). This has been done, except for the HTTPS certificate of tny.im… that alone is another story:

As I tried to retrieve the existing cert from StartSSL (because, stupid me, automated backups were not copying everything SSL-related, and I didn’t save it locally), I found my authentication certificate had expired. This basically means my StartSSL account is lost, unless I create a new account and ask their staff to link it to the old one. They probably won’t do that without a payment and some ID checks so… out of question. I guess, if the BlueVM server doesn’t come back, that I’ll just create a new StartSSL account and generate a new cert for tny.im. There’s no security issue with this, as the previous certificate has not been compromised (unless BlueVM is collecting certificate private keys from inside their clients’ machines…) and so it doesn’t need to be revoked.

To conclude the HTTPS point: tny.im has the HTTPS service unavailable for now, until I can retrieve the existing certificate from the previous server, or until I get a new one.

Is all the fault in BlueVM’s side? Of course not… I could have lost my love for the money earlier, bought the SecureDragon VPS yesterday already and reduce the downtime by 24 hours. But since I had hope the problems on BlueVM support were just Sunday-related, I thought that by Monday they would have it fixed. They didn’t.

On related news, I’ll take this downtime and new server acquisition as the motivation for setting up a new advanced and redundant system, so that if one server goes down, tny.im (and possibly this blog too) will continue to operate as normal. I have two servers already (assuming the BlueVM one comes up), and I plan on developing a system where firing up new instances of tny.im on any empty server will be really easy. The system will be always prepared to lose any server at any point, without data loss, and restore full service within five minutes. That way, I can add less reliable hosts, perhaps even VPS trials, to the redundant system. This also allows for scaling the service as needed. Sounds ambitious? Of course it won’t happen in a week, but I have the full summer to develop and test the system…

Why don’t I just go with some SaaS that supports scaling? Two reasons: the price is too high, and the tny.im software is not coded in a way that’s compatible with these services. Let me remember you, that while not exactly being a CGI script, tny.im is not coded in one of those fancy modern languages, and even though PHP is not exactly outdated or unmaintained, the quality of the code can make it pretty horrible or pretty good. And the code is… not perfect – it doesn’t use any popular framework, it is based on YOURLS and has many, many hacks feature additions, plus a… very close relationship with the database.

Let me finish by saying that downtime of this kind is something to be expected if I were still hosted by FreeVPS and the like. But believe it or not, on FreeVPS and other sponsorships I’ve never seen a customer service as bad as the one of BlueVM (and it’s hard to remember an outage as big as this one). It is definitely not adequate for a paid service. In addition to S19-NY, they have many other instances down, with similar or worse downtime. The admins don’t appear to be online or reachable in any way, even by other staff members. The latest news/excuse on the S19-NY situation is that IPMI is broken and they are waiting for the provider to fix it… now tell me, does this look like a serious company, or some poor-man’s sponsorship?

EDIT: The BlueVM server is still down. tny.im is now hosted by three servers with round-robin balancing. HTTPS service was restored with a new certificate.

EDIT 7/7/2014: I forgot to update this post in time, but the BlueVM server has been up since three days ago. But I only got to know that the service was restored thanks to a friend of mine, because they didn’t reply to my ticket to inform me about it. Anyway, I don’t plan to renew this server, and BlueVM lost me as a customer (except for some really cheap deal which I’ll use as a personal/development box, and never in production).

On related news, Mirasm – the Tiny Server Redundancy Manager – is mostly finished, only needs some more testing to be put on production servers, managing the new tny.im redundancy system.

Server changes

I moved this blog to a different server (no downtime because the transfer was planned). Please report if something is broken.

On other topic, I have been very busy with school and real life in general, even on weekends; that sums up the reasons for my recent online absence.

Server reloaded

The dedicated server where the VPS that hosts this blog was installed had a serious hard drive failure, and all the data in it was lost. As a result, I was prompted to reinstall my server. The “hydrogen” server was down from  around the 2:00 UTC to around 17:00 UTC (11th September).

Fortunately, I had a backup of the blog contents from 8th September. Reinstalling WordPress and all the software required to run it (Nginx, PHP, MySQL) was still a big hassle, and looks like the PageLines settings aren’t saved in the backup WordPress generated, so I didn’t feel like setting up PageLines again and the blog is back to a white-ish theme (which I have already modified to include a widget area at the bottom…).

It also looks like restoring a backup results in subscribers being sent an email per each post restored… fortunately, the blog only has one subscriber at the moment so this wasn’t a major spamming event.

Oh, and I almost forgot: tnyCloud is gone, but that shouldn’t be a problem since nobody seemed to use it, not even me. Also, since I was forced to move to a smaller VPS, I no longer had enough disk space to host a service like tnyCloud, so basically this hard drive failure saved me the hassle of declaring it dead.

I hope the prolonged downtime didn’t annoy you much. The tny.im URL shortener is hosted on another server which was up all the time – UptimeRobot tells me it has a whooping 99.97% uptime ratio. Now that it already has the three nines, it’s time to work to get the five nines 😉

A server reload is nothing compared to what happened eleven years ago.

Theme changed

I felt the previous theme was a bit too white-ish so I changed to another one. I hope you like it.

This website is back!

Looks like my servers and websites have all decided to take some holidays and go offline, fortunately not at the same time. Some weeks ago, it was 4.l.to/l.f.nu that decided that some days sleeping would be good, after its domain went down (causing the change to a new one and the whole service rebranding). And more recently, the VPS where I was (yes, was) hosting this blog, which by its turn was hosted in a friend’s dedicated server, went down the trash too: the guys at the provider my friend uses decided to play around with the hard drive of the dedicated server, and we ended up without any of data that was in it.

Unlike what’s usual, this time I had backups (yepeee!). But as always, they were outdated (from January!) and consisted of a WordPress export file. So, I didn’t have any backup of the server configuration or the other scripts and data I had in the server. Conclusion: I had to set up everything from scratch – but wait, first, I need to explain: my friend offered to install WordPress for me, (as I’m very busy with real life, I’ll explain later), but he used CentOS, and since I really don’t like CentOS and there were some tiny “wrong” details in the WordPress config (just a matter of personal choice: I do not like to use “admin” as the admin username, even for security reasons), I reloaded the VPS with Ubuntu.

*Ubuntu: I would have used Debian, if it weren’t for the fact the software in its repos is, although stable, far from being the latest version. And my idea of “stable-recent” ratio for software is not quite the same as Debian’s idea.

As I was saying, I had to setup everything from scratch on a new VPS, on another dedicated server that’s not from the same provider (but the dedi is from the same friend). That means some hours around the shell installing and configuring nginx, PHP and MySQL, as well as configuring WordPress-specific rewrite rules and other server settings – and I’m not finished yet, the current settings are not how I’d like them to be.

I said above I was very busy with real life: yes I am, I’m busy with lots of school work, and I’m also a bit tired of the online world for now (the part of the internet I use/follow has no news lately, things are pretty boring currently). But today I had a school trip for the whole day that got me really tired, and when I got home, I felt like I wouldn’t be able to study anything for the school tests I’m taking in the next weeks. I had a server to configure and a blog to restore, and thought I could use the free time… and here I am, blogging when it’s almost midnight on my clock.

Despite the hours spent, I’d say my work has been done without major problems. I’m getting either too used to installing nginx+php+mysql, or it’s because it is/was Friday 13th.

Yeah, is/was. It’s four past midnight.

EDIT: this server is now much faster, its host was suffering from some misconfiguration – again, my friends are awesome and fixed it 🙂

Forcibly changed servers; data loss!

Let’s keep things short as I have lots of things to do. On 30th November, this website was working functional as it had always been since April 2011. Then suddenly, in the morning of 1st December, the server had been reloaded.

At first I thought it was because someone with admin powers at cheapvps.co.uk, the provider of my previous server that hosted this website, reloaded the VPS. But after some searches, I ended concluding the VPS, which luis123456 had given to me in April 2011, was still owned by someone else – and that someone was trying to make use of the VPS.

I found the email of the real VPS owner (in fact, I used it for logging in to the VPS control panel, but I always had thought it was just some random address, because it only had two letters and five numbers!). I sent an email to that address, and some hours ago, I got a reply from David W. – the real owner of my previous gbl08ma.com server, to which I called “hydrogen”. So what happened in fact? According to David, he told luis123456 to “maintain” (and no more than that) the VPS. luis123456, whose real name is Luis A. (so we talk about real names here) was not authorized, to use the VPS. Still, (and I repeat, this is the saying of David), Luis gave me the VPS. Luis never said anything about this: I thought the VPS was some kind of sponsored VPS which the sponsor forgot about.

So, the old server was reloaded. But things were worse for my side: I was supposed to have an automated backup system, but it was broken and I had no time to fix it. Shortening: I have no backups of the old server, except Google cache, which didn’t cache one or two blog posts. Apart from the text of the posts, cached by Google, I lost everything on the server, that is, all files, images, configuration files, scripts, WordPress plugins, themes… hosted within the server. This also includes the few Anti-Aliased fonts for Rockbox, which I’ll have to upload again some day.

I just finished restoring all the blog posts I could. I’m still wondering how to restore comments done by other users on the various posts. But wait, I missed one part, right? How did I get this new server?

Indeed, this is a new server. I say the website keeps being the same only because the matter and intention of it keeps being the same, but in technical terms, this is a whole new WordPress install, on a whole new server. While this has some advantages, it also has lots of disadvantages – you can compare this to formatting a dog slow Windows computer, without making backups first: after re-installing the operating system, you get a clean system but most of your data, configuration and software is lost.

This new server was given by Humza Bobat, Infinity at freevps.us. So yes, now I have two server provided by freeVPS.us; since I know other users will get angry with me by having the admins of freeVPS opening an exception of the one-vps-per-user rule, I must provide some good argument to it.

In fact, I need two server for various reasons. One of the reasons, is that since the virtual servers I own are not very powerful (they are low-end boxes), they can barely handle two websites on the same server – note that we are talking about WordPress on this website, and while WordPress can run on fairly modest setups, I want some speed both for me and for the sometimes 10 concurrent users of the websites (it happens, for example when my stories get featured on Slashdot).

The multiple virtual host configurations, while they work well if you never touch the configuration files, are harder to maintain, in my opinion. But the main reason why having multiple servers is necessary is the following: being servers provided for free, you never know when one goes offline or you loose it forever (OMG! It just happened!). If one goes offline, you can still use the other for temporarily hosting an additional website or at least some informational page. Plus, you should never put all your eggs in one basket: the server that served gbl08ma.com went away with its data, but the situation could be worse if all the things were on it. If I had what I have on my “helium” server, the one that serves 4.l.to, on the server I lost, then I would have lost some hundreds of MB of information that is important (some of it even a bit confidential).

A interesting thing is, I have automated backups of the helium server working, although it gets much less visitors (not including short url clicks) than gbl08ma.com gets. The Murphy’s law regarding backups applies: even if you have backups of your things, they will never include what you just lost.

What about naming conventions? The “hydrogen” name now refers to this new server, while I’ll call the old one from on now “deuterium” 🙂 . The 4.l.to server keeps being called “helium”, as ever. I also have one testing server called “lithium”, but it isn’t used for anything permanent – as I said, it’s a test server.

If you have any questions regarding the data loss on this blog, please post on the comments. I’ll be busy for the next days/weeks/months trying to restore more of what was lost.

Now running with nginx!

I was really fed up with Apache on this server. It would use huge amounts of RAM, even after all the visitors left the website. Having done all tweaks to the memory usage of Apache and PHP, the amount of RAM used would never get below 450MB (out of the 512MB this VPS has). Hell, Apache was consuming even more memory than MySQL!

For those who don’t know, nginx is an alternative, lightweight webserver which is generally used (by many popular websites) as a load balancer. However, it can also act as the single web server on a system, like what Apache and Lighttpd do. I had worked with nginx before on some small websites on low-resource servers, and I was quite satisfied with it. As I explained with an earlier blog post, nginx is great as long as the website you want to serve with it does work with nginx – that is, doesn’t heavily depend on Apache rules or some Apache-specific thing. Sure, those rules can be converted to nginx config options, but I never succeeded on making eyeOS 1.x work fully with nginx.

WordPress is one of the scripts that works best with nginx. Since this website is mainly powered by nginx (although I have some custom scripts laying around, mainly the scripts providing alternative WiiMC internet media), I made my mind and decided I would go through the hassle of switching from Apache to nginx. It wasn’t a big hassle after all: apart from having to restart the server at some point due to a RAM outage, the website wasn’t offline much time, and there was no data loss.

After putting Apache off-use and starting nginx, the server was still using 300MB of RAM. I though nginx couldn’t be using so much RAM, and there was another problem laying around. Turns out to be a problem in MySQL config: I don’t need InnoDB functionality, so usually I add a “skip-innodb” line to my.cnf. The problem was, this line needs to be added under the [mysqld] section and in my case, it was somewhere else. So I moved skip-innodb to the right place, restarted MySQLd, and that’s it:

The server is now using 240MB of RAM, which still fits inside the dedicated RAM (256MB), so I’m not taking any of the burst RAM, which resides in the server swap space. The RAM usage is still high, because I have other things running such as dovecot for mail delivery.

It also looks faster to load pages, but probably someone with a faster connection than me will notice a bigger difference.

I have been upgraded to .com!

Following the latest changes in this website domain, it looks like it will have to edit the URL for this website in wordpress settings again, in no more than a week.

But this time, unlike last time, I’m incredibly happy. No, my birthday is only on 8th October, but the owner of http://freevps.us, dmmcintyre3 has registered for me the domain “myself on .com”, that is,http://gbl08ma.com . Yeah! That’s myself on a .com domain – in case you haven’t understood.

Goodbye malware-false-markings due to the use of crappy free domains! Now I only have to make sure my real TLD is not marked as malware itself.

Now I can freely post URLs for posts in my blog without fear for supposedly containing malware! This is an huge step!

Remember, the Bitcoin donations thing is still valid; at the end, I want to renew this domain next year 🙂

An huge thank you to dmmcintyre3! And to Namecheap for providing cheap domain names.