Because, why not? Let’s Encrypt makes it so easy…
Let’s Encrypt certificates are now used on all the websites maintained by Segvault, but not all of the websites of the TNY Network – the CPUVInf website, for example, seems to be using CloudFlare-provided TLS.
tny.im and this website were down for about 42 hours, starting on June 29 at 03:16 UTC.
The problem? BlueVM’s S19-NY server went down, taking with it the server I have/had there (and which I paid for a full year!). Other than this outage, the service had worked fine for three months, – fast network, full resources availability – since I bought it.
S19-NY is still down, without any ETA for it to come back. There’s no information in what conditions it will come back, or *if* it will come back (with the previous contents, at least). BlueVM staff is pretty much unresponsive, other than a guy who sometimes hangs on IRC and says he can’t fix the KVM instance because he doesn’t have access to it.
Of course, I no longer recommend BlueVM and I don’t plan on renewing the server I have with them.
The “solution” to put an end to two days of downtime, was to buy the cheapest SecureDragon OpenVZ server, (OpenVZ! so hard to live without my beloved KVM, I can’t even use davfs2 because there’s no fuse module!) and restore the backups I had (from four hours before the BlueVM instance went down). This has been done, except for the HTTPS certificate of tny.im… that alone is another story:
As I tried to retrieve the existing cert from StartSSL (because, stupid me, automated backups were not copying everything SSL-related, and I didn’t save it locally), I found my authentication certificate had expired. This basically means my StartSSL account is lost, unless I create a new account and ask their staff to link it to the old one. They probably won’t do that without a payment and some ID checks so… out of question. I guess, if the BlueVM server doesn’t come back, that I’ll just create a new StartSSL account and generate a new cert for tny.im. There’s no security issue with this, as the previous certificate has not been compromised (unless BlueVM is collecting certificate private keys from inside their clients’ machines…) and so it doesn’t need to be revoked.
To conclude the HTTPS point: tny.im has the HTTPS service unavailable for now, until I can retrieve the existing certificate from the previous server, or until I get a new one.
Is all the fault in BlueVM’s side? Of course not… I could have lost my love for the money earlier, bought the SecureDragon VPS yesterday already and reduce the downtime by 24 hours. But since I had hope the problems on BlueVM support were just Sunday-related, I thought that by Monday they would have it fixed. They didn’t.
On related news, I’ll take this downtime and new server acquisition as the motivation for setting up a new advanced and redundant system, so that if one server goes down, tny.im (and possibly this blog too) will continue to operate as normal. I have two servers already (assuming the BlueVM one comes up), and I plan on developing a system where firing up new instances of tny.im on any empty server will be really easy. The system will be always prepared to lose any server at any point, without data loss, and restore full service within five minutes. That way, I can add less reliable hosts, perhaps even VPS trials, to the redundant system. This also allows for scaling the service as needed. Sounds ambitious? Of course it won’t happen in a week, but I have the full summer to develop and test the system…
Why don’t I just go with some SaaS that supports scaling? Two reasons: the price is too high, and the tny.im software is not coded in a way that’s compatible with these services. Let me remember you, that while not exactly being a CGI script, tny.im is not coded in one of those fancy modern languages, and even though PHP is not exactly outdated or unmaintained, the quality of the code can make it pretty horrible or pretty good. And the code is… not perfect – it doesn’t use any popular framework, it is based on YOURLS and has many, many
hacks feature additions, plus a… very close relationship with the database.
Let me finish by saying that downtime of this kind is something to be expected if I were still hosted by FreeVPS and the like. But believe it or not, on FreeVPS and other sponsorships I’ve never seen a customer service as bad as the one of BlueVM (and it’s hard to remember an outage as big as this one). It is definitely not adequate for a paid service. In addition to S19-NY, they have many other instances down, with similar or worse downtime. The admins don’t appear to be online or reachable in any way, even by other staff members. The latest news/excuse on the S19-NY situation is that IPMI is broken and they are waiting for the provider to fix it… now tell me, does this look like a serious company, or some poor-man’s sponsorship?
EDIT: The BlueVM server is still down. tny.im is now hosted by three servers with round-robin balancing. HTTPS service was restored with a new certificate.
EDIT 7/7/2014: I forgot to update this post in time, but the BlueVM server has been up since three days ago. But I only got to know that the service was restored thanks to a friend of mine, because they didn’t reply to my ticket to inform me about it. Anyway, I don’t plan to renew this server, and BlueVM lost me as a customer (except for some really cheap deal which I’ll use as a personal/development box, and never in production).
On related news, Mirasm – the Tiny Server Redundancy Manager – is mostly finished, only needs some more testing to be put on production servers, managing the new tny.im redundancy system.
I moved this blog to a different server (no downtime because the transfer was planned). Please report if something is broken.
On other topic, I have been very busy with school and real life in general, even on weekends; that sums up the reasons for my recent online absence.
The dedicated server where the VPS that hosts this blog was installed had a serious hard drive failure, and all the data in it was lost. As a result, I was prompted to reinstall my server. The “hydrogen” server was down from around the 2:00 UTC to around 17:00 UTC (11th September).
Fortunately, I had a backup of the blog contents from 8th September. Reinstalling WordPress and all the software required to run it (Nginx, PHP, MySQL) was still a big hassle, and looks like the PageLines settings aren’t saved in the backup WordPress generated, so I didn’t feel like setting up PageLines again and the blog is back to a white-ish theme (which I have already modified to include a widget area at the bottom…).
It also looks like restoring a backup results in subscribers being sent an email per each post restored… fortunately, the blog only has one subscriber at the moment so this wasn’t a major spamming event.
Oh, and I almost forgot: tnyCloud is gone, but that shouldn’t be a problem since nobody seemed to use it, not even me. Also, since I was forced to move to a smaller VPS, I no longer had enough disk space to host a service like tnyCloud, so basically this hard drive failure saved me the hassle of declaring it dead.
I hope the prolonged downtime didn’t annoy you much. The tny.im URL shortener is hosted on another server which was up all the time – UptimeRobot tells me it has a whooping 99.97% uptime ratio. Now that it already has the three nines, it’s time to work to get the five nines 😉
A server reload is nothing compared to what happened eleven years ago.
I felt the previous theme was a bit too white-ish so I changed to another one. I hope you like it.
Looks like my servers and websites have all decided to take some holidays and go offline, fortunately not at the same time. Some weeks ago, it was 4.l.to/l.f.nu that decided that some days sleeping would be good, after its domain went down (causing the change to a new one and the whole service rebranding). And more recently, the VPS where I was (yes, was) hosting this blog, which by its turn was hosted in a friend’s dedicated server, went down the trash too: the guys at the provider my friend uses decided to play around with the hard drive of the dedicated server, and we ended up without any of data that was in it.
Unlike what’s usual, this time I had backups (yepeee!). But as always, they were outdated (from January!) and consisted of a WordPress export file. So, I didn’t have any backup of the server configuration or the other scripts and data I had in the server. Conclusion: I had to set up everything from scratch – but wait, first, I need to explain: my friend offered to install WordPress for me, (as I’m very busy with real life, I’ll explain later), but he used CentOS, and since I really don’t like CentOS and there were some tiny “wrong” details in the WordPress config (just a matter of personal choice: I do not like to use “admin” as the admin username, even for security reasons), I reloaded the VPS with Ubuntu.
*Ubuntu: I would have used Debian, if it weren’t for the fact the software in its repos is, although stable, far from being the latest version. And my idea of “stable-recent” ratio for software is not quite the same as Debian’s idea.
As I was saying, I had to setup everything from scratch on a new VPS, on another dedicated server that’s not from the same provider (but the dedi is from the same friend). That means some hours around the shell installing and configuring nginx, PHP and MySQL, as well as configuring WordPress-specific rewrite rules and other server settings – and I’m not finished yet, the current settings are not how I’d like them to be.
I said above I was very busy with real life: yes I am, I’m busy with lots of school work, and I’m also a bit tired of the online world for now (the part of the internet I use/follow has no news lately, things are pretty boring currently). But today I had a school trip for the whole day that got me really tired, and when I got home, I felt like I wouldn’t be able to study anything for the school tests I’m taking in the next weeks. I had a server to configure and a blog to restore, and thought I could use the free time… and here I am, blogging when it’s almost midnight on my clock.
Despite the hours spent, I’d say my work has been done without major problems. I’m getting either too used to installing nginx+php+mysql, or it’s because it is/was Friday 13th.
Yeah, is/was. It’s four past midnight.
EDIT: this server is now much faster, its host was suffering from some misconfiguration – again, my friends are awesome and fixed it 🙂
Do you remember the OpenID standard, that aims to describe “how users can be authenticated in a decentralized manner, eliminating the need for services to provide their own ad hoc systems and allowing users to consolidate their digital identities.”? Well, if you happen to frequently authenticate on a service or website that supports it, or if you happen to run or maintain one of these websites or services, most likely you remember. But the surprising part is, OpenID is used in more things than you can imagine.
Till some time ago, I don’t recall seeing much opportunity for logging in with an OpenID – except on the websites of the ID provider themselves. The first OpenID authentication method I recall using was using Twitter IDs, although in that case I could as well have used Google or Facebook. But people use OpenID without actually recognizing it as an implementation of that standard. Yes, OpenID is that “Login with Facebook” or “Login with Twitter” thing. These login methods are usually just not (visibly) branded as being OpenID.
So basically, that represents a win for OpenID, right? Well, in theory yes, but my opinion is different. While many websites carry out OpenID in such a way that it is comfortable for every user, others simply don’t. What do I call a “comfortable usage” of OpenID? An implementation of the standard in such a way that it allows you to choose the ID you want to use. Eventually, it also lets you not use OpenID, through the creation and authentication of a traditional account, where the chosen authentication parameters are isolated to the website or service in question, like we’ve seen before the OpenID boom.
This “comfortable implementation” fits the most users I can think of: by assuring authentication using accounts on the most popular OpenID providers, such as Google, WordPress and Facebook, and using simpler, standalone (i.e. not tied to any service in particular) and/or less-known providers such as chi.mp, claimID and myOpenID, the chances of the person willing to be authenticated having an ID with one of the providers supported is way bigger. But because not everyone likes the OpenID idea, or they might simply not have a registered account with one of the IDs supported, an additional “traditional” authentication method should also be provided, so people can create an account with the website or service in question, and not tie that account with an OpenID.
The advantages of what I call a “comfortable implementation” are very noticeable in my opinion: it increases the user base of a website, since if people find it easy to login with an account they already have on other service, it’s very likely they’ll login on that website. It also makes the act of engaging with the website a breeze, because people don’t need to go over the hassle of maintaining yet another user/password combination, there is no signup form, captcha or email validation. While this may change depending on the OpenID provider and on the service or website implementing OpenID authentication, in most situations the OpenID login process is easier. We just got to recognize another advantage: if users find registering and logging in easier, the website or service will not only get more users, as it will have its users more satisfied. As I said, for the user there’s not the hassle of not remembering the specific password and having to reset it, and for the website management, there can be also a reduction in the number of support requests, assuming OpenID is properly implemented. All I did here was point some of the advantages of OpenID, but it can also have a lot of disadvantages when its implementation is not so comfortable for the user.
A website that I remember having a proper implementation of open IDs is Blogger, at least when posting a comment on a blog – it allows you to choose which profile you want to comment under, from a Twitter, WordPress or Google account to a OpenID, discussed here.
But what is an “uncomfortable implementation”? From my point of view, OpenID can become a very negative thing if, for example, the website the user’s tying to authenticate to doesn’t offer the ID provider on which the user has an account. It is also possible that an OpenID implementation fits most, but not all. A very clear evidence of this problem is given with websites that offer “Login with Facebook” as their only authentication method – I don’t think this can be called an OpenID implementation, even though Facebook is an OpenID provider. But why is this a problem? People just start based on the premise that all the internet users have a Facebook account. False. I can illustrate this with personal situations… it’s not happened once nor twice, but dozens of times: *le me browsing the ‘net*, *le me finds a website he likes*, *thinks he should signup*, *looks for the signup link*… oh crap, looks like all we get is this:
Call me stupid, “forever alone”, or whatever you want: I might even have a Facebook account, but I may not use it and even if I do, I don’t want all dozens of websites being authenticated with that s*ht Facebook is, and eventually with these websites being able to post to my Facebook wall, access my status, photos or other things “normal” people put on Facebook.
I’m giving this example for Facebook, but the problem goes for other ID providers. There are websites that support open IDs, and a few even say they support OpenID (the standard), but then you’re presented with a “Login with XYX” link where XYX is a single ID provider of their liking. Sometimes you’re lucky enough and you have an ID from this provider, other times you just need to go registering for yet another ID, defeating all the purpose of open identification and OpenID.
Although, there are cases where requiring a login with a specific service is mandatory. For example, on services that are dedicated to changing your Twitter profile background with a generated one, a Twitter account is of course required, so a Twitter-only login makes all sense. Same goes for Google/Blogger/Facebook/WordPress dedicated services, but please, if it’s not required to be tied into a specific service, then just let people use whatever ID provider they want, or provide a traditional signup and login method. Else, open authentication and OpenID might become hassles that drive users away.
Other things can be discussed about OpenID – I can argue that it is unsafer than traditional user/password logins, because if the OpenID provider gets cracked and authentication information gets exposed, then all the accounts authenticated with OpenID on other websites are open to the crackers – much like an user that always uses the same password and username on multiple websites. We can also discuss about these shiny buttons provided by social networks and the like, that allow you to authenticate using your account on them, to “like” or to “share” posts – these are used for user tracking, and seeing what the crowd likes, helping on creating even more directed advertising. There are plugins that block these trackers, and usually some hosts file or iptable rules work well, fortunately (if you don’t use the service from which the shiny trackers are coming).
I do not represent the OpenID foundation, Facebook, Google, Twitter or other OpenID provider. I am not encouraging their use or otherwise; I’m just exposing my very irrelevant opinion on the subject. If you spot any factual or spelling mistake, please contact me or comment below. Thanks for spending some minutes of your life reading this post!
Let’s keep things short as I have lots of things to do. On 30th November, this website was working functional as it had always been since April 2011. Then suddenly, in the morning of 1st December, the server had been reloaded.
At first I thought it was because someone with admin powers at cheapvps.co.uk, the provider of my previous server that hosted this website, reloaded the VPS. But after some searches, I ended concluding the VPS, which luis123456 had given to me in April 2011, was still owned by someone else – and that someone was trying to make use of the VPS.
I found the email of the real VPS owner (in fact, I used it for logging in to the VPS control panel, but I always had thought it was just some random address, because it only had two letters and five numbers!). I sent an email to that address, and some hours ago, I got a reply from David W. – the real owner of my previous gbl08ma.com server, to which I called “hydrogen”. So what happened in fact? According to David, he told luis123456 to “maintain” (and no more than that) the VPS. luis123456, whose real name is Luis A. (so we talk about real names here) was not authorized, to use the VPS. Still, (and I repeat, this is the saying of David), Luis gave me the VPS. Luis never said anything about this: I thought the VPS was some kind of sponsored VPS which the sponsor forgot about.
So, the old server was reloaded. But things were worse for my side: I was supposed to have an automated backup system, but it was broken and I had no time to fix it. Shortening: I have no backups of the old server, except Google cache, which didn’t cache one or two blog posts. Apart from the text of the posts, cached by Google, I lost everything on the server, that is, all files, images, configuration files, scripts, WordPress plugins, themes… hosted within the server. This also includes the few Anti-Aliased fonts for Rockbox, which I’ll have to upload again some day.
I just finished restoring all the blog posts I could. I’m still wondering how to restore comments done by other users on the various posts. But wait, I missed one part, right? How did I get this new server?
Indeed, this is a new server. I say the website keeps being the same only because the matter and intention of it keeps being the same, but in technical terms, this is a whole new WordPress install, on a whole new server. While this has some advantages, it also has lots of disadvantages – you can compare this to formatting a dog slow Windows computer, without making backups first: after re-installing the operating system, you get a clean system but most of your data, configuration and software is lost.
This new server was given by Humza Bobat, Infinity at freevps.us. So yes, now I have two server provided by freeVPS.us; since I know other users will get angry with me by having the admins of freeVPS opening an exception of the one-vps-per-user rule, I must provide some good argument to it.
In fact, I need two server for various reasons. One of the reasons, is that since the virtual servers I own are not very powerful (they are low-end boxes), they can barely handle two websites on the same server – note that we are talking about WordPress on this website, and while WordPress can run on fairly modest setups, I want some speed both for me and for the sometimes 10 concurrent users of the websites (it happens, for example when my stories get featured on Slashdot).
The multiple virtual host configurations, while they work well if you never touch the configuration files, are harder to maintain, in my opinion. But the main reason why having multiple servers is necessary is the following: being servers provided for free, you never know when one goes offline or you loose it forever (OMG! It just happened!). If one goes offline, you can still use the other for temporarily hosting an additional website or at least some informational page. Plus, you should never put all your eggs in one basket: the server that served gbl08ma.com went away with its data, but the situation could be worse if all the things were on it. If I had what I have on my “helium” server, the one that serves 4.l.to, on the server I lost, then I would have lost some hundreds of MB of information that is important (some of it even a bit confidential).
A interesting thing is, I have automated backups of the helium server working, although it gets much less visitors (not including short url clicks) than gbl08ma.com gets. The Murphy’s law regarding backups applies: even if you have backups of your things, they will never include what you just lost.
What about naming conventions? The “hydrogen” name now refers to this new server, while I’ll call the old one from on now “deuterium” 🙂 . The 4.l.to server keeps being called “helium”, as ever. I also have one testing server called “lithium”, but it isn’t used for anything permanent – as I said, it’s a test server.
If you have any questions regarding the data loss on this blog, please post on the comments. I’ll be busy for the next days/weeks/months trying to restore more of what was lost.
I was really fed up with Apache on this server. It would use huge amounts of RAM, even after all the visitors left the website. Having done all tweaks to the memory usage of Apache and PHP, the amount of RAM used would never get below 450MB (out of the 512MB this VPS has). Hell, Apache was consuming even more memory than MySQL!
For those who don’t know, nginx is an alternative, lightweight webserver which is generally used (by many popular websites) as a load balancer. However, it can also act as the single web server on a system, like what Apache and Lighttpd do. I had worked with nginx before on some small websites on low-resource servers, and I was quite satisfied with it. As I explained with an earlier blog post, nginx is great as long as the website you want to serve with it does work with nginx – that is, doesn’t heavily depend on Apache rules or some Apache-specific thing. Sure, those rules can be converted to nginx config options, but I never succeeded on making eyeOS 1.x work fully with nginx.
WordPress is one of the scripts that works best with nginx. Since this website is mainly powered by nginx (although I have some custom scripts laying around, mainly the scripts providing alternative WiiMC internet media), I made my mind and decided I would go through the hassle of switching from Apache to nginx. It wasn’t a big hassle after all: apart from having to restart the server at some point due to a RAM outage, the website wasn’t offline much time, and there was no data loss.
After putting Apache off-use and starting nginx, the server was still using 300MB of RAM. I though nginx couldn’t be using so much RAM, and there was another problem laying around. Turns out to be a problem in MySQL config: I don’t need InnoDB functionality, so usually I add a “skip-innodb” line to my.cnf. The problem was, this line needs to be added under the [mysqld] section and in my case, it was somewhere else. So I moved skip-innodb to the right place, restarted MySQLd, and that’s it:
The server is now using 240MB of RAM, which still fits inside the dedicated RAM (256MB), so I’m not taking any of the burst RAM, which resides in the server swap space. The RAM usage is still high, because I have other things running such as dovecot for mail delivery.
It also looks faster to load pages, but probably someone with a faster connection than me will notice a bigger difference.
Following the latest changes in this website domain, it looks like it will have to edit the URL for this website in wordpress settings again, in no more than a week.
But this time, unlike last time, I’m incredibly happy. No, my birthday is only on 8th October, but the owner of http://freevps.us, dmmcintyre3 has registered for me the domain “myself on .com”, that is,http://gbl08ma.com . Yeah! That’s myself on a .com domain – in case you haven’t understood.
Goodbye malware-false-markings due to the use of crappy free domains! Now I only have to make sure my real TLD is not marked as malware itself.
Now I can freely post URLs for posts in my blog without fear for supposedly containing malware! This is an huge step!
Remember, the Bitcoin donations thing is still valid; at the end, I want to renew this domain next year 🙂
An huge thank you to dmmcintyre3! And to Namecheap for providing cheap domain names.