Minecraft

End of year network review

  • Comment
At the start of the year I outlined my plans for 2013 with our focus this year on improving the stability and reliability of the service on offer. Now, quite clearly, this hasn't exactly gone to plan. We've hit snag after snag along the way that has caused massive delays, dampened spirits and made for a rather sorry year for us. We're having our most fun when we're coding exciting new features and updates to the sites that you can actually use. We do not have fun working on behind the scenes stuff and it's made worse when it doesn't actually work out, but it really needs to be done.

We started the year with a hefty sum of money to invest that had been saved up from before Skyrim was launched, primarily from Premium Membership fees. I'd been saving this money over the course of a few years so we could roll out a whole new server setup that would not only provide us with the power we needed but also the quick expandability necessary to deal with the ever increasing traffic and load placed on the servers. This was most definitely the right course of action and I have no regrets on that decision at all. Look at it like you would the gears on your car. In second gear you can only reach a certain speed before you start redlining, you can't go any faster without switching to another gear. Problem is, when you've only got a 2-speed gearbox with no potential to go to third gear the only thing you can do is rip out the old one and install a new 7-speed gearbox. And that's what we did. We installed our new gear box, switched into third gear and opened the way to fourth, fifth, sixth and seven gears in case we need it in the future. What we didn't take into account was just how hard it would be to switch gears and how many unforeseen circumstances you can actually run in to.

We ordered the database cluster servers at the beginning of February and received them at the beginning of March. We initially thought it would only be a matter of weeks to get things all set up and have the sites moved over to the new database cluster. 3 month's later, at the end of May, we thought we had it but after a straight 48 hours of work we couldn't get it to work properly and had to accept defeat. At which point we paid a rather hefty sum of money for two separate professional consultants to come in and take a look at the setup. It wasn't until the end of July that we finally managed to get our database cluster setup completed and moved to, 5 months after we'd originally started working on a project we thought would take 2-3 weeks. In September I proceeded to buy a further 2 cluster servers, taking the setup to 5 servers, each running 96GB of RAM and dual processors for a combination of 480GB of RAM and 80 CPU cores running at 2.1Ghz (or a hypothetical 160 CPU cores with threading).

The move to our database cluster helped to remove one issue only to highlight another major issue that needed to be rectified. While we could now serve the database requests we were struggling with a bottleneck with the HTTP requests. Thankfully we'd already earmarked funds for a move to a Cloud setup that would form the basis of our expandability in to the future. Towards the end of July I was in discussions with our server provider to have a Cloud setup specially requisitioned for us. They'd need to build it from the ground up for us so we weren't actually given the keys until the middle of October. All the while the sites on the old setup were on their last legs. Well, they weren't really standing up so much as they were spluttering blood all over the place with multiple puncture wounds in this analogy. Once we got the keys we needed to run extensive tests and mock runs and get it ready for an eventual move over, which didn't occur until the start of December to coincide with our centralisation of the sites. By centralising the sites we managed to make our future jobs a lot easier. No longer were we dealing with 20 different databases for 20 different sites on multiple separate servers, we were now dealing with 1 database with 1 site on a distributed, but for all intents and purposes centralised, server setup. This simplifies things drastically with the down-side being that the problems that were originally localized mainly to just Skyrim Nexus were now problems for all the sites,



Traffic is at an all time high, as usual...



If we were still on the old setup then you wouldn't be reading this right now, even if it did take you 5 page reloads and 20 minutes to reach this page. The old setup would simply have been incapable of handling this Christmas/Steam sale traffic. So you can picture this situation like a dramatic Indiana Jones scene if you wish, there's Harrison Ford (the Nexus sites in this analogy), stood on a crumbling platform about to collapse at any second. With a crack of his whip he hooks on to a low dangling tree branch and swings on to a new platform. Albeit this platform still doesn't seem too stable but it's a darn sight better than the previous platform he was stood on which has now fallen into a seemingly bottomless chasm. And this is where we find ourselves at the end of 2013, stood on a precarious platform right now, but the right platform, from which we need to build from and solidify our position.

The situation is extremely infuriating for us. I've said it before and I'll say it again, we know it's frustrating for you when you can't download your mods or update your file pages but my god, try working 18-20 hour days trying to sort it out in the background all the while trying to answer people's ferocious questions and trying to remain calm. I can't do it and I blow my lid sometimes. It is infuriating to have spent this much money and time on an issue that still isn't resolved. But it will be. It's not a matter of if, it's a matter of when.

As users you get understandably upset. Lets do a quick Q&A on the regularly asked questions on this topic:

Q: Isn't it about time you spent more money to fix this issue?

We've spent close to £150,000 ($250,000) on sorting out our stability issues this year. Aside from the fact it's no longer about the amount of money that needs to be spent but what we need to do with what's already been bought, do you have a spare extra £150,000 lying around that we can use? Please? No? Didn't think so (if you do, by all means get in contact. I can offer you a bag of Nik Naks and half of a Panettone as payment).



Q: It was way better before. Why not just go back?

If you're talking from a Skyrim perspective: It wasn't. Your memory has failed you.

If you're talking from any other Nexus site: it probably was, although it wouldn't be right now in the midst of all the Christmas traffic. You've been lumped in with the same lot as Skyrim now. Your fate is linked to Skyrim's as much as Skyrim's is to yours. It's a necessary temporary down-side in the interest of future stability.



Q: Why do you keep performing "maintenance" at peak times?

We put the sites into "maintenance mode" to give the servers time to catch their breath and recover. If we put the sites into maintenance mode they'll recover 90% of the time within 1 minute. If we don't then the problem will remain the same, or get worse, for hours.



Q: Why don't you work on fixing the issues rather than releasing new NMM versions/updating the sites/eating/sleeping/leaving the house?

I think by the 20,000+ words written on this very topic this year alone, and indeed the regular maintenance updates and redirects, you can infact see that work is being done. However of the 5 people (including myself) working on the sites only 2 of us can help in this field.

Telling us to not release an NMM update until the site stability issues are rectified would be like telling a UI designer at Microsoft to not release any fixes or updates to the Windows UI until they've sorted out all their security bugs. The UI designer doesn't work on that aspect of Windows, he doesn't know anything about it and he can't be roped in to help. Telling him to stop working until the bugs on a completely different aspect of the program are sorted would be dumb. And a waste of money.

I'd like to have another full-time person on board to be able to handle this side of things but there's a few barriers in this regard. Money. The time it would take to hire the person. The time it would take to bring that person up to speed. All negatives for issues we need resolved right now, and not in 3-6 months time.



Q: I haven't been able to login to NMM since you updated the sites at the beginning of December, what gives?

So you found the forums and the "new topic" or "reply" buttons but you didn't notice the multiple news posts and indeed the 1,000 other threads and posts about this issue raised in the past week? Sometimes you really need to help yourself. Or do a search! You need to update to version 0.46.0 of NMM. Download it from the site and simply install it over your current NMM installation, making sure your folder locations are the same as your old NMM install. If you do this you won't lose any of your mods.

Having said that we are going to make documenting things in NMM a bit better over the next year.



So what has 2014 got in store? Well hopefully we can resolve these issues once and for all very soon and get back to doing what we enjoy doing. I don't think I need to say anything more on that topic.

At the start of the new year we're going to be decommissioning all our current file servers as they're close to their space capacity, replacing the current 19 file servers: 15 for everyone and 4 for Premium Members with 23 brand new ones, 20 for everyone and 3 for Premium Members. Why a drop for Premium Members? A couple of reasons. We've bought Premium servers in various locations across the globe to act as alternatives to the normal servers. These haven't worked out as well as I'd hoped and they're often unreliable for people. I'll be replacing them with some top quality stuff in Dallas, Washington and Amsterdam that will cost twice as much. I want Premium Members to realise that the real bonus on the downloading front isn't the Premium-only servers but instead is the speed cap being removed and the ability to multi-thread your downloads in NMM. Some Premium Members come undone because they will only use the Premium-only servers when the normal servers are equally or often times better suited for their connection.

We haven't touched the Image Share section in quite some time. I want to overhaul many aspects of the Image Share section and implement some updated features like galleries, and removing those horrible image pages some users make that scroll on and on into infinity as the author has added their entire back catalogue in to the image description. We'll replace it with a more suitable system that hopefully won't diminish what some people try to do with their descriptions but instead simply make it a lot more aesthetically pleasing and manageable for users. I'll begin consulting with certain users from the Image Share soon, likely to be ones that cause as little drama as possible. No need to get in contact with me, I'll get in contact with you if I'm interested. Image Share drama is the worst drama on the Nexus and I try to steer well clear of it as much as possible.

I also want to begin work on a Videos section, much like the Image Share section but for YouTubers to showcase their work on the Nexus. I've no interest in hosting people's videos so this system will link straight into YouTube's API system. I think it'd be great to have all the top Skyrim mod video authors, for example, having their own channels on the Nexus from which they can easily showcase their work and the work of the mod authors and you can easily keep up to date with all of them. It won't take away from their subscriber base, it will simply augment it, and best of all for us it won't cost us any valuable bandwidth or server resources.

On the NMM front we're still working on the 0.50 update with it's profiling features. We want to get it right, and we also want to create a full backup system for users so that they can revert to a legacy system if needed. We've managed to gather valuable feedback from the people using the 0.50 alpha but what we cannot ascertain is how successful it's been for users as a percentage. We don't know how many people it hasn't worked for, e.g. how many people couldn't update to 0.50. We want to create a version of NMM that we'll likely dub NMM Legacy. This will be a version of NMM that we shall feature-freeze at 0.46.0 for ever but that we'll always ensure is able to access the NMM web services (for logging in/downloading/mod version checking). It's our hope that each time we release a major update to NMM we'll ask you if you want to backup your current copy of NMM. If the update fails for any reason then your backup will work with NMM Legacy, so you won't have lost anything.

We've also got that NMM design overhaul to look in to. I'll be writing up a blog post about NMM soon as it seems some people are upset we're still in Beta after a couple of years. I'll set the record straight on that one because we're not coming out of Beta any time soon.

I've absolutely no idea how long the stabilization of our services is going to take. I'm not even going to fathom a guess because each time I do, each time I assume we're close, we hit another snag. We've now got a huge backlog of stuff to do after the centralisation, however, so I hope we can get some updates out thick and fast soon.

I hope your Christmas, New Years and indeed your entire year has not been as stressful (or expensive) as ours has. I hope that 2014 brings us all more success.

296 comments

Comments locked

A moderator has closed this comment topic for the time being
  1. esharp
    esharp
    • member
    • 0 kudos
    Thank you for all your hard work! I really enjoy the layout, and use nexus exclusively for Skyrim and Fallout modding. Not knowing anything about IT I had no idea it was such a struggle to keep this site afloat, I'm so appreciative of the efforts the nexus team makes to keep me happily dl'ing! <3
  2. Ashenfire
    Ashenfire
    • member
    • 5 kudos
    Wow, that is tough. Sounds like the issues we have on the creation kit scripts, collaberation and production. Only you have it on a massive scale.

    I am sorry for your headaches. I know what it means when you have all these ideas
    you want to make public but spend 90% of the time haggling over every single
    'mechanical issue' that is required to make you go one step forward.
  3. WaltC
    WaltC
    • premium
    • 5 kudos
    I rarely comment here, but this is one of those times when I think I have something worthwhile to say... Thanks so much to you and all of your cohorts there for whom these sites are obviously a passionate labor of love. Your blood, sweat, and tears, of course, is deeply appreciated by every person who uses this site--especially, the non-premium members, for whom the Nexus sites are much like Christmas every day of the week. And thanks to the premium members, too, who help to make your toiling fruitful and productive.

    Looking back to the first day I signed up with a Nexus site--a long time ago, so long ago I don't recall the game I was modding--back to that day when I was convinced that Nexus was run by some sort of clandestine Chinese shadow figure who no doubt was planning something nefarious for the world at large--a' la' Dr. No--all I have seen since is progress and the tangible results of the hard work you guys have been doing. Progress and nothing but progress. (And of course I was relieved that my sinister Chinese suspicions were *apparently* unfounded!.....?)

    Thanks for the hard work and these sites as it all is very impressive, and it's impossible to estimate the positive impact your sites have had on PC gaming in the last few years. Every single game developer worth his salt who makes modifiable games owes your sites much, and I hope that all of them are premium members as from their point of view a Nexus premium membership is more like an investment in the longevity and the support of their own products!

    I have no complaints and look forward to your no-doubt very bright future!
  4. AuronH
    AuronH
    • member
    • 0 kudos
    Just wanted to say keep up the good work

    Having a site like this helps ease people with no experience with modding (making or using), it makes them feel happy to try when you explain they are easy to remove and just how many people make them.

    The truth is the community itself generates the content (the most important part), but there are reasons that modders and users alike keep coming back to the nexus, even many flatly rejecting other sources such as the steam workshop.

    Anyway I am grateful you bother to put in this much effort, writing software and managing the website to a small start-up myself I can imagine just how difficult it can be at times, both dealing with technical problems and user expectations (often at the same time).

    Thanks ^_^
  5. MrGrymReaper
    MrGrymReaper
    • premium
    • 47 kudos
    @Dark0ne - Sorry if this seems to potentially be off topic. Though then again may be not but I sent you a PM with a link to some suggested reading material on this.
  6. amc2002
    amc2002
    • supporter
    • 0 kudos
    I found this site/community when I decided not to buy the new Xbox One, opting instead to upgrade my video card and get back into computer gaming after years with the 360. 
     
    Very late to the party, I know, but I just can't believe the community you have here, and the amazing Nexus Mod Manager. You have made things so easy for us. 
     
    Playing Fallout3 Wanderer's Edition with Project Reality and Mart's Mutant Mod, along with a bunch of other enhancements has been just incredible. 
     
    For the past few days I haven't been able to log in from NMM, so I stopped here to see what was going on. 
     
    All your work is very much appreciated!!! I just had to stop and say thanks! I'm sure it's a thankless task a lot of the time. Happy New Year! Hope 2014 is much smoother for you. Sounds like you've set the foundation for it once the Steam sales are over. I heard Steam had troubles of their own over the holidays, so if a several-billion dollar company is having trouble, I guess you're in good company!
    1. grotte
      grotte
      • member
      • 40 kudos
      berk if you enjoy mods, at last endorse them man

      Posts: 1 Topics: 0 Files tagged: 0 File images: 0 "Endorsements given: 0"

      it's weird
  7. Sdesser
    Sdesser
    • premium
    • 13 kudos
    I'm a bit late, but happy new year! And I don't mind the downtimes, I'm just happy that we got this great modding community going on here. That said. As a university student, I spent a good chunk of my last years tax returns (read: non-noodle money) to get my self a lifetime premium. Thank you for the hard work and I hope this site will be here for many years to come! Now... to plan for my next modding project.
  8. badpeoplesuck
    badpeoplesuck
    • premium
    • 1 kudos
    It's not the fact that it's in open beta, it's the fact that you hold a monopoly on the Skyrim modding community, and we are basically forced to download from this site, and use this MOD manager.

    If you are going to hold this monopoly, and force people to use your website and software, by now after 2 years you should of gotten NMM to work correctly.

    You should also consider letting people use the MOD manager and download without signing up or separate your FORUM bans from the website bans, so people can still get MODs.

    There is no reason to ban someone from at least downloading MODs. This creates a big problem, when you ban people from this website for whatever infraction.

    You are cutting them off from the majority of MODs for the community, which is morally wrong in my eyes.

    The people who speak out as well, whether they are premium or not, you ban them. If you want better support and not having to write novel like explanations, change your policies and consider not banning people for minor infractions.
    1. Dark0ne
      Dark0ne
      • Site Owner
      • 2,887 kudos
      You posted this in the wrong news topic...

      It's not the fact that it's in open beta, it's the fact that you hold a monopoly on the Skyrim modding community, and we are basically forced to download from this site, and use this MOD manager.


      Any such monopoly hasn't come from anything I've explicitly done. I don't tell people to exclusively use the Nexus. I'm not making people sign exclusivity contracts. In fact, my posting history both here and on the Elder Scrolls forums is full of examples where I've told mod authors not to do this, and to use as many sites as they're willing to use. I wipe my hands clean of that one.

      If you are going to hold this monopoly, and force people to use your website and software, by now after 2 years you should of gotten NMM to work correctly.


      "You hold a monopoly ergo NMM should work correctly after 2 years" makes absolutely no sense.

      The biggest problem is, when you ban people from this website for whatever infraction, you are cutting them off from the majority of MODs for the community, which is morally wrong in my eyes.


      I think the biggest problem is the fact the sites have been unstable. I couldn't give a flying crap about the people who've broken our rules and weren't able to file an unban request that was acceptable.

      The people who speak out as well, whether they are premium or not, you ban them.


      Depends how they speak out. Speaking out on these sites seems to go hand-in-hand with breaking our rules, in that people who speak out invariably end up swearing, insulting staff, insulting other users and generally proving why they don't deserve to be in this community. It's not the act of speaking out that gets people banned, it's how they do it and what they say. There's plenty of examples of people who have spoken out but have done so in a way that was respectable. They're still here, even if they're just downloading/lurking rather than talking out of their own choice. There's plenty more examples of people who have spoken out and completely broken our rules in the process. They get banned. Rules are rules whether you think you're protesting or not.

      If you want better support and not having to write novel like explanations, change your policies and consider not banning people for minor infractions.


      Makes no sense. I write novel like explanations because I like keeping people informed about what I'm talking about. This news was nothing to do with banned users and wasn't sparked by the way we ban people in any way. You sure do seem to have a mighty big chip on your shoulder to want to talk so much about banned users in a 2,500 news article that made no mention of them.

  9. Sekigahara
    Sekigahara
    • account closed
    • 0 kudos
    As good as it is there are two known problems plaguing the Nexus - ongoing networking issues and frigid member relations. Glad you are working on the technical side.
  10. DRThree
    DRThree
    • premium
    • 0 kudos
    As a fellow IT professional I do understand the complexity of keeping a site like this afloat.
    There are no easy answers and everything has its down side.
     
    All in all I believe you are doing a great job.