How on earth is MA / FPC running release Projects... ???

AJack10600

Elite
Joined
Jun 13, 2005
Posts
2,732
Location
Switzerland
Society
SPU ¦ Smart Professionals United
Avatar Name
Larios Semai Niva
Following yet another unbelievable code bug after a release I can not help myself but to be astounded as to how MA or FPC run their Code Releases ...

In every release we have known bugs and issues, and now we get a major revamp and it totally fails...

Now I am not in the gaming industry, but I have some interaction with critical software development.

I work in an area where I am at the interface of the business and IT and I deal with this stuff quite a bit.

Firstly, when we change things in code in an application there is a concept called 0 Impact Testing. This should prevent any old code from being corrupted or changed.
On top of that we have like 3 Test environments, an unlimited amount of test runs in the first 2 until you sign off.
Then another 3 Production Like Test runs in a PRD equivalent Testing environment. Then a regression run. Only when all these items work and are signed off, will an item be promoted to Production...

On top of that we can ICE new code or immediately rollback code when it cocks up....

I just don't understand how they can ignore all this... How on earth do they run their Releases ? I'd really love to know... :roll:

Do they need an IT project manager ? This all seems like schoolboys programming code... Sorry but I can't help it... Error is human, it happens... but in IT you need to minimise the errors to a max. Why does this keep happening ?
 
I suspect they have no "multi-layer" code features with retrofit functions.

That may explain why the bugs fixed in some minor revisions reappear in major versions (started before), the fixes are simply overwritten when the major version is delivered. Really, these frequent regressions are a pain. Like the arm bug who came back yesterday. Honestly, I wait for the Inventory bug to be back in 11.5. or 11.6.

IMO they are glued with some old system, maybe the guy who developed any critical portion (like loot engine) was fired or suicided and they cant switch to a more pro system. I bet the dev guys suffer more than us.


Anyway thats for sure an interesting topic and Id like to have official answer.
 
Sure would!

I'm a Programme Test Manager for a major consultancy in the UK. I'l love to get a call from MA :eyecrazy: Not. Their processes and any form of QA just don't seem to exist.

I would be fired along with my firm if we allowed a piece of software to go live in the state that MA does. Again and again.
 
It has been this way for the 6+ years or so I've been around. You might as well accept this is how they do business.
 
I'm in the business too for long time and is completely ridiculous this situation I have a dome in game with some investment like me more some of us and this keep happen. More investments for future ? Sure not just a crazy do that 2 times
 
i would love to know the answers to the questions here.

im ok with the odd bug here qand there, but the performance of 11.4.4 is completely unacrptable. i havent even been able to install the bloody thing yet.

and the timing for this couldnt be worse. right in the miuddle of a huge influx of newbies.
 
Maybe they don't test because they are afraid that hiring more programmers might cause some one to accidentally leak the "secret formula" of the loot pool... not likely, but since EU is too darn dynamic for it's own good, it would not surprise me.
 
and the timing for this couldnt be worse. right in the miuddle of a huge influx of newbies.

That`s what disturbes me most as well
 
FPC does not code.
How FPC runs our release projects is therefore not relevant to this discussion.

But to the question of test environments; Of course both FPC and MindArk does have test-environments and QA. The fault is not always there.
 
I am in the IT field as well and I will be fired if I delivered such work every single time.

I wonder how the MA techies have an "iron rice bowl" in that they don't seem to have gotten better / changed to better people. I doubt such a system would have worked for any other company, except for MA.......... which sort of manages, somehow.
 
FPC does not code.
How FPC runs our release projects is therefore not relevant to this discussion.

But to the question of test environments; Of course both FPC and MindArk does have test-environments and QA. The fault is not always there.

Is all code coded by MA?
-How will MA handle the bigger deman in both stability, quality and quantity as the universe expand with more planets demanding and requesting things to be coded and implemented.

The fault is not always in QA and testing. This is correct as things don't have to occure as planned.
But: When new (not listed under known bugs) is occuring, as faulty icons, missing ground and crashing
this proves that the new version is NOT tested with condition very similar or identical to the Live environment (where we customers are sitting).

How big is the QA teams in MA or FCP?
- do ypu need a new QA director? (or one at all)

The extended deadline and things that couldnt be foreseen and foretold is also up to a point accepted. But not in the way you guys are doing it.

And if this was only a MA release, why are you Hanne doing all of the community work and why on this forum?
 
Last edited:
FPC does not code.
How FPC runs our release projects is therefore not relevant to this discussion.

But to the question of test environments; Of course both FPC and MindArk does have test-environments and QA. The fault is not always there.

Hanne, you're gonna ruin your good standing with the community if you keep throwing smoke screens out like Marco and the rest. IMO someone else should be speaking to us on this issue.

And YES the fault is exactly THERE. This update was NOT properly tested PERIOD. It may have been tested on MA's "supercomputers" in an on-site test server.... BUT, it was NOT tested in an environment typical of YOUR end users.

I've seen you state in another thread that the release was tested and the fault is not that of QA. If QA tested it in the proper environment they would have caught it. AND QA MUST know by now that programming is NOT one of MA's strong points. (Then again, neither is customer service). You keep popping in on these threads and stating this blatant B.S. you'll do nothing but harm your own relationship with the community. I fear you are already becoming one of "them".

Menace
 
That`s what disturbes me most as well

That's why I'm super upset. How some company turn over billions in last 7 years wait to take this risk when a huge new costumers arrive like never before. One word useless managers
 
and the timing for this couldnt be worse. right in the miuddle of a huge influx of newbies.

Yep yep, this is incredibly bad!!

Imagine how many new players we lost, only because they could not install the game!

Jeeeeezz, this is really not good...
 
And if this was only a MA release, why are you Hanne doing all of the community work and why on this forum?

Because MindArk does not have a Community Manager or a forum of their own.
And yes, MindArk does all the coding.

And YES the fault is exactly THERE. This update was NOT properly tested PERIOD. It may have been tested on MA's "supercomputers" in an on-site test server.... BUT, it was NOT tested in an environment typical of YOUR end users.

I've seen you state in another thread that the release was tested and the fault is not that of QA.

I have not commented on how the release was tested or who's fault it is, please don't twist my words. What I've said is that MindArk does have both Test enviroments and QA personell and that the fault is not always in the hands of the QA personnel. Regarding this case in particular I don't know where the fault is and (even if I did) I am certainly not going to throw someone out to be attacked by the community if there's been any error done. Not because of "smoke screens" just because its not a very decent thing to do.

I don't blame you collectively blaming the company as a whole, I think we can all agree this release was "less then optimal", but I certainly do not like when people jump to conclusions on who to blame within the company. Let MindArk handle their own personnel based on the actual facts, and try keeping in mind that when you blame someone it may very well be aimed at the wrong people since you do not have those actual facts.

Understand this, I am here to communicate with you but I am certainly not here to provide you with someone handy to blame!
 
Yep yep, this is incredibly bad!!

Imagine how many new players we lost, only because they could not install the game!

Jeeeeezz, this is really not good...

Yup.... scary. I guess RT would have been down as well with this update..... can you image 10k Noobs all with fresh accounts downloading this for various planets? OUCH!

Menace
 
hanne in all respect, as this isnt a fpc release, you shpuldnt have to be the one posting here. someone from ma needs tp get somewhare and give us some answers.


how the hell can eu have any hope of surviving running 9+ planets when they release stuff like this?
 
hanne in all respect, as this isnt a fpc release, you shpuldnt have to be the one posting here. someone from ma needs tp get somewhare and give us some answers.

This thread is named "How on earth is MA / FPC running release projects".
 
Because MindArk does not have a Community Manager or a forum of their own.
And yes, MindArk does all the coding.



I have not commented on how the release was tested or who's fault it is, please don't twist my words. What I've said is that MindArk does have both Test enviroments and QA personell and that the fault is not always in the hands of the QA personnel. Regarding this case in particular I don't know where the fault is and (even if I did) I am certainly not going to throw someone out to be attacked by the community if there's been any error done. Not because of "smoke screens" just because its not a very decent thing to do.

I don't blame you collectively blaming the company as a whole, I think we can all agree this release was "less then optimal", but I certainly do not like when people jump to conclusions on who to blame within the company. Let MindArk handle their own personnel based on the actual facts, and try keeping in mind that when you blame someone it may very well be aimed at the wrong people since you do not have those actual facts.

Ok.... let's take the blame off QA then.

1.) MindArk CEO's fault for hiring people not capable of performing Quality Assurance duties.
2.) FPC's fault for not requesting a half hour of Test Time by their own QA in order to ensure the end users will be happy KNOWING that MA's coding history is not exactly flawless.
3.) Computer Hardware manufacturers for not making systems "good enough" for QA to be able to perform the required duties.
4.) The end users for not quitting 7 years ago so we wouldn't have to worry about this at this point.
5.) ISP because it briefly lagged out the connection during the upload of the patch to the data base (Oh wait, scratch this one cuz QA should test AFTER the data is uploaded).

Get my point? QA's SOLE PURPOSE is to ensure the end users get a good solid product and to FIND the errors like this.

Hanne you CAN'T take the blame off QA on this one. They are our last line of defence and TBO, FPC should be testing these releases with their own QA before it gets to the end users as well. So it's BOTH the fault of MA QA, and FPC QA.

A car dealership will NOT sell a BMW that won't start. Even though the dealership did not make the car, they check it over FIRST, and THEN put it on the lot for sale. Now the Volkswagon Beetle car lots in Mexico.... they get the car from the factory, put it on the lot... and if it don't start what the hell do they care..... Wait a minute................ ;)

Menace
 
The Standish Group from Massachussets (USA), formed in 1985, makes a study about failures inside Proyectos of IT. Using data about 50.000 proyects, in 1994, they obtained this results:

  • Aborted Proyects: 31 %
  • Proyects with problems: 53 %
  • Succesfull Proyects: 16% (But in this, only is satisfied 61% about the expected functionality)

Trying to enhance the IT Proyects, in the next 10 years, and after using methodologies and technologies very best than the older. In 2004, results continued not good:

  • Succesfull Proyects: groth to 29 %.
  • Fail Proyects: 71 %.

And in the Standish Report the ten principals causes implied in the fails are this:

  • Poor user participation
  • Incomplete Specifications
  • Incomplete Requirements
  • Frecuently Changes in Specifications and Requirements
  • Low level linked to Executive Support
  • Technological Incompetence
  • The expectatives are not realists
  • Unclear objetives
  • the chronograms are unrealistic
  • New Technologies

Today (2009) Roger Sessions says that mother of all fails, is no more no less, than the Complexity of the Proyects.

The Sessions Report explain how the complexity is the cause of fails in cronograms, budgets, and expectatives. And all this fails, in all the world, it has a cost aproximately USD $500.000 millones per month.

The solution from Roger Sessions for this problem, is to divide the problem, dividing the complexity of the project with clear and simple goals, then, it is possible to reach a better efficiency with less costs and time.

Well, all this say to me, that the problem is not only a thing about quality control and many tests.

I hope Mindark can management this problems, in the future, better than today.

;)


Following yet another unbelievable code bug after a release I can not help myself but to be astounded as to how MA or FPC run their Code Releases ...

In every release we have known bugs and issues, and now we get a major revamp and it totally fails...

Now I am not in the gaming industry, but I have some interaction with critical software development.

I work in an area where I am at the interface of the business and IT and I deal with this stuff quite a bit.

Firstly, when we change things in code in an application there is a concept called 0 Impact Testing. This should prevent any old code from being corrupted or changed.
On top of that we have like 3 Test environments, an unlimited amount of test runs in the first 2 until you sign off.
Then another 3 Production Like Test runs in a PRD equivalent Testing environment. Then a regression run. Only when all these items work and are signed off, will an item be promoted to Production...

On top of that we can ICE new code or immediately rollback code when it cocks up....

I just don't understand how they can ignore all this... How on earth do they run their Releases ? I'd really love to know... :roll:

Do they need an IT project manager ? This all seems like schoolboys programming code... Sorry but I can't help it... Error is human, it happens... but in IT you need to minimise the errors to a max. Why does this keep happening ?
 
Last edited:
[...]Of course both FPC and MindArk does have test-environments and QA. The fault is not always there.

Technically the fault is NEVER at QA as the only thing they do is to check for faults that other people make.
That is why QA is among the most unrewarding jobs, because if they do their job well it means more work and sometimes even much aggravation of everyone else in the company.

Many companies make the critical mistake to believe that the support staff is the logical choice to do QA work,
because they need to stay in touch with the tech staff anyway.

This just doubles the stress on these poor sods and wears the out far too quickly.

To have the coders do the QA is to set up a thief to catch a thief - I say this as a coder myself.

Frankly there is no really good way to do manual QA, even if you hire dedicated staff to do it.
So the best way is automation of the tests.
Now the argument "but that and that is too complex to be automatically tested" is a danger sign,
you need to translate it to "this is so complex and fragile that it will very likely break"
and this means such issues are so critical someone has to look into them and either find a way for automated testing
or even have the stuff in question designed in a different way.

No one in the QA department can be blamed if errors escape unfixed because of lacking processes, man power
or ignorance of reported errors.
But this means that the area of QA is lacking and needs attention.

My assumption is that MA is doing what most companies do, declare some employees of different departments responsible for some quality checks and when the shit hits the fan the blame game is played.
Anyhow if such a big heap of shit can happen, it means whatever structure QA does have, it is lacking severely and needs a review.
 
Is the test environment a mirror of the production environment?
 
FPC does not code.
How FPC runs our release projects is therefore not relevant to this discussion.

But to the question of test environments; Of course both FPC and MindArk does have test-environments and QA. The fault is not always there.

Well for us customers it should not matter who's fault it is in the end. As you already agreed this release was "less than optimal" the question remains how this can be avoided in the future. But i don't demand discussion in the open on this forum, since i believe this is stuff which needs to be discussed in Gotheburg ( and call me optimistic, i'm sure they do ).

Anyway, what concerns me most is basically the picture such failures draw on whole Entropia Universe since this Environment is supposed to deal with not even small amounts of money. And ppl might ask when such updates where relatively easy mechanisms are dealt with fail how would the update fail if core mechanisms of the engine functionality are upgraded. I think you get the point.

Kind regards
 
I bet this is what it looked like at the office. The role of MA being depicted by Bill of course. :)


sorry... just had to necro this video as it's so fitting to almost EVERY release we get!

Menace
 
What I've said is that MindArk does have both Test enviroments and QA personell and that the fault is not always in the hands of the QA personnel. Regarding this case in particular I don't know where the fault is and (even if I did) I am certainly not going to throw someone out to be attacked by the community if there's been any error done. Not because of "smoke screens" just because its not a very decent thing to do.
!

Absolutely right Hanne, but speaking from experience it's the Test / QA who take the brunt of things.

This release might have passed through all the test environments successfully passing all relevant regression, new system tests and quality gates, BUT then been deployed in a incorrect fashion in to PROD which has caused all the problems we have seen.

These kind of scenarios do happen. The key to not letting them happen is checkpoints, checklists and a 100% agreed upon deployment plan with a rollback option.

Of course this implies that these processes and the right staff are in place to implement them.
 
This thread is named "How on earth is MA / FPC running release projects".
yes, but we alll know many still see fpc and ma as one, wrongly.


my comment was more aimed at the fact clean up after this vu shouldnt be completely up to you and ma really do have some explaining to do.


Serious cash is involved with eu, so ma needs to take things like this seriously and treat its investors with the respect they deserve.

and lets be honest, this issue isnt exactly a one off is it...
 
I know we are not suppossed to talk about "other games" too much here... but... MA should really look at their competitors ways of doing thing.

Diablo II (pre WoW) was one of the best games out there. The RCE is not "official" there, but I suspect that it was one of the types of games that got the original folks at MA excited creating the RCE in the first place.

DII came out in the year 2000. They had a few various versions done after that... and there was much anticipation about DIII for several years now... but they DID NOT RELEASE DIII yet because they want to do it right...

http://us.blizzard.com/diablo3/faq/#3_1
When will Diablo III be released?
It's too early to estimate Diablo III's release date. As with all Blizzard Entertainment games, our goal is to create a game that is as fun, balanced, and polished as possible. We intend to take as much time developing Diablo III as is necessary to ensure the game meets our own high expectations and those of our players. We're aiming to release Diablo III on both Mac and Windows simultaneously in as many regions as possible, and to localize the game in several languages. We'll have more details to share about countries, languages, and specific dates as we get closer to release.

In other words, they are taking years to get it right because they know that when they do get it right, the customers will love it. EU seems to always get an influx when there's some massive sale like the one of CP or CND... BUT... how long do all of those customers stick around? If there's a lot of broken systems, stuff that's been broken for over a year that's not fixed yet, and various problems in releases of new versions (such as a repair tool that doesn't work), there's a major problem... Instead of releasing ASAP, take the time to do the QA tests BEFORE releasing a product that is really only at the beta stage as seems to have been done in the past... It does take time and money, both of which are not limitless, but if you can release an awesome product it might just well be worth it because you might just see more customer retention. :)
 
I bet this is what it looked like at the office. The role of MA being depicted by Bill of course. :)


sorry... just had to necro this video as it's so fitting to almost EVERY release we get!

Menace


Hahahaha,, really a very good video. and its explain how a genius in IT management can do very wrong too.
 
My very personal point of view.

We don't know what happened. We don't have any clue about what was the testing before, what kind of problem they stumbled upon and how difficult it was to solve. We don't have elements to judge any dept of MindArk. And personally, I don't give a damn about the culprit.

Folks, this is a GAME: can't we deal with 1 day downtime of our GAME? Yes, we can. Many other games have such downtimes, and nobody yells at the moon.

But, last two things.

1. RCE, doesn't this word ring a bell at MA offices? What if someone *IS* running an investment here into?
2. What really makes me angry is the usual silence by MindArk. We've been left alone playing with FTP sites and filename guessing for hours. That's very, very, very unprofessional.

I'm really very disappointed by the indifference MindArk has with their customers.

Sorry, Hanne, but I personally would like to see people from MindArk in this forum. Too many times the problems (probably the 99% of the times) are on MA side and nobody ever pops up to talk with us.
 
Because MindArk does not have a Community Manager or a forum of their own.
And yes, MindArk does all the coding.

Hanne, I woant have a go at you, cos clearly this is not your fault. I do suggest however that since you are a Community Manager who should intereact and feed back from the community that you really raise this issue with MA. When their core system fails, they need to have some sort of official communication somewhere, online, on the website or on here. I understand that MA do not want to interact with the End user as you are facing the end user. But this leads to this funny situation that my service provider is not responsible for the error, but the service provider of my service provider does not talk to me... so who do I talk to ? I am sure you understand this...

I appreciate you are communicating to us though!

I have not commented on how the release was tested or who's fault it is, please don't twist my words. What I've said is that MindArk does have both Test enviroments and QA personell and that the fault is not always in the hands of the QA personnel. Regarding this case in particular I don't know where the fault is and (even if I did) I am certainly not going to throw someone out to be attacked by the community if there's been any error done. Not because of "smoke screens" just because its not a very decent thing to do.

I don't blame you collectively blaming the company as a whole, I think we can all agree this release was "less then optimal", but I certainly do not like when people jump to conclusions on who to blame within the company. Let MindArk handle their own personnel based on the actual facts, and try keeping in mind that when you blame someone it may very well be aimed at the wrong people since you do not have those actual facts.
Understand this, I am here to communicate with you but I am certainly not here to provide you with someone handy to blame!

I dont' think anybody is jumping to conclusions here. As you say, we do not have facts so we can only speculate, but what is for sure, and this is what I said, the way these Releasese are run and the way they are tested is insufficient... Fullstop.

And even if the error is in the code promotion to PRD and even if testing was 100% fine, MA might need to think about how to chip up the code so that they can promote and reverse individual code lines. The current Code Pack makes the whole system unflexible and dangerous to control. This is why we get these recurring bugs. This is an IT code design issue. And you sure can feed that back to MA... they should seriously think about this.

As an FPC community Manager you have all right to be on this forum in this thread and I would welcome if you feed back some of this customer feedback to "your" service provider because here, clearly he is endangering your business model with his actions...

They really need to do something at MA to bring the word "Quality" up in their next meetings. We don't want more features and more gadgets for now... Just get it right ! Bug fixing has been a never ending story ... and how high is it on the agenda ? You know for yourself ... You have been a player... but I guesss bugfixing does not generate revenue and hence is not top priority...
 
Back
Top