Location: Forum > Announcements > News > thread
Crashes issues
Thread in News started by Marcus, October 31st 2018, 12:15 CET
Page:
1
Marcus
Administrator

Reputation
Posts:
38

Joined:
September 2018
As many of you probably know, we have some crash issues in MasterCores.

First off, don't worry about your exp/skill progress, the game is backupsaving so frequently that neither your char nor houses will face any crazy rollsbacks.
Second, don't worry about duping items, these crashes is not created by players directly, there's no "group" of people who crash the server to dupe millions of gp or items.
We've been monitoring item activity on daily basis and looking for possible duped items, but so far nothing dangerous has been found since we cleaned the game from the previous bank issue.

Now, to my point about the crashes..
Like a player could have asked: "Why don't you fix these ****** crashes!?!?!".
In the last month, we've had a few different types of crashes happening, and so far, all but one type have been fixed.
The one that remains, is one that seems to be incredible hard to solve, yet we're not giving up!

The crash is a memory crash, it crashes because it tries to delete something that isn't there anymore.
I've made a picture explaining a little what the crash is about.

As you can see, the entire game is made up of memory-slots.
And at some point, one of the slots goes corrupted which crashes the whole thing.

We have logs about crashes, but they don't point towards where the source of the problem comes from.
And we yet don't have any way to reproducing the crashes.
If we had, the crash would be fixed VERY quickly.

These crashes happens frequently in Azera/Demera (very populated worlds), but rarely happens in Benera/Canera.
Benera have had 2 crashes in total, in a month, Canera have had 1 crash in a months period.

Now since how hard this one have become, I've put a $500 bounty to whoever can get closest to a solution (even a critical tip would be enough to collect that bounty, as far it's helpful in finding the problem).
Now that's a lot of money and I've had multiple C++ devs taking a look at it yet nobody has come close to an answer what's causing it and how to prevent it.
And this is where logic plays in, some people think there's people abusing the crashes, but then.. who would not share their info how they crash a game for $500?
It's near impossible to dupe anything, and if it would happen I'd know about it and the player would get deleted if it happened multiple times, or just robbed of the items, if it was a one-time thing.

What happens to you during a crash?
practically nothing, your exp/skills/inventory and even house, will still have whatever it was before the crash.
However, a crash closes the game, meaning the floor is reset, so the only thing players really lose, is lootbags.
Which is why I'd recommend people not to keep valuable items on the floor, unless they're inside a house.
Additionally, if you're online during a crash, you will be sent to your temple for your own protection.
So you don't login to a monster area that's freshly respawned and risk dying due to it.

No matter what, I will never stop looking for a solution for this issue, and I hope you can understand the huge pressure this causes towards the staff.
And I hope you can enjoy MasterCores even thou our issues, and we all hope we find a solution soon!

Sincerely,
Marcus


October 31st 2018, 12:15 CET
Vorp

Reputation
Posts:
1

Joined:
October 2018
is there a possibility to simply set this as when someone logs out there won't be possibility to reserve the same slot of ram by another player instantly but with a short delay?


October 31st 2018, 13:29 CET
Derot

Reputation
Posts:
2

Joined:
October 2018
 
Ascer here.

I know it o.O it's this same when you try example read mem value from non existing game region by registry.
It's a exmaple of OPcode.

mov ecx, client.exe+12056
lea ecx, [ecx+04]
mov ecx, [ecx]
<crash>

In this way we need to check if addr from [ecx] is valid

mov ecx, client.exe+12056
lea ecx, [ecx+04]
cmp ecx, 10000  ---> value depend on location addr.
jl region ---> go store all crash reason or smth
mov ecx, [ecx]
<continue registry>
 
btw. nice caesar 8 but don't worry I don't gonna reopen Rifbot


October 31st 2018, 13:37 CET
Mrparrot

Reputation
Posts:
6

Joined:
October 2018
Hello,

I am not from the dev staff, and it is hard to give any insight to you guys without the source code, but I am interested in helping.

So according to your picture, you store the logged in players into an array. There is a lot of possibilities of implementing such data structure, but I think it is done by:

struct Player {
    // stuff here.
    char name[NAME_MAX];
}

struct Player player_arr[SIZE];

Let's say your initialization retrieved the player name from the database, and its name length is just equal to he NAME_MAX. When writing it to the Player.name attribute, it is a very common mistake to think that it will fit into the buffer, However, what will happen is that the null character will be placed outside name buffer. This will override the first byte of the next struct Player. If the first attribute of the Player struct is a pointer to somewhere, you will get an invalid reference and a segmentation fault.

I think your problem is somehow just a small variation of this problem. What you can do to check it in UNIX systems is to run the server binary with Valgrind. It will monitor all memory accesses, and if something goes outside the bounds, it will complain. Note, however, that it will slow down things massively and therefore it must not be used in production.

With regards

*Flyes Away*
 


October 31st 2018, 14:23 CET
Dondomdon

Reputation
Posts:
1

Joined:
October 2018

Can you pastebin the logs?

If they have too much vial information, you can take that out and paste the rest if people get some ideas?

Since it's serversided game, I dont think many things inside the game can crash itself, have you looked on the server connection logs? as in ddos/portflooding etc?



October 31st 2018, 15:51 CET
Morte

Reputation
Posts:
4

Joined:
October 2018
You guys should find someone to translate News do Portuguese, looks like people dont understand whats going on and keep complaining on the Help Channel.


I could do that 


Nice work btw, thanks for the feedback.


October 31st 2018, 15:59 CET
Bullshit

Reputation
Posts:
1

Joined:
October 2018
My Colleague has sent you an email at bounty@mastercores.com with his potentional solution as he can't post on this forum for some reason, please check it out


October 31st 2018, 19:21 CET
Guihenrique

Reputation
Posts:
3

Joined:
October 2018
By Marcus
As many of you probably know, we have some crash issues in MasterCores.

First off, don't worry about your exp/skill progress, the game is backupsaving so frequently that neither your char nor houses will face any crazy rollsbacks.
Second, don't worry about duping items, these crashes is not created by players directly, there's no "group" of people who crash the server to dupe millions of gp or items.
We've been monitoring item activity on daily basis and looking for possible duped items, but so far nothing dangerous has been found since we cleaned the game from the previous bank issue.

Now, to my point about the crashes..
Like a player could have asked: "Why don't you fix these ****** crashes!?!?!".
In the last month, we've had a few different types of crashes happening, and so far, all but one type have been fixed.
The one that remains, is one that seems to be incredible hard to solve, yet we're not giving up!

The crash is a memory crash, it crashes because it tries to delete something that isn't there anymore.
I've made a picture explaining a little what the crash is about.

As you can see, the entire game is made up of memory-slots.
And at some point, one of the slots goes corrupted which crashes the whole thing.

We have logs about crashes, but they don't point towards where the source of the problem comes from.
And we yet don't have any way to reproducing the crashes.
If we had, the crash would be fixed VERY quickly.

These crashes happens frequently in Azera/Demera (very populated worlds), but rarely happens in Benera/Canera.
Benera have had 2 crashes in total, in a month, Canera have had 1 crash in a months period.

Now since how hard this one have become, I've put a $500 bounty to whoever can get closest to a solution (even a critical tip would be enough to collect that bounty, as far it's helpful in finding the problem).
Now that's a lot of money and I've had multiple C++ devs taking a look at it yet nobody has come close to an answer what's causing it and how to prevent it.
And this is where logic plays in, some people think there's people abusing the crashes, but then.. who would not share their info how they crash a game for $500?
It's near impossible to dupe anything, and if it would happen I'd know about it and the player would get deleted if it happened multiple times, or just robbed of the items, if it was a one-time thing.

What happens to you during a crash?
practically nothing, your exp/skills/inventory and even house, will still have whatever it was before the crash.
However, a crash closes the game, meaning the floor is reset, so the only thing players really lose, is lootbags.
Which is why I'd recommend people not to keep valuable items on the floor, unless they're inside a house.
Additionally, if you're online during a crash, you will be sent to your temple for your own protection.
So you don't login to a monster area that's freshly respawned and risk dying due to it.

No matter what, I will never stop looking for a solution for this issue, and I hope you can understand the huge pressure this causes towards the staff.
And I hope you can enjoy MasterCores even thou our issues, and we all hope we find a solution soon!

Sincerely,
Marcus
 
Using a bit of logic, would it be possible to add a wait time for the released cache to be available again?
I do not understand much but when someone performs a re-login (ctrl + g) would happen for example to release 
the slot and for being a very fast action, occur to occupy the same slot before it is even available and lock everything?


October 31st 2018, 19:30 CET
Mrparrot

Reputation
Posts:
6

Joined:
October 2018
Using a bit of logic, would it be possible to add a wait time for the released cache to be available again?
I do not understand much but when someone performs a re-login (ctrl + g) would happen for example to release 
the slot and for being a very fast action, occur to occupy the same slot before it is even available and lock everything?

You are proposing it to be a race condition issue? Sure, It is possible :-) . But a better solution is to add a mutex for each cell, and the player will try to acquire it in login. If yes, then write its information to the cell, the mutex guarantees that no other player has this cell. When a player logs out, he must clean his stuff and as the last thing, unlock the mutex so another player can take it on login.

-----

Back to my previous post, an quick'n'dirty workaround is to only use the even cells. So if something is written on the next player memory space, nothing will happen because it won't be used anyway.

*Flyes Away* 


October 31st 2018, 21:08 CET
Mania

Reputation
Posts:
10

Joined:
October 2018
while we wait for the solution of bugs with the client...

https://mastercores.com/forum/?thread=220

https://mastercores.com/forum/?thread=214

https://mastercores.com/forum/?thread=156

https://mastercores.com/forum/?thread=305



Mania


November 1st 2018, 07:23 CET
Page:
1
This site requriers javascript to be enabeled to run properly!
(Please enable javascript in your browser)