Technical solution - a distributed server

[wowmad]

.

[Crossbreed]

Have three clients all perform the same operations. If one disagrees, ignore it (and indeed, mark is as suspect). You'd need to control and compromise two or more clients performing a single operation, to break trust.

[...]

The RNG is controlled by anyone except the peer performing the operation.

How do you protect the system from some client providing skewed random numbers, for whatever intention?

And how do you pick the clients performing the checks and e.g. RNG? Obviously the client requiring this input shouldn't chose the providers, otherwise you can just have two clients collaborate on the whole cheating thing. So you need some neutral authority to assign these - which probably comes back to a central master server though?

2. Not everybody ingame is equal in terms of permissions and rights. Without having any central server, how would you assign special privileges to some accounts, such as gamemasters and admins?

Have private/public key pairs built into the system. Only admin/GMs have the necessary private key to sign their actions.

That one key better don't leak/get hacked then, otherwise the entire game would be ruined forever
(ps, maybe that issue could be solved by assigning privileges on a "democratic basis", e.g. sending messages to all clients and only accept if 50%+ agree or so)

I don't have a clue how big the entire DB is, I'd guess several GB (think i've read some larger games such as eve easily reach the terabyte threshold).

I vaguely remember tens of GB, now you mention it, for a 1k server for a year or so. I suspect the per-day growth rate is actually much lower than you'd think - megabytes maybe. Something which could be downloaded at game start.

If the incremental rate is not huge, maybe the disk size problem would be solvable

How do you keep this in sync with thousands of players, many of which will have less than 100kb/s upload bandwitch

Game startup is slowed, because you need to catch up to current state. Once you've caught up, you're in, and off you go. The entire DB does not need to be on every client - if you have 10k clients, and you have 3x duplication, then each client has 3/10000th of the database.

In that case, you'd have 10,000 database fragments, and if any of these are not available - i.e. if just for a single one out of these 10,000 groups if happens that none of the 3 players holding a particular fragment are online - you'd have a broken database, which would statistically probably happen.. pretty much all the time.
But generally, yes agree, you could maybe cut it down a little further that way (though I'd expect realistic numbers to be closer to 50% than to 0.03% or so).

But even if the pure size of the DB wasn't an issue anymore, the number of transactions might well be. Again for Eve online, I've read there are ca.
1500 DB transactions per second. Sure, they probably have a lot more players etc., but even scaled down to the number of players a WoW server would have, and shrinked down by a certain factor if you assume a client will only handle a part of the entire DB, that might well be a massive bandwidth and even raw hardware power for most clients.

[r00ty]

I may be wrong, but I don't think there is a chat server as such - not as a seperate entity. It's part of the general, monolithic, WoW server. Running an *unmodified* server in a distributed mode implies duplicating the input to the server, i.e. having a number of full-sized servers. That seems to be fundamentally the same situation as now, in that a C&D can take them down - however, at the moment of take-down, there would be no loss of service since you'd fall over to one of the others. Expensive, though, to have those other servers running, and of course they too could be targetted by a C&D.

On retail as far back as BC (and probably in vanilla, in fact it must be the case it was for their multi node demo to work on vanilla) there was a system of chat/guild/friends/groups etc on one server. Then, continents could (and often were) separated to different servers, and instances were pooled (and most likely shared between realms in the same battlegroup or server farm, there were 3 or 4 farms in EU as I remember).

All of the current stock server cores don't support it. They have one auth server, one worldserver and the worldserver does the job of all worldservers on retail, the instance servers and the chat/guild/etc server.

There have been projects to split them out. But honestly for the average server it's just not required. It only becomes an issue at this kind of size.

Now, as for the idea of a distributed server. The idea sounds nice and all. But, with the standard client you are bound by the wow protocol. In that while you can separate so much, a lot just won't be separated because of the way the protocol works. The only real way round it would be to write your own client. In which case you may as well make the next "big MMO" while you're there.

The idea floated by the admins is probably the best one, if they're able to go through with it.

[Crossbreed]

Oh and for the record, its an interesting discussion and I would love to see an implementation of such a P2P principle. But as usual, planning is crucial, and my gut feeling at the moment is this wouldn't be manageable for a bigger MMO server, if at all.

I think a more promising way would be checking in detail was copyright laws Nostalrius supposedly broke. There would be little point working on the almost herculean task of P2Ping an entire server's functions when in reality, only 1% of its functions may be critical from a copyright point of view.

For example, CMaNGOS source code has always been publicly available on Github - a website whose owner is based in the USA. If that infringed copyright laws in any way, Blizzard could probably file some DMCA complaint and have it taken down within 5 minutes; that would permanently rid them of any and all private servers now and in the future.
Which makes me think, there's actually nothing illegal about this (and why would there be, it doesn't use any of WoW's assets - if anything, only some of its networking protocols).

Then there are content databases for CMaNGOS. Personally, I would have assumed these are indeed critical from a copyright point of view, as they contain data which I think would be copyrighted (NPC names etc. etc.) - but even these are actually still up and running on Github.

Again, not a lawyer, but that looks to me like the majority of a WoW server's functions are fine with copyright. So you might just need to identify the parts that are questionable, and check if these could be outsourced to the client network.

[ceoddyn]

2. Not everybody ingame is equal in terms of permissions and rights. Without having any central server, how would you assign special privileges to some accounts, such as gamemasters and admins?

Have private/public key pairs built into the system. Only admin/GMs have the necessary private key to sign their actions.

That one key better don't leak/get hacked then, otherwise the entire game would be ruined forever
(ps, maybe that issue could be solved by assigning privileges on a "democratic basis", e.g. sending messages to all clients and only accept if 50%+ agree or so)

This is a pretty good idea though. It doesn't seem like it would take much work for all of the clients to agree on the next one to make a key. A majority of clients would need to be cheating together to come up with their own.

The question there would be how to trust the signing of admin/GM clients.

I guess that client could also just wait until it was selected to make the next key and then cheat. There's no real reason to trust an admin/GM team unless they are some how proven to be behind development and 'invested' in the game not being corrupted. This would also lead to their identification as owners of the server. Ideally development should be distributed across a much more well-informed player base who are able to contribute in some part and also would understand if the code itself had become altered maliciously.

[wowmad]

.

[Winterflaw]

The only thing I can think of is to have a set of players with faster connections be the nodes. You're already going to need this because many players will not have enough bandwidth to support handling distributed processing of any kind, even broken down into modules as you describe. They won't contribute even to small modules.

I may be wrong, but I think the bandwdith used by WoW for a single player is very small - it's on the order of a few kiloytes per second. Everyone or almost everyone is connected over cable or ADSL or something similar - they will have probably at least 10x as much bandwidth as is needed for their own connection.

At that point you would only need a few node clients within a certain range to control error checking amongst yourselves. And so, for example, one tryhard guild that is already in the know, and own better computers, could easily come up with a hack to exploit this system. How do you get around the most likely nodes collaborating with each other against the other players?

Well, the scenario described is one I think won't happen. However, if it did, you would have a number of additional nodes in the system, geographically separated (geo IP lookup) which cross-check by performing calculations made by other nodes, so check they come up with the same answer. You can also perform ongoing statistical analysis of PRNG output from any given node. If the output significantly deviates from the expected mean of 0.5, you stop trusting that node.

My plan would be to start with Nost's own server clustering to make it easy to swap out servers and just hop between service providers regularly and preemptively to stay ahead of slow legal systems. You have much more concern about hiding your own identity than you do with hiding the possible short term location of the servers.

In the short term, given how long it takes to write software, and how speculative a distributed server is, it seems the only way.

[Winterflaw]

The RNG is controlled by anyone except the peer performing the operation.

How do you protect the system from some client providing skewed random numbers, for whatever intention?

Maintain a mean of the PRNG output from any given node. Do this on other nodes. If it deviates, stop trusting the node.

Of course, if the node(s) performing the mean calculation are compromised, then you're in trouble. In general, this is like ECC; you have a certain tolerance for error - you can detect it, given the system working properly to a sufficient extent.

We could imagine the entire system is compromised, every node is lying, but that's unrealistic - we do assume that by and large the system is truthful.

However, one interesting point is that nodes which are lying will have to lie in exactly the same way to present the same result, where-as nodes which are telling the truth will all naturally reach the same answer.

Forms of cross-check which validate the actions of other nodes by re-computing their actions and checking the result will require the majority of the nodes to be compromised, *and compromised in the same way*, i.e. probably by the same agent. This may be a significant challange.

And how do you pick the clients performing the checks and e.g. RNG?

Randomly. Or GeoIP and make sure they're a long way from the peer they check.

Obviously the client requiring this input shouldn't chose the providers, otherwise you can just have two clients collaborate on the whole cheating thing. So you need some neutral authority to assign these - which probably comes back to a central master server though?

One way is that the peer issues a request (just like someone issuing a chat message in game) and it propogrates through the network. Each peer receiving the message has a fractional chance of responding.

That one key better don't leak/get hacked then, otherwise the entire game would be ruined forever
(ps, maybe that issue could be solved by assigning privileges on a "democratic basis", e.g. sending messages to all clients and only accept if 50%+ agree or so)

This is true of any system with an admin user. It's true of your PC it's also true of the current system, where the admins have a system with an account which allows them special powers.

Game startup is slowed, because you need to catch up to current state. Once you've caught up, you're in, and off you go. The entire DB does not need to be on every client - if you have 10k clients, and you have 3x duplication, then each client has 3/10000th of the database.

In that case, you'd have 10,000 database fragments, and if any of these are not available - i.e. if just for a single one out of these 10,000 groups if happens that none of the 3 players holding a particular fragment are online - you'd have a broken database, which would statistically probably happen.. pretty much all the time.

So you have 5 copies, or 10, or 20. It's like key length. If i have a key length of 24 bits, I can solve it in a day on my laptop. If I have a key length of 32, it's a week. If I have a key length of 512, well, in theory I could find the answer at any second but in practise I'll die first. Increasing the number of copies gets up rapidly into "players will die first of old age before this happens".

Moreover, a missing object does not break the database, because the database is written to tolerate this. By tolerate I mean that the database does not break, or fail; the game of course with regard to whatever happens with that object cannot continue - so the object will for example, temporarily disappear from a users inventory, or a mob will vanish. But if we have 10,000 players, and a sufficient number of copies, then we might be looking at this happening say one per century. I suspect more mobs vanish due to OTHER bugs than this...

But generally, yes agree, you could maybe cut it down a little further that way (though I'd expect realistic numbers to be closer to 50% than to 0.03% or so).

If 50% of 10,000 players have the whole database, then it's the same as 5,000 having 100% - i.e. you're arguing half of all players need a full copy of the database for the system to consistently be able to reach all objects. This is crazy. If you have say 1,000 players, and each has 100th of the database, you already have 10x over-duplication. If they each held 50% of the database, you'd have 500x over duplication. Amazon S3 runs on *3x*.

But even if the pure size of the DB wasn't an issue anymore, the number of transactions might well be. Again for Eve online, I've read there are ca.
1500 DB transactions per second.

That issue is solved by the large number of nodes in the system. High transaction counts are only a problem for a centralized database. If a typical peer has say a tiny fraction of the database, he also only has a tiny fraction of the transactions.

[Winterflaw]

Now, as for the idea of a distributed server. The idea sounds nice and all. But, with the standard client you are bound by the wow protocol. In that while you can separate so much, a lot just won't be separated because of the way the protocol works.

I have absolutely no knowledge of the WoW protocol. However, if I run a local proxy, the WoW client issues requests to that (it thinks that's the server), then the proxy can issue whatever it likes, so long as it replies in the protocol the WoW client understands. That seems to be generic enough that the protocol in use just didn't matter.

The idea floated by the admins is probably the best one, if they're able to go through with it.

What's this idea? I've not heard anything but I've been hoping for something!

[Winterflaw]

Oh and for the record, its an interesting discussion and I would love to see an implementation of such a P2P principle. But as usual, planning is crucial, and my gut feeling at the moment is this wouldn't be manageable for a bigger MMO server, if at all.

I think a more promising way would be checking in detail was copyright laws Nostalrius supposedly broke. There would be little point working on the almost herculean task of P2Ping an entire server's functions when in reality, only 1% of its functions may be critical from a copyright point of view.

The P2P approach is not a solution to the immediate problem, because it will take too long to implement, even if it is possible.