Technical solution - a distributed server

[Glader]

Yups, and it's a damned if you do, damned if you don't problem - consider; in the naive solution for chat, each client must emit to every other client. That won't work. What's the alternative? every client needs to know where every other client is in the game world, so I can only emit to those who can read what is written - but to do that, every client needs to tell every other client where they are in the game world...

Prolly you can combine the two though. Every client knows where IT is in the game world, and so IT can filter out stuff it shouldn't hear. So the emitter can afford to error generously.

Another thought is that you have something like a write combining tree. Each chat emit enters the tree, then fans out along matching nodes - i.e. retransmission occurs, like NTP. It's not just the job of the emitting node to tell everyone else. You get more latency this way of course.

Hmmm, I don't think clients should filter out messages they aren't interested in. You said there are many clients managing a section of each zone. Well, depending on the messaged type (Ex. Say, Yell, Private or Channel) those people managing the selection of zone you're in should handle the chat request. They know who is in their little section so if it's a /say /yell they can send it to other clients in that section or adjacent sections.

Private messages seems more difficult to handle.

Channel specific messages could require players to actively register themselves in the channel. Each node involved in this could keep a ledger of subscriptions to a channel and broadcast the messages to subscribers. But who should do this broadcasting? Since we want many people verifying this and many nodes working together who should to the broadcasting to save bandwidth?

These world-wide problems seem difficult and maybe for them we should just hijack transient services like vent, mumble or discord that can easily move around the world at will to manage messages that reach outside of specific zones sections. Maybe people could host them too. Senders send to each one but maybe only a couple actually send the responses back since the the load of all clients is spread between them all.

I edited the original, after your reply but before I saw it, to say it's *harder*, rather than hard.

It's because symmetric designs are easier to reason about.

I guess but I think any way is hard, at least to me. Nothing about anything proposed seems easy.

*grin* I differentiate between C and C++ THANKYOU VERY MUCH

I am a hardcore C programmer. C++? pah!

C#??! double pah!!

Hmmm, I don't know why people dislike C#. Very portable, feels very productive and quite clean compared to other managed languages I think. Guess neither of us is fully suited to implement these ideas then seeing as neither of like the language these massive emu projects default to lol.

[wowmad]

.

[Winterflaw]

I may be wrong, but I don't think there is a chat server as such - not as a seperate entity. It's part of the general, monolithic, WoW server. Running an *unmodified* server in a distributed mode implies duplicating the input to the server, i.e. having a number of full-sized servers. That seems to be fundamentally the same situation as now, in that a C&D can take them down - however, at the moment of take-down, there would be no loss of service since you'd fall over to one of the others. Expensive, though, to have those other servers running, and of course they too could be targetted by a C&D.

[Winterflaw]

One thought about peer design.

Imagine a peer which is connected over the network to all those peers which are phsyically nearby in the game world. This peer then issues a chat message. An obvious solution is that the peer is connected to every nearby peer. This is a problem for World PvP where we might have hundreds of nearby players. However, if we are happy to allow chat messages to make *two* hops, rather than *one*, then we do not need to connect to all our peers - we only need to connect to a fraction of them, as those we are connected to will relay the message on to the others (who are of course also phsyically close to the same players).

This if I recall is a lot like how the eDonkey/Kad networks propagate search requests.

[Crossbreed]

Hmm, I'm a dev, but not at all in the networking or gaming sphere, so may thoughts may not be worth too much.

I've also been thinking of various ways to overcome this, but another big issue is that there is always going to be at least one weak link in the chain who is going to abuse the system and corrupt the data in some way to try and get themselves better gear than he/she deserves.

Well, one thing at a time.

First, see if the basic idea is possible.

I can offhand imagine for example ensuring any objects owned by a player are never stored on his machine.

One thing at a time - but security/anti cheating is probably one of the very first things that should be thought of. Major parts of a server only exist explicitly because you can't trust what clients are calculating. As you may know, P2P was very common for many network games in the early days (not MMOs of course), but that hasn't really been done anymore in the fast.. decade for a good reason I suppose.

1. How do you validate information? Who "controls the RNG"? All while needing to bullet proof this vs. a most likely not insignificant percentage of players who'd try to cheat the system, and keeping latency down to a minimum.

2. Not everybody ingame is equal in terms of permissions and rights. Without having any central server, how would you assign special privileges to some accounts, such as gamemasters and admins?

3. Where/how can you permanently store state information? Sure, if you want to share some transient information, say the amount damage you deal in a hit, it'll be plenty to notify all clients in some radius around you. But permanent data? You don't have a clue who will be around next time you log in, so in theory, everybody would need to be able to provide that state back to your client then.
I don't have a clue how big the entire DB is, I'd guess several GB (think i've read some larger games such as eve easily reach the terabyte threshold). How do you keep this in sync with thousands of players, many of which will have less than 100kb/s upload bandwitch, and on top of that, without corrupting it with any invalid transactions? How can you deal with rollbacks - both in terms of information needed to rollback, as well as triggering such a process?

[Winterflaw]

One thing at a time - but security/anti cheating is probably one of the very first things that should be thought of.

Yes and no.

You are right, but a decentralized design is complex and ground-breaking enough of its own accord that upping the complexity even further by having to deal with trust and security is just too much, too soon.

Get a viable decentralized design up, then with the experience gained, tackle the trust problem.

Major parts of a server only exist explicitly because you can't trust what clients are calculating.

Have three clients all perform the same operations. If one disagrees, ignore it (and indeed, mark is as suspect). You'd need to control and compromise two or more clients performing a single operation, to break trust.

Perhaps a fully decentralized system in some ways actually has solutions to this problem which are not usually available.

1. How do you validate information? Who "controls the RNG"? All while needing to bullet proof this vs. a most likely not insignificant percentage of players who'd try to cheat the system, and keeping latency down to a minimum.

The RNG is controlled by anyone except the peer performing the operation.

2. Not everybody ingame is equal in terms of permissions and rights. Without having any central server, how would you assign special privileges to some accounts, such as gamemasters and admins?

Have private/public key pairs built into the system. Only admin/GMs have the necessary private key to sign their actions.

3. Where/how can you permanently store state information? Sure, if you want to share some transient information, say the amount damage you deal in a hit, it'll be plenty to notify all clients in some radius around you. But permanent data?

Same way Amazon do S3. Have n copies, so that it's practically 100% certain a copy will be available at all times.

I don't have a clue how big the entire DB is, I'd guess several GB (think i've read some larger games such as eve easily reach the terabyte threshold).

I vaguely remember tens of GB, now you mention it, for a 1k server for a year or so. I suspect the per-day growth rate is actually much lower than you'd think - megabytes maybe. Something which could be downloaded at game start.

How do you keep this in sync with thousands of players, many of which will have less than 100kb/s upload bandwitch

Game startup is slowed, because you need to catch up to current state. Once you've caught up, you're in, and off you go. The entire DB does not need to be on every client - if you have 10k clients, and you have 3x duplication, then each client has 3/10000th of the database.

and on top of that, without corrupting it with any invalid transactions? How can you deal with rollbacks - both in terms of information needed to rollback, as well as triggering such a process?

Rollbacks only require each object to come with and retain its own change history. A prived user issues the rollback command, the game stops, every client rolls back every object to the given timestamp, and then off you go. A nice lesson there in how to make something extremely complex sound simple when it's not

[wowmad]

.

[Winterflaw]

In my p2p proposal I defined "Intermediate Servers" that could call webservices of "Core Servers" (Cluster). The intermediate servers only exist inform the clients where they can connect to and how to send authenticaded data to "Core Servers". They could also have some services running to free the servers to process more than needed. This kind of design will need some hard work to implement, only with time we could think about this.

Any *fixed* server can be taken out by legal action. So ideally the game is hosted purely on the clients (peers) playing the game. A design with a built-in hierarchy implies asymetric node types - is this node a server? which type? how many of them do we have? what services does it offer? how does it scale?, etc, which complicates matters - it's easier to think about the design if every node in the system is identical.

It's easier for sure to imagine a distributed server based on a set of fixed servers, but this is vulnerable to a C&D/legal action.

[theunknowing]

Unfortunately at this point there is no way the guys behind nostalrius are going to do any thing, they have already showed us they aren't willing to do what it takes to keep the server up and running. So we just wait til another legit vanilla server pops up, hopefully one with nostalrius coding, we will see.

[ceoddyn]

The only thing I can think of is to have a set of players with faster connections be the nodes. You're already going to need this because many players will not have enough bandwidth to support handling distributed processing of any kind, even broken down into modules as you describe. They won't contribute even to small modules.

At that point you would only need a few node clients within a certain range to control error checking amongst yourselves. And so, for example, one tryhard guild that is already in the know, and own better computers, could easily come up with a hack to exploit this system. How do you get around the most likely nodes collaborating with each other against the other players?

My plan would be to start with Nost's own server clustering to make it easy to swap out servers and just hop between service providers regularly and preemptively to stay ahead of slow legal systems. You have much more concern about hiding your own identity than you do with hiding the possible short term location of the servers.