Wednesday, February 24, 2010

Choosing A Network Library in C#

I am a bad, bad man for not having blogged about this sooner. Last year I blogged about How I Chose SlimDX as my graphics/sound/etc library of choice. And I've been meaning to blog about how I chose my network library, and what I think of it, yet have failed to do so. This was particularly important to me because (spoiler alert) I really love the networking library I'm currently using and think the guy making it is also a real class act.

Early Alpha
For the first versions of AI War, I was using an earlier version of the Lidgren Network Library. I found this easy to implement, and got the game up and running pretty quick. The Lidgren library is a UDP-based solution, which means that it is a layer over an unreliable, connectionless protocol. That means that for applications that require reliability (like AI War), and for all messages to arrive in order, you need something like the Lidgren library. This library does a really good job of managing this whole process in a transparent way, fortunately, so you don't have to think about most of that sort of thing as a programmer working a layer above it.

There were a few bugs in the Lidgren library at the time, though, that eventually led me to create my own sockets-based code based on what I'd learned from working with Lidgren's code. Mainly I was concerned because the Lidgren library was flagged as beta software by its own creator, and I didn't want to have my game running on critical code that was beta. So I coded my own replacement.

Later Alpha Until A Few Months Post-Release
My sockets-based implementation was surprisingly easy to implement. I used TCP instead of UDP, you see, which meant that all that reliability and orderliness that the Lidgren library was having to create when running on top was built right into the protocol. I mean, after all, isn't that the entire point of TCP as a protocol? There were dire warnings on many game programming sites about not using TCP for these sorts of purposes, so I was wary, but in one afternoon programming I had a working prototype of my event-driven TCP-based code, and it worked perfectly.

In further testing, for months and months with my alpha testers, it continued to work perfectly. I stopped worrying about the TCP issue, clearly it was a non-issue. The game released in May, quietly at first, with most players playing solo at that time. Gradually it picked up an increasing playerbase, and more of them started wanting to play online together. Largely it was fine. I think it was around late June that I started getting scattered reports of really terrible performance in networking, however. Mostly it was people playing from the US to Europe, or between Australia and anywhere else.

My first thought was that these were simply connections that could not handle AI War -- AI War is a big game, and uses a fair bit of bandwidth when a lot of ships are being moved around at once. However, since this issue was a constant thing, even when no significant amount of command data was being passed, I was rather stumped for a month or so. I implemented a "Network skip" feature that adjusted network latency, and this largely let people work around the problem, but it then made the command lag pretty horrific when they used that feature.

I don't remember what or who eventually helped me realize what the problem was (it's been a while), but the key realization was this: TCP networking absolutely stinks in high packet-loss situations, which is what was interfering with these specific connections that were otherwise really fast and capable. In other words: all those articles warning against using TCP were (of course) absolutely right, and the issue had come back to bite me in a major way. All those months of testing were for naught, despite all the various network load conditions we had subjected it to, because we'd never subjected it to a high packet-loss environment.

Drat.

Returning To The Lidgren Network Library
So the very first thing I did was look up the Lidgren library again, since I had liked it so well in the alpha stages of AI War. I figured that it might have had a number of updates in the intervening 6ish months (it had -- substantial ones), and I also figured that if there were errors in it, it would still be less of an issue than packet-loss issue with TCP. Basically, I was running monthly public betas, and I could let everyone beat on the Lidgren library in the game for a solid month, and see what shook out.

Another afternoon spent implementing the very-differently-organized Lidgren library (all changes very much for the better), and it was up and running. I pushed it out to the current beta release, and the affected parties noted that their issues with latency had all disappeared. And there was much rejoicing! All seemed to be well. It was working well for somewhere between a few dozen and a few hundred people (always hard to know, since not everyone who downloads beta versions posts or registers on the forums), and it remained doing so for the remainder of that beta month. That beta then turned into the official version of the game, and the Lidgren library has been an integral part of the AI War library ever since. But my story isn't quite done yet.

Sending Large Messages
Sometime after that first release that made the Lidgren library an official part of the AI War game, I started getting reports from players that the initial connection speed was super slow for them (when they would be transferring between 500kb and 2MB of data for the initial "full sync" of the game), but then that after they were connected, it was fine. And it wasn't just a little bit slow -- it was taking upwards of 30 minutes for a few of them, sometimes.

The strangest thing was that it wasn't all the time, even though it was mostly centered in that same crowd that had been having the issues with my TCP implementation. And weirdest of all, these same folks had had very speedy and nice sending of the initial game state under the TCP solution, and it was just the in-game performance that had then stunk. Groan. There were suggestions that I should use the TCP for file send, and UDP for the game itself, but that would have been a nightmare to code and maintain on a number of levels (dual connections that are really the same connection, but used for different purposes -- yuck).

So after my initial investigation of the issue, I had basically determined that any data that was above a certain size was causing a performance impact in the library, but that breaking that same message up into smaller bits was no problem. I talked to Michael Lidgren, the author of the eponymous library, via email. Basically what it boiled down to, in talking to him, was that based on the way the library was having to manage resending of lost packets, and the way that it was breaking up the data that I was unceremoniously dumping into its buffers, was causing a lot of extra re-sends that were problematic.

This was something he moved to look into (I could have done the same, since his library is open source, and I did poke around in it some -- but despite the fact that his program is extremely well organized and commented and formatted, it is nonetheless quite a complex library simply by nature). It's actually quite possible this was largely a bug in the network drivers on the specific cards, which is what it wound up looking like was most likely in the end. In the meantime, with some suggestions from Michael, I implemented some logic on my end that put less load on the network card by spacing out that content slightly, and the result was perfect.

Until a few months later...

MTU
The concept of MTU was not something I was familiar with, not being formally trained in any way for network programming. So when a few players later started having some disconnections, I was not sure what the issue was, but it seemed network-card related. The Lidgren library had been working fine for months for an ever-wider group of players, after all, so I doubted it was a problem there.

As fortune would have it, the player with this problem happened to be extraordinarily knowledgeable about networking in general, and with the help of the debug data that is built into the Lidgren library (which I make available through a key combo in AI War), he was able to figure out that this was an MTU problem. This was not a bug at all, per se, but the default value in the Lidgren library at that time was high enough that it would cause problems with a super tiny minority of players and network card/router combos.

Fortunately, I was able to set this via a simple property in AI War, to override the Lidgren library defaults, and the user's problem vanished. Voila! I had contacted Michal Lidgren again when this specific problem came up, since I'd been through a ton of debugging steps with the player and had videos of what the network was doing, and Michael had had some more good suggestions that helped lead to the discovery of the MTU issue. So I contacted him again to let him know what the player had figured out and what I had implemented as an override in my program, and suggested that he consider lowering the default MTU in his library. He agreed, and that was another problem solved.

Conclusion
It's been around 5 months now, during which AI War has had it's playerbase grow 400%, with no more issues since those two. Additionally, those two were really unusual and outside the bounds of even what most games would require in terms of load spikes (AI War is quite hefty on the network when it comes to moving lots of units or doing a full sync), and really they were less issues with the Lidgren library, and just unfortunate confluences with hardware that was nonstandard in some way, we basically determined in the end.

In other words, the Lidgren network library has been rock solid since I implemented it into a post-release beta version of the game, with the only two "issues" relating to it not really being issues with it directly, but with network hardware under it. The reason that I related those issues at all was because I think it illustrates the character and devotion of Michael Lidgren, as he helped to solve both of them despite suspecting that they were not really problems with his library from the get-go.

The last thing I want is for this post to result in a slew of simple help requests to Michael, so don't take this to mean he's going to help you set up networking or learn the basics of what is going on with how to use his library. I made sure not to bother him until I had collected a ton of data that I thought he was the only one who could reasonably interpret, about really specific complex issues that I'd already spent significant time trying to debug on my own. I'm pointing out his devotion because for any team that is worried they might implement the Lidgren library and then just be stuck if they discover some sort of rare but fatal bug (of which I have seen none, anyway), they can rest easy that the creator of the library is very much on the ball and is interested in making sure his library is rock solid. He's not a tutor or a guide, and I wouldn't expect him to give out a ton of free support (he's giving this out for free, come on), but it's also not one of those headless, abandoned projects that no one cares about or updates anymore.

For any team working in C# and looking for a UDP networking library, I can't recommend the Lidgren one enough. It's been rock solid for a demanding game like AI War, the creator stands behind the library, and it's small and efficient -- and open source, if you need to make changes of your own. Given all of the various sample projects he includes with it, you can probably get something up and running in an afternoon, or at least a day or two, as well. This is precisely why I've felt so bad for not making this blog post before now, because there are few (free, nonetheless) products which I can recommend so wholeheartedly and unequivocally.

Fanboyism And The Cycle Of Perceived Inferiority

I'm not known for being brief, but I'll try to keep it as short as possible for this post. Fact is, I have a ton of work I should be doing on Tidalis, but I was struck by inspiration today after watching the various shows of the Penny Arcade TV Show, which I hadn't managed to see until recently.

Some Of My Own Fanboyism
I love me some Penny Arcade, I've got all their books and have read most of their archives more than once, and even have three or four of their shirts, but for some reason I'd been putting off their TV show as being something I wouldn't like. Instead, when I finally got around to watching it, I discovered an insightful look into the work life of a company full of interesting people. It's rather like what some of the later anthologies of Bill Watterson and Gary Larson provided: insight into creative talents that fascinate me. Or the same reason that I enjoyed the two Michael J. Fox memoirs so much.

Sounds like I'm a real voyeur when you put it all together like that, right? I'm not, or at least I don't think I am. I'm not into celebrity gossip or whatever else. My own curiosity is mostly limited to a few creative individuals over a span of decades, and I'm mostly interested in their work, how they accomplish it, how their work affected them, and their outlook on life based on doing some sort of job that has impacted me in some way.

We're All Just People
This is a fact most people know, but few people seem to really believe. We're all just people, every last one of us on this planet. President Obama is just a guy who gets up in the morning and does a very difficult, very important job. Shigeru Miyamoto is just a guy who makes video games, and has been doing so for a long time. John Williams is just a guy who writes film scores. And, frankly, Einstein was just a guy that came up with lot of important ideas, worked really hard, and turned those ideas to a few practical applications.

This is not an attempt to minimize any of those people. On the contrary, I hold them all in the highest esteem. They are visionaries who do exceptional work, and have made notable contributions to the world in their lifetimes. What they are not, is mythical. These are just folks that have a lot of attention focused on them, for various reasons -- they are not, well, unicorns.

This is something that everybody knows, but it can be so hard to believe. I think it really sank in for me when I was in high school and had the chance to speak to Orson Scott Card for about 10 minutes at a library fund raiser. Lately I've come to realize that he holds a number of views that I very much disagree with, but that doesn't detract from the fact that he wrote what I consider the two very best books, ever. And yet while I was standing there talking to him, I was struck by just how ordinary he was; he had written these amazing books, but he was still just a guy, and once we were done talking he got in his car and made the drive back to Greensboro to be with his wife and kids.

Fanboyism Is Nonlinear And Relative
What I fully realized only today, while watching the PA TV show, is that Mike Krahulik and Jerry Holkins feel inferior. They were amazed that the guys at Naughty Dog thought it was cool to be in a Penny Arcade (PA) strip. And Mike, for years, has commented in the "braying and neighing" section of their news post on how he feels his art is inferior and derivative to the comic book artists that he particularly admires.

But it gets better: thinking back to Bill Watterson, I can remember that he definitely seemed to have at least a bit of an inferiority complex regarding much older comic strips (Peanuts and Pogo, mainly), and Gary Larson was in awe of the scientists that he was depicting in many of his strips. When he did a strip about Jane Goodall, there was a minor misunderstanding with some of her staff, and to make a long story short Larson and his wife went out to spend a bit of time with Goodall at her preserve. I think that each was a bit intrigued with the other, reading between the lines of Larson's account of the events.

The oddest thing is how this cycle of fanboyism is not linear. Goodall and Larson can be impressed with one another. The PA guys can be impressed with staff from a game developer, while I'm impressed with PA and yet likely to casually know someone from a game developer. I don't happen to know anyone from Naughty Dog, but I know enough people from companies like them that I wouldn't be at all starstruck. Seeing Gabe Newell's name on Arcen's Steam contract was a bit of a thrill for me, but come on... it's Gabe Newell!

See what I mean? This whole fanboyism thing, and why we humans tend to feel this way, is very mysterious and unpredictable. But I'm not done yet (predictably, this post has turned longer than I meant it to).

"Fame" Is In The Eye Of The Beholder
Well, first off, the obvious examples: Gabe Newell, or Mike and Jerry from Penny Arcade. I'd be a bit shaky talking to any of these folks, at least at first, as would maybe a few million other people. Generously, let's say 50 million. And then to everyone else out of the world's 6 billion inhabitants, to meet these guys would be utterly ordinary, possibly even boring.

Kind of puts things in perspective a bit. Let's take a more famous example, like for instance Tom Hanks or Brad Pitt. Either one of them probably has at least (I imagine) a billion or two people who would be nervous upon meeting them. The reaction of the rest of the billions of people on the planet would be: nothing. Tom Hanks, Brad Pitt... they're just guys. Maybe unusually funny, maybe unusually good-looking, but still just guys. There are funny and good-looking guys all over the place, though. That's less notable than people might assume.

Few People Seem To Feel Famous
Looking back at Mike and Jerry from PA again, Jerry made a comment that I found pretty fascinating: that he's not really famous, he's just "Internet famous." In other words, he doesn't feel famous at all, and the fact that a lot of people know who he is (and want autographs, come PAX), doesn't really change his perception of himself. He knows he's well-known, but Mike mentioned that both of them felt strange signing autographs until well into their careers.

In the Michael J. Fox memoirs, Fox talks about how his own super-stardom didn't really feel real for years. He was on the cover of multiple magazines in every US newsstand for a few years there, but he still felt odd when meeting other celebrities and expected them not to know he was. Of course they did. He was tremendously famous at that point. But that's still how he felt, and I don't think that he was unusually insecure.

Most Famous People Are Less Famous Than You Think
People overestimate the fame of everyone. Brad Pitt and Tom Hanks? I'm sure there are plenty of people who imagine those two are known to every last person on the planet. And yet there are plenty of people who don't watch TV or movies, or don't watch American TV or movies. Even with the most-famous examples I can think of, we're overestimating.

Apparently we do that. When I was in 8th grade, there was a new comic strip that came out in the local paper, called I Need Help. Do you remember it? I thought it was really funny, and sent the creator, Vic Lee, a snail-mail letter. That was one of the few times I ever sent fan mail, and apparently it made a huge impression. I don't think Vic had gotten much yet, and so he sent not only a lengthy response that was as witty and funny as it was informative, but also a signed copy of his first book (three weeks before it hit stores), despite the fact that I'd asked for no such thing. He also included his AOL handle, and we wound up chatting via instant messenger about a variety of things (that wasn't a taboo thing for a grown man to do back then, stop frowning -- he was just connecting with a fan that was excited about his work).

Anyway, the point is that I'd wrongly assumed that Vic Lee was as famous as Bill Watterson, who was the only other author I'd ever sent fan mail to (I got a note back from Watterson's publisher saying he doesn't accept fan mail). Anyway, if Watterson doesn't feel famous (and I'm pretty sure he doesn't), Vic Lee really didn't feel like he was famous. Yet he had a nationally syndicated cartoon that a lot of people felt was a promising new strip. He had a book that was in stores. What else was needed to make him famous?

Fanboyism Is A Dangerous Thing
Fanboyism is dangerous because it leads people to have unreasonable expectations. When I sent Bill Watterson a letter, I expected a response. It was a punch in the gut what actually happened, but it also wasn't Watterson's fault. A lot more people wanted to send him mail than he could reasonably reply to, and so he just didn't accept any mail whatsoever. I think that's why a lot of game developers also hide behind PR departments and private email addresses. It isn't that they don't want to talk to fans: it's that they need a buffer from the raging fanboyism (or fanboyism's inverse) that is pointed at them at any given time.

In the end it doesn't matter what anyone says, or what we intellectually understand. I still want to hear back from Jerry Holkins at some point. I'd like to know what he thinks about AI War, which he might love or he might be indifferent to. I'm mildly jealous of both Cliff Harris and Vic Davis, because "Tycho" played their games and raved about how good they are. It's easy to fall into the trap of thinking "why not AI War?" Thankfully, I have a daily reminder of why I shouldn't: there are enough folks who get mildly sulky and resentful when I don't have time to respond in detail to their communiques.

I have it easy at Arcen at the moment compared to what someone famous has to deal with, but it's already at the point where I could spend all day, every day, just responding to posts on the forums and still never catch up. My hope is that, perhaps, anyone who is starting to turn resentful can be referred to this post for an explanation. I still respond to about 50+ messages per day, but that no longer covers anywhere near responding to everyone. With luck, as Arcen grows I won't piss too many people off, and I'll keep my sanity, too. With luck.

Lastly: I Am Not Famous
This heading is a pretty obvious statement. I don't think I'm famous. Most people that I meet don't think I'm famous. In fact, I'm a bit embarrassed to tell strangers that I'm a game developer because they've never heard of my work.

And yet there are still people out there who, when sending me fan mail, clearly think I'm famous. It's all over the way they write, the exact same patterns of how I wrote to Vic Lee. It took me a while to realize what was going on, honestly, and so I have always simply responded like I would to anyone who gives me a compliment: saying thanks, making whatever comments are appropriate, and so on.

Fortunately, that response seems pretty appropriate, so I'm sticking with it. Better than getting a swelled head. Generously, maybe fifty thousand people think I'm some sort of super mild case of famous. Maybe half a million people have even heard of my games, and most of them still have no idea who I personally am. Most people who meet me from the forums or this blog seem to accept that I'm a regular guy, for which I am grateful.

This whole phenomenon is interesting to me, though. As a kid I thought that there was a very clear line differentiating the mythical famous people from the rest of us. I don't know when everything got so murky. But evidence suggests that's how it's always been.