Category: Debugging

We’ve been using Manuscript (originally Fogbugz) to manage the project, starting during the first week of development in August 2014. Since then, the team and playtesters have created 3412 cases. This breaks down into Manuscript’s categories as:

123 Inquiries. This includes playtester feedback and comments, and 75 completed games. I try to do some analysis of each game, to learn where players get stuck and to make sure things are tuned OK. Once in a while this uncovers bugs.

485 Features and 61 Schedule Items. These represent tasks like like “Map Creation,” “iPhone X support” or “Sweep to be sure ChooseLeader is followed by a leader test.” The two categories are pretty similar, but a Feature would probably be passed to QA to check, and a Schedule Item could usually just be marked completed.

2714 Bugs. These are things that didn’t behave as expected. They’re typically fixed, then verified by QA as working correctly. Since we added playtesters over time, we tended not to get a lot of duplicate bugs (though it’s never a problem if we do, since a different report may give insight into reproducibility, and they’re easy to verify).

Overall, we closed 3313 of the 3412 cases. Of the 99 cases not closed, 26 are feedback that I was keeping handy (and probably should close to clean up the project). Most of the rest are issues that we deferred as part of the triage process, (see part 1) or as features that would be nice to have in an update.

2695 of 2714 bugs have been closed. A few are still being verified as fixed, or were deferred for an update. (That’s about 1.8 bugs per day over the entire project.)

I don’t really like managing purely by numbers (the way a really large project might have to), but it’s good to see that our gut feeling that the game is solid is also backed by data.

(part 1 of the series on bugs is here)

I don’t know if this is a dark secret of game developers, but games ship with bugs. Known bugs.

Part of the late development process is reviewing the bug list to see what most needs fixing. We all hate bugs, but we also love shipping. Not everything has to be fixed before release. The closer we get to making the release build, the stricter the criteria become. We’ve recently deferred a bunch of bugs that we would have fixed when we started the triage process.

Why be so strict? Because many fixes can have unintended consequences. We just discovered a fairly rare bug yesterday, which had been introduced by a change we made last year (which was tested back then and found to work perfectly as designed). Since even a simple bug fix can destabilize the game, we want to make sure any last minute fixes are truly warranted.

At the end of the process, the only bugs to be fixed are those that affect all players, prevent play, or make us look stupid. Most of what we’ve been finding lately have been issues that come up if you have tons of feuds, or when you made certain decisions and then ally a particular clan. These aren’t going to be that common. And even the players who end up doing that won’t be blocked from the game, or find it illogical. My classic “look stupid” bug was a game I saw years ago, where the publisher spelled their own name wrong on the title screen. Luckily we haven’t found anything like that.

But of course, we do want to fix all the bugs. So I’ve been making the fixes in an alternate branch of the code. As soon as everything is locked down in the App Store, we’ll switch over internally and start testing that.

Part 2 of “Bugs”

Just got a bug report from QA that a scene threw up an error. It was pretty easy to see that it couldn’t find a friendly clan to visit you.

That turned out to have happened because of using the debug tools to have everyone feuding (to make it easy to test resolving feuds). This is an unlikely situation, but it’s certainly possible to get a lot of clans to hate you, if you play the right (wrong?) way. So it’s a valid bug, and I fixed it.

Given how many scenes are in the game, I figure any bug is probably somewhere else as well. So figured I should go back to our automated tests and run them in their nonstandard configurations. I don’t usually do this, partly because they’re mutually exclusive, but also because some of them can take a long time (in fact, as I type this I’m running one … that just finished in just under 26 minutes).

But sure enough, enabling TEST_ALL_FEUDING found several scenes that would have trouble. It also gave a lot of false positives, since the brute force testing ignores scene conditions. (Scenes almost always have conditions that prevent them from being randomly chosen when they’re not appropriate.)

So today I’ve been dealing with that, and also TEST_NO_CHIEF, TEST_NO_GOODS, TEST_NO_RING, TEST_NO_WARRIORS, TEST_ALL_HATE, and TEST_Q_BRANCHES.

Right now, the game is undergoing internal testing while being finished. Most of the bugs are easy to deal with: fix a typo, clarify something that’s unclear, fix a logic error, or make sure advice and recommendation match.

One that came in today had to deal with being unable to proceed after having decided to give gifts, when your clan had no goods to give. The specific script was

[ChooseYesNo("Do you take a gift?")]
yes: {
    w = ChooseGoods(“What do you give to them?”)
    TransferGoods(otherClan, w)
}

Normally players who have no wealth wouldn’t be choosing to gift, and it wouldn’t be a problem. But of course part of QA’s job is to test the boundaries.

It would be easy enough to fix this scene, but this has the potential to be a wider problem. ChooseGoods is used fairly often. Any of those situations had the potential of failing, too. One answer to that is to do a code sweep, searching for all occurrences, and making sure they are conditioned. But that’s a manual step, which means it can be prone to error (especially if there are lots of places).

Another is to have the computer do this. We already have a unit test that exhaustively runs scenes. So I set goods to 0 in this test, and found six other problem scenes.

So human QA gives the best results, but automation can give decent testing over the entire game, and it’s easier to set up certain conditions (usually a clan doesn’t stay at 0 goods long, since crafters and traders are continually making more).

Previously, we saw that the game keeps a pair of debug logs on the player’s device. Back in the late 1990s when were did this for King of Dragon Pass, we asked players to navigate to those files (with Finder or Windows Explorer), and send them.

When I adapted King of Dragon Pass for iOS, I had to use a different approach (since the files aren’t accessible). Luckily, iOS had an easy way to send an email message, and we were using Fogbugz to do bug tracking. So the game let the player mail in a report (to the special Fogbugz address), automatically attaching the log, as well as the most recent saved game.

In Six Ages, I’ve improved that slightly, compressing the files into a ZIP archive, and including a few other debug files that help with tuning.

I’m also detecting certain kinds of crashes, and offering to email the report. (In this case, I take a screen shot and include that in the ZIP file.)

Since some crashes prevent this, I also try to detect a crash on the next launch, and offer to send the report then.

This is all fairly trivial code (I’m not using one of the third party libraries that no doubt do a much better job with edge cases), but it makes sending the debug info much simpler for the player. Which makes fixing the bug much easier for me.