Log in

28 April 2009 @ 02:04 pm
Open Letter to the World Puzzle Federation Regarding Sudoku Championships  
As a solver at all four WSCs and as a past champion of the event, I am writing this letter to address recent problems with the World Sudoku Championships and offer some recommendations on changes that should be instituted into the Rules of Competition for all future hosts.

Puzzle Selection and Classic Sudoku:
Following WSC1 and 2, a letter by Simon Prett appeared in the 12th Newsletter of the WPF. Its point can be summarized as this: “By not including classic sudoku, the championships have failed to capitalize on the international craze”. While I found parts of the article insulting or at least ill-informed, such as the arguments that a top sudoku solver such as myself or Jana Tylova also being good puzzle solvers was “proof” of the failure or that the championships in Lucca did not include classics when they had several, the board should consider what are appropriate sudoku for these championships.

The response by organizers to the concerns stated by Prett and others was first, at Goa, to emphasize classic sudoku by running a separate classic tournament for just the regular puzzle and now, to run rounds like the “Guinness” round to show off the skills of the room. However, an analysis of the puzzles that have determined the winner of the Goa Classic Playoff (puzzle 4), or of the World Record Puzzle in Zilina amongst many others, show that these are not puzzles that can be solved by “human-identifiable” logical means aside from bifurcation (aka "guessing"), at least within the time limits placed upon them by competition. These puzzles would never be printed in any newspaper for their solvers to do, at least if they intended to keep their audience.

Therefore, even by emphasizing classic puzzles, organizers have not approached the process correctly. In part, this may result from the ease of computer-generation of tough classic puzzles. I for one look forward to the variants over classics specifically because I know many of them had to be hand-constructed, and therefore have some logical path to an answer, where the classics might be impossibly difficult without “guessing”. At the last two WSCs, there have been reasonable classics and unreasonable classics, and the number of unreasonable classics has been increasing. This trend leads me to wanting to “retire” from competition as having to guess on a classic sudoku at least once on a puzzle dozens of times in a competition, after having logically solved tens of thousands of sudoku without guessing in my lifetime, does not feel correct to me.

Online tools, such as Scanraid Sudoku (http://www.scanraid.com/sudoku.htm), provide an easy means to canonicalize the types of puzzles and types of required logical steps that should appear. This site includes approximately all the logical techniques known to lead to placements in a sudoku puzzle. They are listed in approximate order of difficulty. All solvers use techniques in the first 7 steps (singles, naked/hidden pairs/triples, pointing pairs, etc.). Tough strategies like X-Wing, Simple Colouring, Y-Wing, and Sword-Fish may be used, but typically by a Y-Wing or Swordfish the “fastest” solution of a puzzle is bifurcation. Techniques farther down on the list from Sword-Fish are far too difficult to spot within the time constraints of a round and should not be encouraged as the recommended steps in a puzzle. The Guinness Record puzzle, for example, requires a use of step 28-Finned Sword-Fish, to place the second number in the grid! Several more difficult steps were required to take that puzzle to its conclusion. There is no way any solver in a ten minute round used that Finned Sword-Fish to place the second digit, let alone solve the puzzle, but considering all the logical deductions known to the entire sudoku community, this is the simplest step to make any logical deduction.

The board should set standards for classic sudoku based on the set of known strategies and the easy ability to check the method a solver might encounter using a tool like Scanraid. It should helpfully develop standards for ranking puzzles that are appropriate for competition that use human-recognizable strategies within reasonable times, and allow for fast solvers, instead of lucky guessing, to lead to solutions. This is good for everyone from the competitors to the newspapers who would want to report on these championships as well. I cannot tell you how many people approached me during the Zilina championship asking if I knew how to "solve" a particular classic puzzle. These are accomplished solvers too, and the puzzle selection has completely failed to get consistently good competition-level classic sudoku. Solvers do not enjoy coming to a sudoku championship and being presented with "classics" that cannot be solved like the ones they have done in newspapers and books.

On the topic of variations, these should occur at the World Sudoku Championship, but alongside reasonable classic sudoku. Many variations are actually relatively popular and known. The standards should be to have puzzles that are sudoku – to me this is a symbol-based cell-packing puzzle in which each cell is constrained in at least three ways – and not puzzles that are very far from the set of standard techniques used to solve classic sudoku. While I’ve written a book of Battleship Sudoku, that combines the puzzles Battleship and Sudoku, we can agree that this kind of puzzle – which rewards experience outside of sudoku – is not appropriate at a WSC. However, there are a whole range of variations that are appropriate. These include variations in grid size (6x6, 12x12, …), grid or cell shape (irregular/jigsaw, parquet, 3D/cube sudoku, …), additional constraints (diagonal, extra regions, …), properties of numbers (Consecutive, Kropki, Greater/Less Than, …), arithmetic (Killer sudoku, Arrow Sudoku, ...), and others I have missed. Having a balance across these variations (to not make all the puzzles "math sudoku" for example) is best. Innovations in design at the WSC, such as manipulative or mechanical sudoku, are the most memorable puzzles that occur and show off the talents of the puzzle designers. Variations should not be excluded from the WSC, as Simon Prett argued, but the proportion of variants should be reasonable. Recent championships, with ~50% variations, are at a reasonable precentage in my mind, particularly when many variations like "Diagonal" sudoku are not very far from the classic puzzles that serve as inspiration.

Rules of Competition:
While organizers may believe some “special” types of sudoku should be revealed during a competition or during a playoff, the WPF board should not allow an instruction booklet to be published without a formal presentation of the rules for the whole competition including the playoffs and finals.

To demonstrate what should have happened in Zilina, the rules should have included this kind of paragraph:
“Following the semifinal, a final will be held for the top 8 finishers from the semifinal. The final will consist of two puzzles, a classic 9x9 sudoku and a 'secret variant' that will be revealed beforehand. The fastest solver over both puzzles will become champion.”

Now, as a likely finalist, I would like to know what the “secret variant” in the above description is, which may be the surprise the organizers want to keep until the finals, but I certainly need and deserve to know the standards of competition as should all others. There should be no question how a champion will be determined. This problem is not unique to the recent event in Zilina but also occurred at WSC Goa where the means of winning including the number and type of puzzles, or the value of a time bonus, was not revealed until the day of the final, briefly before the actual competition. The best example may actually have been the early standard in Lucca where all 7 puzzles in the final, as well as the process of elimination, were clearly spelled out in the initial booklet. Keeping the rules "secret" from the competitors is unacceptable.

Instruction Booklet:
As the variations in sudoku should not take anywhere near the time to discuss as at an event like the WPC, the rules for these puzzles should be standardized and example puzzles released as soon as possible, certainly over a week and likely two weeks before the championship itself. The Czechs released a booklet with ~60 types of puzzles several weeks before the WSC in Prague to give solvers a sense of the puzzles that might appear at that championship. This is the standard that should be followed. Not all variations showed up, and a couple were added, but most of the “rules questions” were worked out well in advance of the competition. Having the rules presented online with a forum as on the worldpuzzle.org site is the right way to deal with “mistakes” in the booklet and add in corrections that can then be translated from English into the competitors’ languages before competition. Much as with the revealing of the rules of competition above, there should not be a benefit to a solver who is more fluent in English because no written version of a correct set of rules occurred.

Unlike many puzzle types seen at a WPC, sudoku have a clear means of determining how far a person is towards a solution – namely the number of correct digits placed in a cell - and WSC organizers should score with this in mind. A scoring system should reward correct solutions but also give partial credit (where deserving) to solutions that are far from blank. In a team round with a single large puzzle where almost no teams finished, the number of correct digits in the respective grids could easily have been used to assign a score to rank the teams in a logical fashion.

I personally propose the following example system as it makes sense to me: Puzzles are all graded out of 100. Correctly solving a puzzle, say a classic with 21 givens, gives you credit for all of the numbers you entered into the grid (60) as well as the givens (21) as well as 19 more points to get to 100 (60 + 21 + 19). Any solver that made a mistake in 1 cell gets a score equal to the number of correctly filled cells, 59. A blank grid is worth 0. If organizers want to score different puzzles differently, the score from this system should be treated as a “percent” of the scores and a correct solution to a 5 point puzzle is worth 5 points while a 59 cell solution with one mistake might be worth 3 (approximately 59% of 5) points. Whatever the scoring system, it should reward sudoku that are closer to solution, particularly since sudoku is almost always a puzzle type where it is easy to determine who is closer to a solution. It should certainly, in the playoffs, reward solvers who are closer to a 4th solution of a puzzle to finish a round than solvers who stopped after solving only 3 of the puzzles with a blank 4th grid.

Bonuses on some rounds should be given to solvers that can complete the tasks early. The best bonuses should be based exclusively on the time to finish the round, with some amount of points per minute awarded to any solver that finishes the round with a correct paper. Rank bonuses are not as good as time bonuses because they cannot handle rounds that are poorly timed for some of the competitors. Jakob Ondrousek’s phenomenal performance in the Relay Round at Goa, India would not have been fairly served if his 12-minute-early finish was given simply 5 points for being 1st instead of the large time bonus it received instead. In Zilina, competitors weren’t even allowed to earn some bonuses as proctors would not throw out incorrect papers to restore the proper bonus for 1st. If 5 people hand in papers and only the fifth is correct, certainly that person should earn the highest bonus for the round. All solvers should be able to turn in their paper until all bonuses have been claimed by correct papers, not by people saying "done" who are not done.

So, if rank bonuses must be used, they should be adjusted to whoever actually is first. However, the best bonus system will be based on time of finish, and a time-bonus should be strongly encouraged to organizers over other systems.

For many years the WPC occurred without a playoff and always crowned a person that the puzzle community would truly consider the champion. Since the introduction of playoffs, a measure of the excitement of sport has been added to the mix by allowing a reordering to occur on as few as one puzzle. Whether the WPF considers sudoku more like chess or more like football, it should agree on some standards that ensure the best sudoku solvers have a fair shot at winning. These standards should include the number of competitors who are admitted, and any bonuses that should be expected for order of solve such as a higher seed or better tie-break bonus.

Playoffs should never be run to encourage people to stop solving puzzles. At Zilina, in both round 2 (to qualify for the “Mini Playoff”) and in the Semifinal, the structure of the ranking encouraged solvers to stop solving the puzzles to post an earlier “time” for an incomplete round. 10-12 people said finished in Round 2 and 10-12 people said finished in the Semifinal and this gave the impression to any watching that someone had finished those rounds. In reality, no one finished correctly in either round. As mentioned above, in a situation where sudoku allows for partial scoring, the progress towards a solution should become the tiebreaker (if not that, then the ranking on the first day, which seems to be rewarded at a WPC but not at a WSC in the time I've competed at both) and solvers should not ever stop solving until time runs out. With 13 solvers in Zilina at 3 out of 4 puzzles solved in the semifinals, and with the solvers reaching that state at different times (I had 16 minutes left when I had just the 4th puzzle to go which unfortunately had that "human-solvable?" guessing property to it), different solvers adopted a different solve or quit strategy. The selection of places 4, 9, 11, 28, 29, 30, 33, 36 as the finalists was a result of both what three puzzles solvers had finished, and when solvers got their third puzzle solved and anyone who had 5 or fewer minutes at this point said finished and moved on to the detriment of the good solvers who were still trying to solve a fourth puzzle that they actually had the time to attempt.

Concluding Thoughts:
I’m sure that there are other particular issues that should be addressed, and that there are other problems from past World Sudoku Championships that should be mentioned that I have forgotten, but the steps of standardizing and checking the classic sudoku puzzles used in the championship to ensure human-solvability, scoring/ranking that is based specifically on sudoku’s ability for partial scoring, and fair revelation of the rules of competition to all competitors well before the finals themselves are to me not just suggestions but requirements before I will really feel like competing at a World Sudoku Championship again. While I do not know if I will compete or construct at the World Sudoku Championship in Philadelphia in 2010, I hope standards are adopted for the WSC to ensure a fair and fitting competition on sudoku for WSCs there and in the future. I am making this an open letter, and posting it on the World Puzzle forum as well as my blog, to encourage further discussion of the standards that should be applied to a sudoku competition from the community of solvers.
(Anonymous) on April 28th, 2009 10:29 pm (UTC)
I understand better what were the problems of WSC this year and I am in complete agreement with Mr. Snyder on the contents of this letter.

byronosaurusrex on April 28th, 2009 11:51 pm (UTC)
Although I have not personally attended a WSC for various reasons, as a 6-time WPC competitor who can reasonably be expected to compete in future WSCs, I co-sign to all recommendations in the above letter.

Additionally, I would suggest that particular care needs to be taken in handling of playoff papers, as well as in their judgment. The incident with Salih Alan's semifinal paper being misjudged could have been resolved easily, had the paper been given to either Salih or the Turkish captain. If there is a concern with judging, the competitor and their captain are the two people best prepared to express it; therefore, it should be ensured in future that one of these two receives the graded papers.

Byron Calver
2006 American Sudoku Smackdown Champion
Team Canada (WPC 2000, 2003-07)
zundevilzundevil on April 29th, 2009 12:49 am (UTC)
As usual, Thomas has a complete grasp of what happened in the past and what could be improved upon for the future. I trust his judgment above my own on all puzzle matters, with two addenda and one minor disagreement to his letter.

As a suggestion that may not have been considered before: bonuses for any rounds (and I believe this belongs for all rounds) should be based as a percentage of remaining time on the given round and the point-value of said round -- as something of a pro-rating. If a 100-point 60-minute round is correctly solved by a competitor in 40 minutes, then that competitor should get 50 bonus points; 100 points in 40 minutes extrapolates to 150 points in 60 minutes. This seems entirely fair.

Also, I do not think grading every paper with regards to completeness (on the 0-100 scale) is necessarily an effective use of the staff's time. I would absolutely say that such a step is valuable in a playoff setting, however.

Lastly, I think not only should the instruction booklet be given out long enough before the WSC, but there should be a full example of any variant puzzle that the ensuing WSC will pose, and this puzzle should ideally be a reasonable approximation of the difficulty the solvers will encounter in the actual competition. This may be more of a matter of taste instead of necessity, but I can only see things improving if this were made the standard.

Jason V. Zuffranieri
US Sudoku Team Member, 2007-present
Adam R. Woodzotmeister on April 29th, 2009 11:58 pm (UTC)
[I use a lighter tone in my message so as to keep me from barking obscenities in response to the idiocy Mr. Snyder is trying to remedy, and rightly so. I sincerely hope this is not seen as reducing his poignancy any. - ZM]

As someone who stands behind the curtain for one of your national-level competitions, and as someone with a solid understanding of game design and balance and personal experience devising and running tournaments of several varieties, I believe I'm sufficiently qualified to say with utter certainty that you are doing it wrong. I'm picturing Mr. Snyder's list of (seemingly) ninety-six points nailed to your door. Care to place odds on when the "World Puzzle Federation, Reformed" springs to life and supplants you?

But seriously, what Tom said. Watch a YouTube video of an American Crossword Puzzle Tournament playoff to see the proper grading hierarchy: accuracy first, then time. (Although Trip Payne may be sorely tempted to disagree, I'm sure he knows better.) The ACPT have been at this over a decade longer than you have - they know what they're doing. The fact that it's a different style of puzzle being competed over doesn't change a thing - it's still a grid the solvers need to fill cell by cell. All of the other points are administrative in nature, and that has nothing to do with the puzzles at all! Those who don't learn history...

Adam R. Wood
USPC contributor
(Anonymous) on April 30th, 2009 09:38 am (UTC)
I totally agree with Thomas’ letter (and all his posts).
I’m really disappointed by the last sudoku championship.
The main reason is obviously that Thomas wasn’t in the final round.
They did the same mistake after both part B and part M (the semifinal).
I think it’s an objective MISTAKE.
Also in part B they should have considered the number of digits to choose the 29th competitor who could go to play off extra. If in part B it was a bad mistake, in the semifinal it was A VERY BAD AND UNACCEPTABLE MISTAKE.
I know that counting digits could take a very long time, but if you can’t do it, you can’t decide to have a play off extra or a 36 competitors semifinal.
I also hope that in the next championships there won’t be “bifurcating” sudoku. I come to use logic skill, not to try to guess and trust in luck. (it concerns mostly the world record sudoku)
Anyway to me Thomas is the best sudoku player.

Elena Mazzini
(italian sudoku team)
(Anonymous) on April 30th, 2009 03:17 pm (UTC)
the best player
Elena, the rules were equal to all competitors. Fair or not, whatever, but equal. The best player had to consult unsuccesfully additionally with not friendly conditions.
I hope that in Philadelphia all will be not only equal but also fair and clear. And the player who'll win also be better in directly competition among the best players after essential parts.

Now, the champion is the only one, who officially. I'm sorry.

Best regards
(Anonymous) on April 30th, 2009 04:31 pm (UTC)
Re: the best player
In fact I didn't say that Thomas is the champion!
Maybe I should have written that he is one of the best players...
I can admit my mistakes!!
I'd like to clarify that I don't think the champion doesn't deserve his title, as he got 36 points in the semifinal and he achieved the right to enter the final without using the famous "time rule".
I don't think that "rule" was so clear for all competitors...
Now I would not dwell too much to comment my post, it's only what I think.
zundevil: Zeebazundevil on April 30th, 2009 06:09 pm (UTC)
Re: the best player
I think you are both right. Elena is right to say that Jan is a very deserving champion, and we all agree that the championship itself could have been better.

The problem with the championship is that very few people really believe that what happened in Slovakia determined who is the best sudoku solver in the world for 2009. Jan may be that person. Thomas may be that person. Or someone else may be that person. But due to the (equally-applied -- except for the lost Turkish paper) rules and format, there will always be doubts, and that is too bad.
(Anonymous) on April 30th, 2009 07:21 pm (UTC)
Re: the best player
Exactly!! Thank you!
Gabriele Simionatogabrieleud on April 14th, 2010 02:17 pm (UTC)
Re: the best player
In fact, I don't believe the rules have been applied equally to ALL players. The "missing paper case" against the player from Turkey is frankly unacceptable to me.
motrismotris on April 14th, 2010 04:10 pm (UTC)
Re: the best player
Wow - action on this old thread again.

The way the WPF board dealt with the missing paper case is also frankly unacceptable to me. Looking at the grid as graded, which I've discussed before with some of my "mistakes", I'd say there is ample cause to say there was a mistake in that grid or at least one if not several ambiguously labeled cells. If the decision of the Slovakian judges was to mark that paper wrong both initially, and then later when it was found and brought for review, then that is how the judging was going on at the tournament. For the board to vote to change the scoring of one paper makes a huge inconsistency in the conduct of the tournament.

Salih also had an error in the Sudoku Cup in Antalya that got 5+ minutes of scrutiny before it was determined - in part because the other competitor relented - that it was acceptable to have a functionally blank cell with a tiny tiny note (similar to identical notes in other cells that did not mark the digit type) to serve as an actual digit. I don't have a photo of that grid, but I'd say judging sudoku can be difficult and while the handing back of the paper was a sad case, the judging of the paper itself was likely accurate.

Edited at 2010-04-14 04:11 pm (UTC)
(Anonymous) on April 14th, 2010 04:50 pm (UTC)
Re: the best player
Looking at the grid, there surely are elements of arguing.

In my opinion, the grid is quite messy to give headaches to the judges. Anyway, a closer inspection shows 2 cells with two smaller numbers [6/7] and a bigger one (that is the correct one) and a cell with numbers 2 or 3 one over the other. In that case you can tell that the "2" is intended to share two vertical cells (as a long number 7 does on NorthEast sector), so the other option is correct.

Anyway, the problem is that the player could not proof his arguments, with or without reason, before the competition went on.
What if in the puzzle there were no ambiguities?
motrismotris on April 14th, 2010 04:56 pm (UTC)
Re: the best player
The most likely result, if there were no ambiguities, is that the puzzle would have been marked correct and having received his paper or not he would have been in the finals.

The hypothetical of a correct but marked incorrect paper that also was not turned back is why attention should always be given to returning papers directly to team captains (and not anyone else) to ensure all papers are returned appropriately.
(Anonymous) on April 30th, 2009 02:55 pm (UTC)
Thomas, I really appreciate your Open Letter, which shows that you care about WPF, and WSC in particular.
It is a welcome contribution to create a set of rules for the WSC, and for any sudoku competition in general, which I think is long overdue.
Seeing all this from the point of view of an organizer, I may have some disagreements with you mainly for practical or logistic reasons, but for the most part, in particular on puzzle matters (like Jason, I too trust your judgement above my own on this) I agree with you.
However, being the Italian representative of the WPF, I would like take my time before posting my thoughts on your Letter on the WPF board because I want to make sure of doing it right (and in proper English).

Riccardo Albini
WPF Italy
motrismotris on April 30th, 2009 02:57 pm (UTC)
I appreciate your comments Riccardo, and I encourage the WPF to take some time to consider what it wants to do before meeting again in Turkey.
(Anonymous) on April 30th, 2009 07:08 pm (UTC)
Yes I agree with most part of your letter but I feel most of your other blogs were a bit too harsh.

Anyways I can understand that you feel cheated after the championship and that there cannot be an explanation to the fact that amongst the 36 semifinalists you were the closest to finish all 4 puzzles and still you weren't in the finals!!!
As a participant of only 2 WSCs and with not much experience behind me, I might be wrong in few of my views but still I feel there should be a standard semifinals and finals format decided by the WPF for every year wherein the organizers can include whichever puzzles they feel best in the regional rounds. This is because every year we have different formats which gives rise to many debates (even if the format is good there will always be few questioning it). So one single fair-to-all format would be good.

Other than that i agree with you on most of the things like selection of puzzles, proper Instruction Booklet etc...

Gaurav Korde
(Anonymous) on May 4th, 2009 01:24 pm (UTC)
I, like Ricardo, really appreciate your open letter. WPF needs an open discussion about its problems, esspecialy after IV WSC. The 19th WPC will take place in Poland – so I’m really interested in such discussion to organize a remarkable event and to avoid previous mistakes in managing it.

I agree with part of your conclusinos and disagree with the rest of them. I’ll write about it on WPF website forum and I’m sure that during wide discussion among people taking care on WPF events we will be able to work out set of rules and set of good practices for future organizers of WPC and WSC.

I also understand your emotions and being upset after IV WSC in Zilina. I hope that these emotions were the reason that you have written things which I couldn’t appreciate. It is not fair when ex-champion is writing about the new one as a „champion” or „that guy may be a good solver”. In my opinion it is lack of respect to new champion and to the other finalists. As you know Jan is one of the best sudoku solver all over the world. The chaos in Zilina is not Jan’s fault. The rules were the same for all competitors an he was in the same situation as you. He won IV WSC and he took the title (not „title”) of World Sudoku Champion (not „Champion”).

I’m sure that V WSC in Philadelfia will be unforgettable event as was the I WPC I had attended in New York in 1992. I hope Jan will defend his title but everything may happen. One thing I’m sure of: if somebody will win with Jan, he will not write that new Champion is „Champion” but he will congratulate and shake hands with her or him.

Jacek Szczap
WPF Poland
motrismotris on May 4th, 2009 04:10 pm (UTC)
Thanks for your comments and I look forward to reading your discussion of what should occur at future competitions including WPC 19.

Regarding my earlier posts - all were written during the championship and the harshest ones were made before the finals had even occurred about a half hour after I learned how I was eliminated, so I was mainly questioning the integrity of the event in selecting the world's best solver and not the person who managed to take the podium. Jan has been on my radar for awhile now, given some of his online times. He was also on my short-list of five or so people I thought could/should win this year. I look forward to having the opportunity to compete for his title at a future WSC, where the puzzles are all logically solvable and where the competition rules are fair.
(Anonymous) on November 6th, 2009 05:16 am (UTC)
Wel i'd have to say Im in full agreement as well. In fact this is an excellent article Im going to consider submitting to daily sudoku (http://dailysudoku.info)
(Anonymous) on November 6th, 2009 05:18 am (UTC)
Well i'd have to say Im in full agreement as well. In fact this is an excellent article Im going to consider submitting to http://dailysudoku.info likley Ill put it on http://sudokusolver.info as well.

Thanks everyone for the excellent comments on this,