Difference between revisions of "Openings FAQ"

From TCEC wiki
Jump to: navigation, search
(→‎TCEC Openings FAQ)
(→‎TCEC Openings FAQ: |League 4)
 
(30 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
==TCEC Openings FAQ==  
 
==TCEC Openings FAQ==  
 
    
 
    
−
*'''Q:  Why do TCEC chess games not start from the opening position?'''  
+
;Q:  '''Why do TCEC chess games not start from the opening position?'''  
 +
:A:  The answer is rooted on our site’s general objectives, past experience trying different play formats, and our judgment on what makes an entertaining competition and therefore a successful website.
 +
::Our primary goal is to provide our web audience with fair and entertaining chess competitions. 
 +
::We believe that a champion chess program ought to be able to perform well in a wide variety of opening systems, not merely those that it favors.  This approach is different from human tournaments by design.
 +
::We see the traditional opening position as merely one position out of many thousands that might present a chess program with a challenge.  What distinguishes the opening position in computer chess is that it is relatively balanced, meaning it has a high probability of resulting in a draw when played by top competitors at long time controls on very strong hardware.
 +
::Many chess programs repeatedly play a given position the same way.  When two such programs meet, they tend to produce repetitive contests.  A 100-game test we ran between leading engines a few years ago resulted in 88 of the games concentrating into just five ECOs.  Such outcomes frustrate the majority’s desire for entertaining chess. 
 +
::The fact that different competitors must cope with an unpredictable and unequal set of openings does add an element of chance to the competition. Mitigating this chance element,
 +
::#competitors play the selected openings from both sides of the board in consecutive games,
 +
::#the length of our season tournaments assures a wide variety of non-repetitive openings, and
 +
::#as a whole, the openings tend to balance out in terms of drawishness and bias.
 +
::This final point underscores the principal risk to our approach, namely, that competitors can be saddled with openings that are too draw-prone or too one-sided, resulting in game-pairs that fail to differentiate the competitors on the cross-table.  Our openings team is skilled at selecting opening positions that are neither hopelessly draw-prone nor so one-sided that the weaker side has no chance of holding the draw. 
  
−
*A:  The answer is rooted on our site’s general objectives, past experience trying different play formats, and our judgment on what makes an entertaining competition and therefore a successful website.
+
;Q:  '''Who selects the openings?'''
−
 
+
:A:  Nelson Hernandez (“CatoTheYounger_TCEC”) has led the openings team since mid-way through Season 5 (2013).  Over time, the team has grown and evolved in order to encompass different approaches, tools and philosophies.  Veteran computer chess book-maker Jeroen Noomen (“Jeroen_TCEC”) has headed Superfinal opening selections since Season 9.  More recently, the addition of Eduardo Saucedo (“Cookie_Monster_71”) and Bastiaan J. Braams (“kbg519v1a”) gave us a quartet.  Each of these gentlemen bring chess knowledge, statistical savvy, practical experience as well as their unique personalities to their opening selections.   
−
:·        Our primary goal is to provide our web audience with fair and entertaining chess competitions. 
 
−
 
 
−
:·        We believe that a champion chess program ought to be able to perform well in a wide variety of opening systems, not merely those that it favors. This approach is different from human tournaments by design.
 
−
 
 
−
:·        We see the traditional opening position as merely one position out of many thousands that might present a chess program with a challenge.  What distinguishes the opening position in computer chess is that it is relatively balanced, meaning it has a high probability of resulting in a draw when played by top competitors at long time controls on very strong hardware.
 
−
 
 
−
:·        Many chess programs repeatedly play a given position the same way.  When two such programs meet, they invariably produce repetitive contests.  A 100-game test we ran between leading engines some time ago resulted in 88% of the games concentrating into just five ECOs.  Such outcomes frustrate the majority’s desire for entertaining chess. 
 
−
 
 
−
:·        The fact that different competitors must cope with an unpredictable and unequal set of openings does add an element of chance to the competition. Mitigating this chance element,
 
−
:1) competitors play the selected openings from both sides of the board in consecutive games,
 
−
:2) the length of our season tournaments assures a wide variety of openings, and
 
−
:3) the openings tend, over time, to balance out in terms of drawishness and bias.   
 
  
−
:This latter point underscores the principal risk to our approach, namely, that competitors can be saddled with openings that are too draw-prone or too one-sided, resulting in game-pairs that fail to differentiate the competitors on the cross-table.  Our openings team is skilled at selecting opening positions that are neither hopelessly draw-prone nor so one-sided that the weaker side has no chance of holding the draw. 
+
;Q:  '''How are the openings structured?'''  
−
+
:A:  Each seasonal tournament, from the Qualifying League through the Superfinal, has a different opening ‘book’ that issues starting positions:  
−
*'''Q:  Who selects the openings?'''
 
−
 
−
:A:  Nelson Hernandez (“CatoTheYounger_TCEC”) selects openings in all tournament stages except the Superfinal, which is selected by Jeroen Noomen (“Jeroen_TCEC”). 
 
−
 
−
:Nelson took the openings reins in 2013, mid-way through Season 5, during founder Martin Thoresen’s period of tournament directorship.  Nelson’s is primarily known for his proprietary ‘Catobase’, a database of human and engine games now over 15 years in active development.  He was a participant in Anson Williams’ three-time champion freestyle chess team.  He has been enthralled by chess programs since 1982, when he bought Sargon II for his Apple II+.
 
−
 
−
:Jeroen took responsibility for the Superfinal at Nelson’s invitation before Season 9.  He has been active in computer chess since the early 1980s.  He has been associated with the Rebel and Rybka programs in the past as an opening book consultant and over the years has participated in numerous live computer chess tournaments in the Netherlands and elsewhere.
 
−
 
 
−
*'''Q:  How are the openings structured?'''  
 
−
 
 
−
*A:  Each seasonal tournament, from the Qualifying League through the Superfinal, has a different opening ‘book’ that issues starting positions:  
 
−
 
 
{| class="wikitable" style="margin-left: 20px;"
 
{| class="wikitable" style="margin-left: 20px;"
 
!Tournament
 
!Tournament
−
!Book             
 
 
!Author       
 
!Author       
−
!Book Length  
+
!Length   
−
!# Positions    
+
!Selection
−
!Selection    
+
!Opening Bias
−
!Opening Bias    
+
!#Exits   
−
!Perpetual
 
 
|-
 
|-
 
|Qualifying League     
 
|Qualifying League     
−
|Book A
+
|none
 +
|0 moves
 +
|none                     
 +
|none
 +
|1
 +
|-
 +
|League 4
 
|Nelson       
 
|Nelson       
−
|2 moves              
+
|4 moves                          
−
|1000           
 
 
|random         
 
|random         
−
|none                      
+
|none  
−
|yes
+
|1000                     
 
|-
 
|-
−
|League 3  
+
|League 3
−
|Book B   
+
|Bastiaan and Eduardo
−
|Nelson     
+
|1 to 8 plies                       
−
|4 moves               
 
−
|1000           
 
 
|random         
 
|random         
−
|none                     
+
|moderate
−
|yes
+
|300                     
 
|-
 
|-
 
|League 2                     
 
|League 2                     
−
|Book C
+
|Eduardo      
−
|Nelson      
+
|varied                
−
|6 moves                
+
|consecutive
−
|2000           
+
|high
−
|random       
+
|56
−
|none                     
 
−
|yes
 
 
|-
 
|-
−
|League 1                  
+
|League 1                    
−
|Book D
+
|Bastiaan
−
|Nelson     
+
|varied              
−
|8 moves              
+
|consecutive
−
|1000           
+
|high
−
|random       
+
|56   
−
|low/moderate     
 
−
|yes
 
 
|-
 
|-
−
|Premier League        
+
|Premier Division        
−
|Book E
 
 
|Nelson       
 
|Nelson       
−
|8 moves            
+
|8 moves                  
−
|1000           
+
|consecutive     
−
|random       
+
|high
−
|moderate/high    
+
|112 
−
|yes
 
 
|-
 
|-
 
|Superfinal                   
 
|Superfinal                   
−
|SuFiBook
+
|Jeroen & Sadler      
−
|Jeroen       
 
 
|varied                       
 
|varied                       
−
|50           
+
|consecutive
−
|sequential 
+
|high 
−
|high/very high     
+
|50
−
|no
+
|-
 
|}
 
|}
  
−
:<nowiki>#</nowiki> Positions signifies the number of unique positions contained within each book. 
+
:Selection indicates how positions are selected.  Random selections are not repeated once used.  Consecutive selections are chosen from a .pgn file in the order submitted to the tournament director.  Bookmakers submit their positions weeks or months in advance and have no insight into which engines will be participating or how they will be seeded.
−
:Selection indicates how positions contained within the books are selected.  (Thus, in League 1 there are 1,000 unique positions in the book which are chosen at random prior to each new game-pair.)
+
:Opening Bias indicates the degree to which the average opening is skewed to favor one color or the other.  In past seasons bias was graduated as the level of competition increased, but with the increases in program strength that we have seen in recent years—and the upgrade of TCEC hardware—we now find that considerable bias is needed starting with League 2.  We reassess after each season.  
−
:Opening Bias indicates the degree to which the average opening is skewed to favor one color or the other.  (An exception to this is that Jeroen sometimes starts the Superfinal with two bookless games.)
 
−
:Perpetual indicates that the first four leagues get starting positions from the same book from season to season whereas Superfinal books are for one-time use with a new book each season. 
 
−
 
−
*'''Q:  How were positions within the books selected?'''
 
−
 
 
−
*A:  For the Qualifying League, League 3 and League 2, Nelson has tried to balance three objectives: 
 
−
:1) to deliver the widest variety of opening systems,
 
−
:2) to keep the overall draw-rate under 80%, and
 
−
:3) to limit the instances of same-color wins from game-pairs. 
 
−
 
 
−
:To find these positions Nelson did a comprehensive search of his Catobase to look for positions that met various criteria within his empirical and evaluative data.  He avoided book-exit positions where the next move was obvious, where pieces already had been (or were about to be) exchanged, or which had happened very rarely in his database.  To achieve a subsidiary objective, he constructed these two books in such a way that openings selected tend to reflect human preferences as indicated in his database.
 
−
 
−
:For League 1 and Premier League the caliber of the competition is much higher and longer time controls at these levels further boost play-quality.  For this reason opening bias must be introduced to a progressively higher degree otherwise the draw-rate will soar to unacceptable levels. 
 
−
 
 
−
:In response, two 8-move books were created for the top two leagues.  In terms of “ECO-diversity” they are rather similar but the most volatile openings within each opening system were assigned to the Premier League.  Separately, the preference for human openings seen in the first two leagues of the season is not present in these books; the positions are informed by tens of millions of Catobase games.
 
−
 
−
:Jeroen’s approach to the Superfinal is less statistical and much more chess knowledge-driven.  His process is to use leading chess engines to check if a line is obviously drawish or is certain to lead to heavy exchanges.  He judges whether a given position offers the prospect of a good fight with multiple playing options for both colors.  His opening picks always include a wide variety of systems, pawn structures and themes.  There are positional and sharp openings, those with short and long castling, gambits, main lines, antique and topical lines, and occasionally a dubious line.  As long as dubious lines are not hopelessly one-sided we think they are fine.
 
−
 
 
−
:Note that both Nelson and Jeroen freely acknowledge that few traditional and neural net engines would find themselves in most TCEC opening positions of their own accord.  This is deliberate.  The whole point is to force engines to play unexpected positions and demonstrate their strength in all kinds of game situations.  This is especially true for neural nets, which in training tend to emphasize favorite lines and neglect what they deem to be inferior ones.  
 
−
 
 
−
*'''Q:  What do you mean when you say you “bias” openings, and why do you do it?'''
 
  
−
*A:  Bias is the degree to which a particular opening, or a set of openings, seem to favor one side or the other according to empirical data (historical outcomes) and evaluative data (deep analysis from top engines).   
+
;Q:  '''To what degree do you bias openings?'''
−
 
+
:A:  This varies from bookmaker to bookmaker.  But it is a bit difficult to quantify because different programs use different evaluation scales.  What is +1.10 to Stockfish is comparable to +0.60 for Leela or Komodo.  Nelson, for example, ranks candidate book-exit positions on a percentile basis and narrows his selections to those between the 84th and 93rd percentile, which comes to an average of around +0.90 on Stockfish’s scale.  Above that, he deems the position too biased.  Below, not enough.  (The other bookmakers do not use MS Excel to select openings.)
−
:The opening position typically offers white a slightly favorable evaluation of +0.15 to +0.20 on traditional chess programs where the “comtempt” setting is neutralized.  Low or zero bias in an opening book makes little difference in a tournament where time controls are shorter and contestants have a wider range of Elos. In that case, there will be ample decisive games.  
 
−
   
 
−
:However, if very strong, closely matched programs are playing at long time controls with no bias that invites a draw-rate approaching 90%.  To combat this, we introduce bias in League 1 and to an even greater degree in Premier League and the Superfinal.  In the latter two, you will see some openings where one color has no realistic winning chances and the game is all about successfully holding a draw from an almost desperate position.  
 
−
 
 
−
*'''Q: To what degree do you bias openings?'''
 
  
−
*A:  In League 1 Nelson is in his comfort zone when opening evaluations in traditional chess engines (i.e. not neural nets) are between +/-0.40 and +/-0.65.  In the Premier League, a score of +/-0.50 to +/-0.80 is more common.  Sometimes he will deviate from these guidelines to the upside or downside to satisfy opening variety goals.
+
;Q:  '''Why such an aversion to draws?  If chess is naturally drawish, why try to distort that reality?'''
−
 
+
:A:  The ultimate objective of a tournament is to crown a deserving champion.  If you play a match of 100 Superfinal games and 95 of them are draws you will not only drive away much of your audience, you wind up crowning a champion that may have only scored three wins against his opponent’s two.  In that event, statistics tell us that the winner may have qualified for the Superfinal based on skill but may have won it entirely on luck.  A greater number of decisive game-pairs (i.e. 1.5-0.5, 2-0) reduce the probability of luck being the critical factor as there is a greater chance of one engine pulling away for a clear win.
−
:In each season’s Superfinal Jeroen has a free hand to do whatever he thinks will result in an interesting and varied contest. Sometimes his book-exit evaluations will exceed +/-1.00; sometimes he will offer speculative gambits.  His goal is to keep the draw-rate in the 65-80% range each season without a surfeit of one-sided openings.  
+
:While we do not object to close contests—on the contrary—we would prefer as many decisive (i.e. not tied) game-pairs as possible, as this way the strongest competitor will have the greatest chance to demonstrate all-around superiority.  Plus, it is more entertaining!
−
 
 
−
*'''Q: Why such an aversion to draws?  If chess is naturally drawish, why try to distort that reality?'''
 
  
−
*A:  The ultimate objective of a tournament is to crown a deserving champion.  If you play 100 Superfinal games and 95 of them are draws you will not only drive away much of your audience, you wind up crowning a champion that may have only scored three wins against his opponent’s two.  In that event, statistics tell us that the winner may have qualified for the Superfinal based on skill but may have won it entirely on luck.  A greater number of decisive game-pairs (i.e. 1.5-0.5, 2-0) reduce the probability of luck being the critical factor as there is a greater chance of one engine pulling away for a clear win.
+
;Q:  '''How long can you maintain your targeted 60-80% draw-rate when engines are continually improving?'''  
−
 
+
:A:  Draw abatement will definitely be more challenging as the competitors increase in strength in the years ahead.  A mental comparison between today’s leading chess programs and those of ten years ago, combined with extrapolation into the future, points to a time where something will have to give.  On the other hand, who really knows what the future will bring?  Recent years have been more revolutionary than evolutionary.
−
:While we do not object to close contests in the Superfinal — on the contrary — we would prefer as many decisive (i.e. not tied) game-pairs as possible, as this way the strongest competitor will have the greatest chance to demonstrate all-around superiority.  Plus, it is more entertaining!
 
−
 
 
−
*'''Q:  How long can you maintain your targeted 65-80% draw-rate in the Superfinal when engines are continually improving?'''  
 
  
−
*A:  Draw abatement will definitely be more challenging as the competitors increase in strength in the years ahead.  A mental comparison between today’s leading chess programs and those of ten years ago, combined with extrapolation into the future, points to a time where something will have to give.  On the other hand, as long as neural nets and the best alpha-beta engines remain competitive with each other their contrasting play-styles may help keep draw-rates lower than they otherwise might have been.
+
;Q:  '''What impact will the advent of neural nets and their very different evaluative frame of reference have on your opening set selections?'''  
−
 
+
:A:  This is impossible to answer as each executable/neural net pairing is unique and presents a different behavioral profile with respect to strength, resistance to draws, evaluation volatility and reliability, etc.  The whole field of AI is undergoing rapid development with innovations happening continually; the competitive landscape changes from season to season in surprising ways.
−
*'''Q:  What impact will the advent of neural nets and their very different evaluative frame of reference have on your opening set selections?'''  
 
  
−
*A:  This is impossible to answer as each executable/neural net pairing is unique and presents a different behavioral profile with respect to strength, resistance to draws, evaluation volatility and reliability, etc.  The whole field of AI is undergoing rapid development with innovations happening continually; the competitive landscape changes from season to season in surprising ways.
 
  
−
:That said, TCEC’s history is one of gradual evolution in response to emerging trends and practical experience; we expect that to continue.
 
  
 
[[Category:FAQ]]
 
[[Category:FAQ]]

Latest revision as of 23:23, 5 February 2022

TCEC Openings FAQ

Q
Why do TCEC chess games not start from the opening position?
A: The answer is rooted on our site’s general objectives, past experience trying different play formats, and our judgment on what makes an entertaining competition and therefore a successful website.
Our primary goal is to provide our web audience with fair and entertaining chess competitions.
We believe that a champion chess program ought to be able to perform well in a wide variety of opening systems, not merely those that it favors. This approach is different from human tournaments by design.
We see the traditional opening position as merely one position out of many thousands that might present a chess program with a challenge. What distinguishes the opening position in computer chess is that it is relatively balanced, meaning it has a high probability of resulting in a draw when played by top competitors at long time controls on very strong hardware.
Many chess programs repeatedly play a given position the same way. When two such programs meet, they tend to produce repetitive contests. A 100-game test we ran between leading engines a few years ago resulted in 88 of the games concentrating into just five ECOs. Such outcomes frustrate the majority’s desire for entertaining chess.
The fact that different competitors must cope with an unpredictable and unequal set of openings does add an element of chance to the competition. Mitigating this chance element,
  1. competitors play the selected openings from both sides of the board in consecutive games,
  2. the length of our season tournaments assures a wide variety of non-repetitive openings, and
  3. as a whole, the openings tend to balance out in terms of drawishness and bias.
This final point underscores the principal risk to our approach, namely, that competitors can be saddled with openings that are too draw-prone or too one-sided, resulting in game-pairs that fail to differentiate the competitors on the cross-table. Our openings team is skilled at selecting opening positions that are neither hopelessly draw-prone nor so one-sided that the weaker side has no chance of holding the draw.
Q
Who selects the openings?
A: Nelson Hernandez (“CatoTheYounger_TCEC”) has led the openings team since mid-way through Season 5 (2013). Over time, the team has grown and evolved in order to encompass different approaches, tools and philosophies. Veteran computer chess book-maker Jeroen Noomen (“Jeroen_TCEC”) has headed Superfinal opening selections since Season 9. More recently, the addition of Eduardo Saucedo (“Cookie_Monster_71”) and Bastiaan J. Braams (“kbg519v1a”) gave us a quartet. Each of these gentlemen bring chess knowledge, statistical savvy, practical experience as well as their unique personalities to their opening selections.
Q
How are the openings structured?
A: Each seasonal tournament, from the Qualifying League through the Superfinal, has a different opening ‘book’ that issues starting positions:
Tournament Author Length Selection Opening Bias #Exits
Qualifying League none 0 moves none none 1
League 4 Nelson 4 moves random none 1000
League 3 Bastiaan and Eduardo 1 to 8 plies random moderate 300
League 2 Eduardo varied consecutive high 56
League 1 Bastiaan varied consecutive high 56
Premier Division Nelson 8 moves consecutive high 112
Superfinal Jeroen & Sadler varied consecutive high 50
Selection indicates how positions are selected. Random selections are not repeated once used. Consecutive selections are chosen from a .pgn file in the order submitted to the tournament director. Bookmakers submit their positions weeks or months in advance and have no insight into which engines will be participating or how they will be seeded.
Opening Bias indicates the degree to which the average opening is skewed to favor one color or the other. In past seasons bias was graduated as the level of competition increased, but with the increases in program strength that we have seen in recent years—and the upgrade of TCEC hardware—we now find that considerable bias is needed starting with League 2. We reassess after each season.
Q
To what degree do you bias openings?
A: This varies from bookmaker to bookmaker. But it is a bit difficult to quantify because different programs use different evaluation scales. What is +1.10 to Stockfish is comparable to +0.60 for Leela or Komodo. Nelson, for example, ranks candidate book-exit positions on a percentile basis and narrows his selections to those between the 84th and 93rd percentile, which comes to an average of around +0.90 on Stockfish’s scale. Above that, he deems the position too biased. Below, not enough. (The other bookmakers do not use MS Excel to select openings.)
Q
Why such an aversion to draws? If chess is naturally drawish, why try to distort that reality?
A: The ultimate objective of a tournament is to crown a deserving champion. If you play a match of 100 Superfinal games and 95 of them are draws you will not only drive away much of your audience, you wind up crowning a champion that may have only scored three wins against his opponent’s two. In that event, statistics tell us that the winner may have qualified for the Superfinal based on skill but may have won it entirely on luck. A greater number of decisive game-pairs (i.e. 1.5-0.5, 2-0) reduce the probability of luck being the critical factor as there is a greater chance of one engine pulling away for a clear win.
While we do not object to close contests—on the contrary—we would prefer as many decisive (i.e. not tied) game-pairs as possible, as this way the strongest competitor will have the greatest chance to demonstrate all-around superiority. Plus, it is more entertaining!
Q
How long can you maintain your targeted 60-80% draw-rate when engines are continually improving?
A: Draw abatement will definitely be more challenging as the competitors increase in strength in the years ahead. A mental comparison between today’s leading chess programs and those of ten years ago, combined with extrapolation into the future, points to a time where something will have to give. On the other hand, who really knows what the future will bring? Recent years have been more revolutionary than evolutionary.
Q
What impact will the advent of neural nets and their very different evaluative frame of reference have on your opening set selections?
A: This is impossible to answer as each executable/neural net pairing is unique and presents a different behavioral profile with respect to strength, resistance to draws, evaluation volatility and reliability, etc. The whole field of AI is undergoing rapid development with innovations happening continually; the competitive landscape changes from season to season in surprising ways.
Retrieved from "https://wiki.chessdom.org/index.php?title=Openings_FAQ&oldid=3130"