CA Logo
CA Logo

Testing Ranking System Performance

[<<] [>>] by Louis Nel
24th January 2005 (Other News)

Introductory Remarks

Whether we like it or not, our World Ranking system is becoming more and more integrated into our croquet lives. It is not surprising that questions about its performance repeatedly surface in discussions on the Nottingham Board. Will Step size 60 perform better than 40? Will a larger Smoothing parameter give better performance? Such questions remain largely unanswered. One reason is that there are no known criteria for what "better performance" means.

The purpose of this article is to initiate the development of performance testing. It involves the following steps.

Setting up a Test Population

Setting up a Trial System (as rival for the Current System)

Setting up a Test

Studying the Test Results

We illustrate the procedure by carrying out these steps one by one. Here is a quick peek at what lies ahead.

Our first illustrative Test involves a Test Population created so that all players start with randomly selected Grades and Indexes within 200 of what the true value should be. The idea is to see which system will recover better from such a deliberately bad start -- a start that gave a Standard Deviation of about 115. At the end of the experiment, we find that the current system had improved the quality of the Grades and Indexes quite nicely. For the 30+ population (i.e. at least 30 games played) it yielded Standard Deviations of 63.0 (for Grades) and 75.3 for Indexes -- well down from the 115 at the start. The Trial System under scrutiny did even better: 59.4 (for Grades) and 60.2 (for Indexes). Note that the Indexes of the Trial System outperformed even the Grades of the Current System. However, that is not the end of the story. In a similar second experiment in which starting Grades and Indexes need only be within 400 of the true value, the Current System flexed some muscle. Its Grades performed better than those of the Trial System; its Indexes only where few games were played. It seems fair to say that the Testing Procedure envisaged will not end all arguments. Hopefully, the arguments that remain will at least become more focused and better informed. For example, as regards the mentioned experiments, the arguments might become focused on whether the system normally operates within 200 of the target or whether the wider margin of 400 is the more realistic one. And, of course, what supplementary Tests are needed in order to arrive at a well founded decision.

the full article (on Louis' site)



 

For results, news, reports, corrections, updates, etc. see contact the CA. As an anti-spam measure when not logged-in, # replaces @ in email addresses - click on the link as usual to send an email.

The CA will not share your personal details with any other organisation and undertakes to keep them confidential as described in our Security, Privacy, and Cookie Policies and Information.

Disclaimer: The opinions expressed on this news web page are those of the Editor and contributors. The Croquet Association is not responsible for statements other than those clearly identified as being made on its behalf. The full editorial policy is available online. The Web News Editor is Sam Murray.