Optimization Benchmarking

The BBOB Example

In this folder, we provide an example data set of testing the application of the optimizationBenchmarking.org evaluator. We use it to investigate some of the results of the Black-Box Optimization Benchmarking (BBOB) 2013 .

1. Example Structure

This example is structured as follows

  1. The archive results.zip contains the results of these algorithms. These results are stored in the format native to COCO without any modification except that we left away files that our system does not need to make the example smaller.
  2. The file evaluation.xml contains an example evaluation definition. We apply a set of different modules and configurations.
  3. The file configForIEEEEtran.xml is a configuration that applies evaluation.xml to the results and renders the output as LaTeX document styled according to IEEEtran.cls into a folder named reports/LaTeX/IEEEtran/.
  4. The file configForLNCS.xml is a configuration that applies evaluation.xml to the results and renders the output as LaTeX document styled according to LLNCS.cls into a folder named reports/LaTeX/LNCS/.
  5. The file configForSigAlternate.xml is a configuration that applies evaluation.xml to the results and renders the output as LaTeX document styled according to sig-alternate.cls into a folder named reports/LaTeX/SigAlternate/.
  6. The file configForXHTML.xml is a configuration that applies evaluation.xml to the results and renders the output as XHTML document into a folder named reports/XHTML/.
  7. The file configForExport.xml is a configuration that applies evaluation.xml to the results and renders the output as text (for import into other tools) into a folder named reports/export/.

2. BBOB and COCO

COmparing Continuous Optimisers (COCO) is a platform for systematic and sound comparisons of real-parameter global optimization algorithms. COCO provides benchmark functions, experimentation templates which are easy to parallelize, and tools for processing and visualizing data generated by one or several optimization algorithms. The COCO platform has been used for the Black-Box-Optimization-Benchmarking (BBOB) workshops that took place during several GECCO conference since 2009.

The example data used here was gathered by other researchers following the COCO experimental procedure on the noiseless benchmark functions.

3. Experimental Data

The algorithms are described here and were investigated independently by their respective authors. We do not claim, own, or assume any responsibility, copyright, authorship, or liability for any of the data provided here.

1. holtschulte2013_ga100 Neal Holtschulte and Melanie Moses. Benchmarking Cellular Genetic Algorithms on the BBOB Noiseless Testbed. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07
2. holtschulte2013_hill Neal Holtschulte and Melanie Moses. Benchmarking Cellular Genetic Algorithms on the BBOB Noiseless Testbed. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07
3. hutter2013_CMAES Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. An Evaluation of Sequential Model-Based Optimization for Expensive Blackbox Functions. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07
4. liao2013_IPOP Tianjun Liao and Thomas Stützle. Testing the Impact of Parameter Tuning on a Variant of IPOP-CMA-ES with a Bounded Maximum Population Size on the Noiseless BBOB Testbed. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07
5. liao2013_IPOP-texp Tianjun Liao and Thomas Stützle. Testing the Impact of Parameter Tuning on a Variant of IPOP-CMA-ES with a Bounded Maximum Population Size on the Noiseless BBOB Testbed. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07
6. pal2013_DE László Pál. Benchmarking a Hybrid Multi Level Single Linkage Algorithm on the BBOB Noiseless Testbed. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07
7. pal2013_fmincon László Pál. Comparison of Multistart Global Optimization Algorithms on the BBOB Noiseless Testbed. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07
8. pal2013_HMLSL László Pál. Benchmarking a Hybrid Multi Level Single Linkage Algorithm on the BBOB Noiseless Testbed. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07
9. pal2013_simplex László Pál. Comparison of Multistart Global Optimization Algorithms on the BBOB Noiseless Testbed. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07
10. P-DCN Thanh-Do Tran, Dimo Brockhoff, and Bilel Derbel. Multiobjectivization with NSGA-II on the Noiseless BBOB Testbed. GECCO'13 Companion, July 6–10, 2013, Amsterdam, The Netherlands. ACM 978-1-4503-1964-5/13/07

4. License

The copyright of the datasets belong to their respective owners. The file results.zip is thus excempt from this project’s general licensing terms.

References

  1. Nikolaus Hansen, Anne Auger, Steffen Finck, and Raymond Ros. Real-parameter black-box optimization benchmarking: Experimental setup. Technical report, Orsay, France: Université Paris Sud, Institut National de Recherche en Informatiqué et en Automatique INRIA) Futurs, Equipe TAO, March 24, 2012. URL.
  2. Nikolaus Hansen, Anne Auger, Steffen Finck, and Raymond Ros. Real-parameter black-box optimization benchmarking 2009: Experimental setup. Rapports de Recherche RR-6828, Institut National de Recherche en Informatique et en Automatique (INRIA), October 16, 2009. URL. Version 3.
  3. Steffen Finck, Nikolaus Hansen, Raymond Ros, and Anne Auger. Real-parameter black-box optimization benchmarking 2010: Presentation of the noiseless functions. Technical report, April 13, 2013. URL. Working Paper 2009/20, compiled April 13, 2013.

[home] • [status] • [atom feed] • [rss feed]

Contact: Dr.  Thomas Weise, http://www.it-weise.de, tweise@ustc.edu.cn