Help us with Testing and Benchmarking the MySQL Server

snaugle-Oracle

Help us with Testing and Benchmarking the MySQL Server

• Provide your MySQL tests to us, and we try to include them in our internal development testing
• Provide feedback on your in-house testing and benchmarking to us, and help us learn to improve our products to your needs
• Help us Help You!

Overview

MySQL has always taken testing seriously. While the feature set of our products grow, our test and benchmarking suites have been steadily increasing in size and scope. Our dedicated engineers are devoted to these tasks, and they test relentlessly every product in every stage of the development process. However, laboratory tests can't always match the real world. We are aware that in the real world, outside our virtual laboratories, MySQL is being used in ways that we don't foresee. Innovative enterprises based on MySQL are a common pattern in the business and technological news. If we would have had the magical ability to predict how companies such as Wikipedia, Flickr, Youtube, or Joost would be using MySQL, we would likely be in the business of buying and selling their stocks today. Instead, we are in the database business, and we stick to our commitment of developing and supporting the best online database. If we want achieve our goal, our database must meet the needs of our users. Since our divination powers aren't better than the average layman, we need the community help to improve and integrate our testing and benchmarking practice.

The goal

We want to expand our quality assurance practice with the community involvement. The main goal is to find faults in our products while they are still in development, using the tests that our users would perform when the product is production ready. If we find a fault using a community provided test, it will be mutually beneficial. We will fix the bug before it affects a production service, and the users can be assured that the new version of the product works in their environment.

What we are looking for

Basically, we need to acquire testing and benchmarking practice from the users perspective. We are interested in learning how our users test and benchmark their systems, and how we can integrate such practice with our quality assurance routine.We would like to hear about:

• Your testing practice. What you test, when, and how.
• Your benchmarking environment and procedures.

Your practice is important to us. If your testing techniques are finding bugs that escape ours, we will definitely want to hear about that. If your benchmarking environment uncover bottlenecks that we did not know about, we definitely need to have a word with you.Moreover, we would like to exchange notes with users about:

• What kind of Hardware, operating system, setups you use for MySQL;
• What kind of data structures, schema and table organization affects real world business;
• What amount of data are spread among your schemas and tables.

In short, our user base is so large and differentiated that we can't possibly imagine every usage for our database. We need to integrate our current practice with information from the field.

What you can do to help

If you recognize that your business is somehow different from the ordinary patterns (in size, scope, concurrency, or usage), talk to us!The practical ways of giving a hand are:

• Provide us your data structure and sample data that we can add to our testing suites;
• If you have a unique environment that can't be easily replicated, or if your data is so sensitive that even a sample can't be shared, run some dedicated tests in your location and share the results with us.

Most important, since we have already confessed our lack of omniscience, we are open to suggestions. If your testing and benchmarking practice include any aspect from which we can benefit, feel free to share.

To share or not to share

We are an open source company, and we are trying hard to make our business model even more open than before. As such, we appreciate community contributions to be open and widely shareable.
At the same time, we understand that many business may have reasonable motive to avoid sharing their data or their environment specifics. We respect and understand your need for privacy, and we are ready to accept discreet and confidential offers of cooperation, under terms that are favorable to our users. Our goal is to improve the overall quality of our products, not to undermine our users business. Therefore, we will listen to any offer, on either open or closed terms.

To ease the problems related to privacy and confidentiality of data, we are planning to release a data anonymizer.

Tools for testing

Testing is difficult. To make your life easier, we have created some testing tools. You can use them to convert a general log (or part of it) to a proper test script. See Test suites for the masses

Unit Tests

* VisiBone's MySQL Unit Tests (see the bullet point Unit Tests), version 4-5, Linux and Windows. Here are a whole bunch of unit tests I made to plumb the corners of the language for researching the cheatsheets. Click on the [+] symbols to see the MySQL code and results. The tables can be seen in the "Country Database" section, "Raw Tables" subsection.Bob Stein - VisiBone 14:22, 29 May 2007 (UTC)