Granted, if you are on a shared hosting solution, you won't have much control over these changes, but most hosts will tune their MySQL as much as possible.
But if you are hosting your forum on a dedicated server and have full control over your configuration, this may help. A lot of times, MySQL is installed without any detailed configuration file. While this works, it may not be optimal for your use.
So, we'll start there. The config file for MySQL is called "my.cnf". On linux servers, it is typically located in the /etc directory. If you are unsure of where yours is located, at a command prompt type "locate my.cnf". If you are on windows, you can edit the file named my.ini in the C:\WINNT or C:\Windows directory.
If you open that file and see it is almost empty, then you have a lot of different options. Typically, the MySQL installation directory will have some example my.cnf files for you to use.
my-small.cnf � For systems with less than 64MB of RAM, where MySQL is used occasionally.
my-medium.cnf � For systems with less than 64MB of RAM, where MySQL is the primary activity on the system, or for systems with up to 128MB of RAM, where MySQL shares the box with other processes. This is the most common configuration, where MySQL is installed on the same box as a Web server and receives a moderate amount of traffic.
my-large.cnf � For a system with 128MB to 512MB of RAM, where MySQL is the primary activity.
my-huge.cnf � For a system with 1GB to 2GB of RAM, where MySQL is the primary activity
These are excellent places to start. You can copy the contents of the file you choose to use into your existing my.cnf file, or just move and rename the appropriate file.
A word of warning, if your server hosts other websites, and is running a lot of httpd processes, you will probably not want to run the huge.cnf file, as almost all of your resources will be used by MySQL, possibly not leaving enough for HTTP.
So, you have filled your my.cnf file with goodies.. its time to put it into action. You will need to restart MySQL for these changes to take effect. That was the easy part, you may or may not notice much performance difference yet. So, to dig a little deeper, we need to analyze what mysql is doing, and how well it is performing.
For you command line junkies, you can find out some information by typing this command: mysqladmin extended-status That will display a lot of important information.
But, to make it easier on you I have attached a script that you can run on your server that will provide current server loads, current running processes, netstat information, the extended-status information from mysql, and some important calculations regarding mysql queries and cache.
So, when you run the script, what do you see? A lot of information that probably gives you a headache That's ok, we don't need to understand ALL of it to get the most out of MySQL.
There are two primary start-up parameters that will affect your system the most: key_buffer_size and table_cache. If you get only two server parameters correctly tuned, make sure they're these two!
The value of key_buffer_size is the size of the buffer used with indexes. The larger the buffer, the faster the SQL command will finish and a result will be returned. Try to find the fine line between finely tuned and over-optimized; you may have a key_buffer_size of 256MB on a system with 512MB of RAM, but any more than 256MB could cause degraded server performance.
A simple way to check the actual performance of the buffer is to examine four additional variables: key_read_requests, key_reads, key_write_requests, and key_writes.
Find the rows that look something like this (your values will differ):
If you divide the value of key_read by the value of key_reads_requests, the result should be less than 0.01. Also, if you divide the value of key_write by the value of key_writes_requests, the result should be less than 1. Using the values above, we have results of 0.000250479809834401 and 0.454032712915962 respectively, well within the acceptable parameters. To try to get these numbers even smaller, more tuning could occur by increasing the value of key_buffer_size, but these numbers would be fine to leave as they are.
The other important server parameter is table_cache, which is the number of open tables for all threads. The default is 64, but you may need to adjust this number. Look for a variable called open_tables in the output. If this number is large, the value of table_cache should be increased.
Each time MySQL accesses a table, it places it in the cache. If your system accesses many tables, it is faster to have these in the cache. A good way to see whether your system needs to increase this is to examine the value of open_tables at peak times (one of the extended status values, above). If you find it stays at the same value as your table_cache value, and then the number of opened_tables starts increasing, you should increase the table_cache if you have enough memory.
The sample configuration files use various combinations of key_buffer_size and table_cache, which you can use as a baseline for any modifications you need to make. Whenever you modify your configuration, you'll be restarting your server in order for changes to take effect, sometimes with no knowledge of the consequences of your changes. In this case, be sure to try your modifications in a development environment before rolling the changes into production.
Here it looks like the table cache has been set too high. The server has been up for ages, the number of opened_tables is relatively low, and the open_tables (remember we're checking at a peak time) is nowhere near what it could be.
Here, although the open_tables is maxed out, the number of open_tables is very low even though the server has been up for ages. There is probably not much benefit in upping the table_cache (this example comes from a development server).
Scenario 3 table_cache - 64 open_tables - 64 opened_tables - 13918 uptime - 33754 This table_cache is set too low. The open_tables is running at maximum, and the number of open_tables is high, even though the uptime is less than a day. If you have the memory, up the table_cache. One thing to note is that even if you only have 64 tables in your database, you may still have more open tables. MySQL, being multi-threaded, may be running many queries on the table at one time, and each of these will open a table.
Scenario 5 key_buffer_size - 16777216 (16M) Key_read_requests - 609601541 Key_reads - 46729832 The values in scenario 1 are looking healthy. The ratio of key_reads to key_read_requests should be as low as possible, no more than 1:100. In scenario 1 it is close to 1:10000. In scenario 2, it is shocking, about 1:15, and the key_buffer size should be increased to as much as the memory allows (you can see that RAM is the primary hardware upgrade you can do to improve your system).
Lastly, if you are using MySQL 4.x, a few new variables have been introduced that will help your performance tremendously:
You may want to drop query_cache_size to 16M if you have only 1GB of ram, and maybe even lower if you have less ram.
Again, these are just guidelines and rules of thumb to help you better get the most performance. Each machine is slightly different, and different values will apply. So you may need to really sit down and experiment to find the fine line between optimized and overkill.
I just ran the script and to be honest I have no idea what I am doing, but I am getting a bit concerned with the speed of the forum and wanted to see if this shows anything that would point the finger to the hosting company.
Here is the info I got:
Fri Jul 11 19:46:19 PDT 2003 7:46pm up 11 days, 4:10, 1 user, load average: 1.83, 3.65, 6.84 319 processes: 303 sleeping, 11 running, 5 zombie, 0 stopped Mem: 2582300K av, 2563864K used, 18436K free, 0K shrd, 185480K buff Swap: 4194192K av, 77868K used, 4116324K free 1885028K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 14707 nobody 16 0 1220 1220 828 R 14.9 0.0 0:00 top Http processes currently running = 95 Mysql processes currently running = 5 Netstat information summary 1 CLOSING 1 LAST_ACK 2 FIN_WAIT2 6 SYN_RECV 15 FIN_WAIT1 22 LISTEN 183 ESTABLISHED 1078 TIME_WAIT
Can someone decifer the above info to me?
P.S.- Jeremy, I have been playing on your site and man is it fast compared to mine. The truck club is growning a lot faster than expected, so it looks like I may have to get away from the $9.95 / month hosting and get you to help me out in transfering it
Well, the first part of your results, and somewhat important is the load average. 1.83 is the 1 minute average, 3.65 is the 5 min, and 6.84 is a 15 minute average. This is showing average to slightly high loads (hard to really say for sure without knowing server hardware)
So.. the first clue is that the server may be slightly overworked. The second clue, is you see 300+ processes running. This is typical for a busy server.. not terribly alarming.
Next on the list is memory. as you can see, only 18436K is free, This isn't a lot. When you run low on memory, things need to be stored on the hard drive... which is the swap. Reading and writing data to a disk is much slower than to ram.
So, that is the jist of what that part you posted is. Again, nothing terribly alarming or really sticks out, it just seems to be a kinda busy server, which may be part of your slowdown. The one load average of almost 7 is kinda high, so if the server frequently spikes like that, that may be part of the problem. Or, it could be just the host not having MySQL optimized as well as they should, or a number of things
well I edited and entered MYSQL user and password and this is all I get when I run the script.
Sat Jul 12 09:53:01 EDT 2003 Http processes currently running = 5 Mysql processes currently running = 6 Netstat information summary
I do not see where to set a path maybe mysql is dieing from improper path. I know the login is correct as I can telnet in no problem and run mysqladmin extended-status however I have to set the path first
[]Sat Jul 12 09:53:01 EDT 2003 Http processes currently running = 5 Mysql processes currently running = 6 Netstat information summary [/]
So, like you mentioned, it looks like it is running netstat, http, and mysql fine.. but nothing after Top.
Can you just type top at the command line without a path?
EDIT: Actually, netstat isn't running either, since that would be listing some information as well. So it looks like just the ps -auxww commands are running, you may need paths for everything.
bash2.04$ whereis top bash2.04$ whereis ps bash2.04$ whereis netstat
It looks to me as though your environment isn't setup well, no paths. We should probably make another thread soemwhere on customizing your environment.
I have a P4 2.4 with 1GB RAM dedicated server 100Mbps net connection
It runs WHM 8.8.0 cPanel 8.8.0-R73 RedHat 9 - WHM X v2.1.2 apache (1.3.29 (Unix)) mysql (4.0.15-standard)
This machine is entirely use for just 1 domain with a vBulletin board. The site has about somewhere from 200+ to 800+ users online concurrent depends on time of the day.
When the number of users about 300- (less than 300) the server seem to react normally (Not Slow)
However, when the number of users increase at night (7mp - 5am PST) the server react extreamly slow (I mean really really slow) and it sometimes refuse to dislay page and not to load images as well.
1. Is this because of MySQL not tuned or Apache not Turned? How do I go about finding what need to fix?
2. I have include the 2 results (when site not busy & busy) of the script Germy provide, with the numbers shown, How I optimize MySQL to be the most efficient for my site?
3. With the hardware I have how many users I could have on the site just to browse the vBulletin board?
I have a P4 2.4 with 1GB RAM dedicated server 100Mbps net connection
It runs WHM 8.8.0 cPanel 8.8.0-R73 RedHat 9 - WHM X v2.1.2 apache (1.3.29 (Unix)) mysql (4.0.15-standard)
This machine is entirely use for just 1 domain with a vBulletin board. The site has about somewhere from 200+ to 800+ users online concurrent depends on time of the day.
When the number of users about 300- (less than 300) the server seem to react normally (Not Slow)
However, when the number of users increase at night (7mp - 5am PST) the server react extreamly slow (I mean really really slow) and it sometimes refuse to dislay page and not to load images as well.
1. Is this because of MySQL not tuned or Apache not Turned? How do I go about finding what need to fix?
2. I have include the 2 results (when site not busy & busy) of the script Jeremy provide, with the numbers shown, How I optimize MySQL to be the most efficient for my site?
3. With the hardware I have how many users I could have on the site just to browse the vBulletin board?
Thank you in advance for helping.
========================== Result when site not busy:
As this is a UBB Threads site, the tuning recommendations made here are with UBB Threads in mind. You may find information more usefull to your needs on a vBulliten sight, as they would be tuning thier systems to run best for vBulliten.
That being said, and a disclaimer that all in all, I know jack about MySQL: This looks like a problem to me and is where I would start: 26 zombie processes. That number seems way too high to me.
I haven't had time to read a lot of of that, but I caught the zombie processes bit, and I agree, that is a problem. On one of my dedicated threads servers, I can have about 300 processes, server loads under 1, and 0 zombie processes.
So IMO, you have way too many zombie processes for one. I don't know if that is ultimately your problem, but a properly setup server and/or software should not lead to that many zombie (defunct) processes.
I'll try to take a look at all the numbers and stats you posted earlier.. but in the meantime, can you say what OS you are using, kernel version, etc?
Seems like a certain script or something is eating up all of your resources, and most likely it takes about 2 hours for this to happen. Probably whatever is causing all of the zombie processes.
Since you are using cPanel/WHM, login to WHM and check this:
On the left menu, under the "Server Status" section, click on CPU/Memory/MySQL Usage
Take a look and see what user is using the most resources (whether it be mysql itself or your one vbulletin account) and it should tell you the scripts or commands that are using the most resources.
Otherwise from your info, nothing is terribly out of whack, other than what looks to be a script bombing out, creating a zombie, and this happening over and over again until the system comes to a grinding halt.
Well, this is my first attempt at tuning MySQL so i thought I'd take the smart approach of asking a few questions before attempting to modify anything on my server...
1. My key 'reads : read requests' & 'write : write requests' ratios are fine at 0.000697 and 0.613655 respectively, so that's taken care of I guess.
2. In the next bit I'm not sure what my table_cache is, but judging by the open_tables stat as well as an almost blank "my.cnf" file I assume that it's 64.
Using Jeremy's script I also see info like "100% of table_cache in use" and "(Cache hit = 99.999303%)".
2. How can up up my cache to 128, and how will this affect my system memory (I've got 1GB RAM)? What do I need to add to the "my.cnf" file, which is very bare at the moment (just 3 lines!):
3. Also, is there a procedure to follow when restarting MySQL? Or do I just switch the forums off and restart MySQL in WHM?
4. Jeremy also mentions "key_buffer_size". Where can I find out what it's set to? If the "my.cnf" file is just the three lines listed above, then does that mean that I have no key_buffer_size?
5. Last but not least, how do I protect the script from getting into the wrong hands (after all, I don't want to make my db login+password public...)? Placing it above the web root probably makes no sense, so am I doing the right thing by creating a new folder for this file and then setting a password on the folder via CPanel? Is that secure enough?
2. How can up up my cache to 128, and how will this affect my system memory (I've got 1GB RAM)? What do I need to add to the "my.cnf" file, which is very bare at the moment (just 3 lines!):
[mysqld] set-variable = max_connections=500 safe-show-database [/] Just add the variables that aren't defined yet to the configuration file. So inserting: table_cache = 128
will set your table cache to 128. Same thing for key_buffer_size and all other thigs mentioned in Jeremy's text. If a variable isn't set in the config file a default value is used.
[] 3. Also, is there a procedure to follow when restarting MySQL? Or do I just switch the forums off and restart MySQL in WHM? [/] Yes, close the board and restart mysql.
[] 5. Last but not least, how do I protect the script from getting into the wrong hands (after all, I don't want to make my db login+password public...)? Placing it above the web root probably makes no sense, so am I doing the right thing by creating a new folder for this file and then setting a password on the folder via CPanel? Is that secure enough? [/] Yes, that's fine.
Conrad, with 1GB RAM you can start table_cache at 512, and yes just add it on the next line in my.cnf: set-variable = table_cache=512. Then on the line after that: set-variable = key_buffer=256M.
Close forums, start and stop MySQL, see what happens. You may need to tweak them up or down depending on the load on your server.
I've been tweaking my my.cnf file, first attempt. Judging from details gathered, most indicators are fine.
However, now my site is busy, there are two indicators that aren't so good:
Open_tables | 512 | 100% of table_cache in use
I can just up that figure, I guess.
But this one: Key_read_requests | 92766222 | | Key_reads | 40777
That's 4.39. WAY over the recommendation here.
Which area of the cnf file should I target to bring this in control? Scenario 4 in the intial post seems to similar to this, but there's no elaboration on what it means.
Sounds like you're using persistant connections (as you have a ton of "sleeping" threads). I'd recommend that you set all of your MySQL scripts to NOT use persistant connections...
Perisstant connections sound cool in theory, that you can have a thread open and ready for users, HOWEVER, these threads STAY open, even after the session is termed... So available resources are out the window.
Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.
Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.