Well, at least we know now it's not IIP causing the problem. <img src="/forum/images/graemlins/smile.gif" alt="" /> Got to run into work, will check back later <img src="/forum/images/graemlins/smile.gif" alt="" />

[]Astaran said:<br />Have you thought of changing your apache configuration to limit the number of httpd processes and to change the timeout for http processes to a lower value?<br /><br />Maybe logging the slow mysql queries would help to determine the bottleneck also. [/]<br /><br />They mentioned this earlier but I'll pass your comments on <img src="/forum/images/graemlins/smile.gif" alt="" />

[]AllenAyres said:<br />[]Duncan said:<br />Hosts:<br />PLESK PSA 5.0.5<br />Apache: 1.3.27<br />MySQL: 3.23.55<br /><br />Some people are going to probably tell you that those are not the latest builds, but these are the latest builds that PLESK uses and they are all stable and secure."<br /> [/]<br /><br />There's a whole page of bugfixes and security fixes for 3.23.56 released more than 2 months ago:<br />http://www.mysql.com/doc/en/News-3.23.56.html - including some issues with database corruption. Just because it's what's in the released version of plesk doesn't mean it's the best option for you... you are the one paying the bill. [/]<br /><br />Duncan is on Standard version of Plesk, and plesk are not going to be supporting this much longer - at the moment they have not upgraded MySQL.<br /><br />Duncan, I would strongly consider switching machines, and having an RPM version of Plesk - believe me you will not want a standard version.

Some snips of info from my hosts which help people find what this problem is (Ive edited out the bits that aren't of much info):<br /><br />DPrincetonNOC [12:49]: i show the load of the server 15 min ago was 65 _ <br />DPrincetonNOC [12:49]: 65 + <br />DPrincetonNOC [12:50]: last pid: 16163; load averages: 1.32, 11.55, 65.54442 up 0+19:55:30 11:50:31 <br />384 processes: 80 running, 304 sleeping <br />CPU states: 1.9% user, 0.0% nice, 7.1% system, 0.4% interrupt, 90.7% idle <br />Mem: 154M Active, 10M Inact, 74M Wired, 788K Cache, 34M Buf, 644K Free <br />Swap: 480M Total, 458M Used, 23M Free, 95% Inuse, 616K In, 1336K Out <br /><br /> 8498 root 29 0 2352K 792K RUN 0:53 3.63% 1.07% top <br />7821 root 29 0 2352K 792K RUN 0:56 3.30% 0.98% top <br /> 187 root 2 0 10040K 1536K select 0:22 0.00% 0.00% httpd <br />15306 apache -14 0 11616K 2128K inode 0:04 0.00% 0.00% httpd <br />15396 apache 28 0 11608K 2120K RUN 0:04 0.00% 0.00% httpd <br />15305 apache 28 0 11692K 1896K RUN 0:03 0.00% 0.00% httpd <br />15477 apache -14 0 12920K 2664K inode 0:03 0.00% 0.00% httpd <br />15278 apache -14 0 11540K 2096K inode 0:03 0.00% 0.00% httpd <br />15294 apache 28 0 11704K 2212K RUN 0:03 0.00% 0.00% httpd <br />15234 apache -14 0 12948K 2596K inode 0:03 0.00% 0.00% httpd <br />15517 apache -14 0 11440K 2160K inode 0:03 0.00% 0.00% httpd <br />15443 apache 28 0 11512K 1772K RUN 0:03 0.00% 0.00% httpd <br />15465 apache 28 0 11496K 1972K RUN 0:03 0.00% 0.00% httpd <br />15476 apache -14 0 11532K 1892K inode 0:03 0.00% 0.00% httpd <br />15480 apache -14 0 11444K 2096K inode 0:03 0.00% 0.00% httpd <br />15464 apache 28 0 11508K 1692K RUN 0:03 0.00% 0.00% httpd <br /> 172 root 2 0 4304K 256K select 0:03 0.00% 0.00% httpsd <br /><br /><br />DPrincetonNOC [12:51]: there are about 500 httpd processes running right now <br />DPrincetonNOC [12:51]: thats why it died again.MrKopTalk [12:53]: hmmm....Could a news site that checks my site every 5 minutes for updates cause any of this? When I update a news item I edit a page which can be found here: http://www.koptalk.com/regulars/newsnow.shtml<br /><br />Those items then appear at:<br /><br />http://www.newsnow.co.uk/newsfeed/?name=Liverpool<br /><br />It was just a thought. <br />DPrincetonNOC [12:53]: that might be whats doing it <br />DPrincetonNOC [12:54]: im rebooting the box again now <br />MrKopTalk [12:54]: I'll remove that page so it cant spider the site <br />DPrincetonNOC [12:54]: i was never told this before so i wasnt loooking for anything like that in the logs <br />MrKopTalk [12:54]: i didnt know it could be that...just a wild guess <br />DPrincetonNOC [12:55]: spiders could be doing that and they create httpd requests and its not like a browser that someone closes <br />DPrincetonNOC [12:55]: and 5 min could be to short of an interval and the process never closes on its own. <br />DPrincetonNOC [12:55]: its taking much longer for the box to die now so its a slow thing <br />DPrincetonNOC [12:56]: as far as the mysql goes, Plesk knows about the patches and they have released hot fixes for PSA <br />DPrincetonNOC [12:56]: which we have applied. <br />DPrincetonNOC [12:56]: no need to worry about that. <br />DPrincetonNOC [13:20]: its about every 10 hours that it goes <br />DPrincetonNOC [13:20]: there are still httpd spawns from the spider and they dont close so they just all add up <br />MrKopTalk [13:21]: so you think this spider thing every 5 mins could the prob? i can soon work around that as I dont have to use it <br />DPrincetonNOC [13:21]: we will be able to get it to work if you can get some stable code from them <br />MrKopTalk [13:21]: when a new headline appears on newsnow.co.uk from my site people click on it and they are taken to my site via a pop-up <br />DPrincetonNOC [13:22]: see if they can make it every 10 min or something <br />MrKopTalk [13:22]: koptalk is the 3rd busiest site on there <br />DPrincetonNOC [13:22]: then if the server goes down every 20 hours we know it was that. <br />MrKopTalk [13:22]: i'll let you guys know what they say - it might not even be that <br />DPrincetonNOC [13:23]: I can format and reinstall and write some extra code into the kernal to allow 5,000 httpd connections at any time <br />DPrincetonNOC [13:23]: right now our custom kernal is set for 2,500 which hasnt ever been a problem for any other customers <br /><br />DPrincetonNOC [13:28]: how many were online last time you were on <br />MrKopTalk [13:28]: 500 <br />MrKopTalk [13:29]: maybe my site is too busy for Threads even if this problem is fixed? <br />DPrincetonNOC [13:29]: i noticed you still had that first page up, can you make it just go to the forum <br />DPrincetonNOC [13:29]: i doubt it <br /> <br />DPrincetonNOC [13:31]: well let me get this think rebooted again and see what I can see and then steve will work on it <br /><br />DPrincetonNOC [13:37]: ask allen if he can put a non beta version of threads on the box with the same index page, just create another DB with same content <br />DPrincetonNOC [13:38]: have the index point to the non beta version on the server and see if that still crashes the box.

The news site checks every 5 minutes and it seems like your problems start growing after 5 mins. It seems like there is a connection there. Hmmmm.

The things is though, only that one page which is on another server at koptalk.com (away from Threads) should be checked for changes etc

Just been thinking again...that page is spidered every 5 mins 24 hours a day so you would expect the server to crash at night time too here in the UK yet it has been going down at around 11am/12noon UK time.

well, I'd think 500+ online might have a little to do with server problems <img src="/forum/images/graemlins/wink.gif" alt="" />

Yeah, if you've already got 500 open processes with that many users - and then the spidering starts - <br /><br />I'd disable it at least - to see if you can rule it out.

Some times when I edit a post and re-post it, it says:<br /><br />[censored]Houllier[censored] [censored]reject[censored] [censored]linked[censored] [censored]with[censored] [censored]Barcelona[censored] <br /> #1580 - 30/05/2003 10:09 (80.194.222.99) Edit Reply Quote <br /> <br /> <br /><br />[censored]Former[censored] [censored]Liverpool[censored] '[censored]keeper[censored] [censored]David[censored] [censored]James[censored] [censored]has[censored] [censored]become[censored] [censored]a[censored] [censored]shock[censored] [censored]target[censored] [censored]for[censored] [censored]Spanish[censored] [censored]giants[censored] [censored]Barcelona[censored] [censored]claim[censored] [censored]various[censored] [censored]reports[censored] [censored]today[censored]. <[censored]br[censored] /> <[censored]br[censored] />[censored]James[censored], [censored]who[censored] [censored]has[censored] [censored]established[censored] [censored]himself[censored] [censored]as[censored] [censored]England[censored]'[censored]s[censored] [censored]number[censored] [censored]1[censored] [censored]goalkeeper[censored] [censored]and[censored] [censored]received[censored] [censored]rave[censored] [censored]reviews[censored] [censored]since[censored] [censored]Houllier[censored] [censored]axed[censored] [censored]him[censored], [censored]is[censored] [censored]reportedly[censored] "[censored]flattered[censored]" [censored]by[censored] [censored]their[censored] [censored]interest[censored]. <[censored]br[censored] /> <[censored]br[censored] />[censored]Since[censored] [censored]leaving[censored] [censored]Anfield[censored] [censored]James[censored] [censored]has[censored] [censored]also[censored] [censored]been[censored] [censored]linked[censored] [censored]various[censored] [censored]times[censored] [censored]with[censored] [censored]Manchester[censored] [censored]United[censored]. <[censored]br[censored] /> <[censored]br[censored] />[censored]Ironically[censored] [censored]Brad[censored] [censored]Friedel[censored], [censored]another[censored] [censored]Liverpool[censored] [censored]reject[censored], [censored]was[censored] [censored]voted[censored] [censored]the[censored] [censored]Premiership[censored]'[censored]s[censored] [censored]best[censored] [censored]goalkeeper[censored] [censored]last[censored] [censored]season[censored] [censored]while[censored] [censored]Sander[censored] [censored]Westerveld[censored] [censored]has[censored] [censored]been[censored] [censored]playing[censored] [censored]out[censored] [censored]of[censored] [censored]his[censored] [censored]skin[censored] [censored]with[censored] [censored]Real[censored] [censored]Sociedad[censored]. <[censored]br[censored] /> <[censored]br[censored] />[censored]The[censored] [censored]sooner[censored] [censored]Liverpool[censored] [censored]axe[censored] [censored]Joe[censored] [censored]Corrigan[censored] [censored]the[censored] [censored]better[censored]. <[censored]br[censored] /> <br /> <br /><br />What's that about then (no censored words are in it) <img src="/forum/images/graemlins/smile.gif" alt="" />

Check you bad words filter - you may have a space in it - if not resave the file, make sure it is saved, by adding another word to it <img src="/forum/images/graemlins/smile.gif" alt="" />

thanks...great fun this Threads <img src="/forum/images/graemlins/wink.gif" alt="" /><br /><br />(420 users online and increasing as the dreaded time period approaches....will it crash again today <img src="/forum/images/graemlins/smile.gif" alt="" /> )

The server was ok Friday, Saturday and Sunday which is when the site traffic is quiet. It crashed at 1pm approx UK time on Monday and today (Tuesday). Allen removed IIP and the beta so 6.2.3 is running. This is what my hosts said to me today when it crashed again:<br /><br />Duncan, <br /><br /> <br /><br />Here is a screen shot of TOP from when the box died. <br /><br /> <br /><br />last pid: 26035; load averages: 127.01, 107.43, 64.99 up 0+22:45:40 12:28:22 <br /><br />398 processes: 196 running, 201 sleeping, 1 zombie <br /><br />CPU states: 2.1% user, 0.0% nice, 3.7% system, 0.9% interrupt, 93.3% idle <br /><br />Mem: 156M Active, 7228K Inact, 75M Wired, 924K Cache, 34M Buf, 644K Free <br /><br />Swap: 544M Total, 514M Used, 30M Free, 94% Inuse, 318M In, 318M Out<br /><br /> PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND <br /><br /> 5745 root 28 0 2368K 636K RUN 1:25 1.03% 0.10% top <br /><br />25893 apache 28 0 11772K 1712K RUN 0:00 0.05% 0.05% httpd <br /><br /> 155 mysql -18 0 21532K 380K RUN 65:44 0.00% 0.00% mysqld <br /><br />15173 admin 2 0 2296K 0K RUN 0:58 0.00% 0.00% <top> <br /><br /> 188 root 2 0 10040K 1964K select 0:28 0.00% 0.00% httpd <br /><br /> 5740 admin 2 0 5292K 0K select 0:07 0.00% 0.00% <sshd> <br /><br />25039 apache 2 0 12684K 0K sbwait 0:04 0.00% 0.00% <httpd> <br /><br />24991 apache -18 0 11452K 2108K RUN 0:04 0.00% 0.00% httpd <br /><br />25073 apache -22 0 11512K 1984K swread 0:04 0.00% 0.00% httpd <br /><br />25014 apache -14 0 11452K 1944K inode 0:04 0.00% 0.00% httpd <br /><br />25076 apache -18 0 12692K 2604K RUN 0:04 0.00% 0.00% httpd <br /><br />25226 apache -14 0 11332K 2436K inode 0:03 0.00% 0.00% httpd <br /><br />25225 apache -22 0 11544K 2520K swread 0:03 0.00% 0.00% httpd <br /><br />25248 apache -18 0 11356K 1756K RUN 0:03 0.00% 0.00% httpd <br /><br />25199 apache -14 0 11092K 2176K inode 0:03 0.00% 0.00% httpd <br /><br /> 173 root 2 0 4304K 244K select 0:03 0.00% 0.00% httpsd <br /><br />25207 apache -18 0 11544K 1264K RUN 0:03 0.00% 0.00% httpd <br /><br />I dont see any reason that the load should be so high, except that php is spawning httpd so much and not closing. Were there any hacks applied to your site?<br /><br /><br />---<br /><br />Any ideas gents?<br /><br /><br />

It's just gone again, hosts say: "mysql died. it tried to dump it self. is that something that threads tells it to do?"

"its still doing httpd in free(): warning: recursive call"

re-booted and gone again within fifteen minutes or so:<br /><br /><br />last pid: 1208; load averages: 22.52, 16.12, 7.42 up 0+00:22:54 16:23:45 <br />328 processes: 5 running, 323 sleeping <br />CPU states: 2.9% user, 0.0% nice, 9.0% system, 1.8% interrupt, 86.4% idle <br />Mem: 157M Active, 22M Inact, 58M Wired, 608K Cache, 34M Buf, 644K Free <br />Swap: 1504M Total, 301M Used, 1204M Free, 19% Inuse, 364K In, 3316K Out <br /> <br /> PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND <br /> 944 root 36 0 2288K 728K RUN 0:07 4.14% 1.37% top <br /> 155 mysql 28 0 20504K 1268K pfault 1:35 0.00% 0.00% mysqld <br /> 526 apache 28 0 11420K 2368K pfault 0:04 0.00% 0.00% httpd <br /> 328 apache 2 0 13000K 0K sbwait 0:04 0.00% 0.00% <httpd> <br /> 333 apache 2 0 11784K 0K sbwait 0:04 0.00% 0.00% <httpd> <br /> 341 apache 28 0 11680K 2248K pfault 0:04 0.00% 0.00% httpd <br /> 343 apache 28 0 11544K 2276K pfault 0:04 0.00% 0.00% httpd <br /> 304 apache 2 0 11764K 0K sbwait 0:04 0.00% 0.00% <httpd> <br /> 336 apache 2 0 11884K 0K sbwait 0:04 0.00% 0.00% <httpd> <br /> 347 apache 2 0 11544K 0K sbwait 0:04 0.00% 0.00% <httpd> <br /> 318 apache 2 0 11548K 0K sbwait 0:04 0.00% 0.00% <httpd> <br /> 325 apache 28 0 11324K 1112K pfault 0:04 0.00% 0.00% httpd <br /> 219 apache 28 0 11472K 1388K pfault 0:04 0.00% 0.00% httpd <br /> 351 apache 28 0 11544K 1408K pfault 0:04 0.00% 0.00% httpd <br /> 332 apache 28 0 11544K 1836K pfault 0:04 0.00% 0.00% httpd <br /> 323 apache 2 0 12756K 0K sbwait 0:03 0.00% 0.00% <httpd> <br /> <br /> <br /> "something is spawning again"<br /><br />--<br /><br /><img src="/forum/images/graemlins/frown.gif" alt="" />

Hosts report that this is an Infopop problem in the software and that they can not debug software for them. They also sent this by email:<br /><br />Tue Jun 3 16:34:25 2003] [notice] suEXEC mechanism enabled <br />(wrapper: /usr/loca<br />l/psa/apache/bin/suexec)<br />[Tue Jun 3 16:34:25 2003] [notice] Accept mutex: flock (Default: flock)<br />[Tue Jun 3 16:35:10 2003] [warn] child process 425 still did not exit, <br />sending<br />a SIGTERM<br />[Tue Jun 3 16:35:10 2003] [warn] child process 426 still did not exit, <br />sending<br />a SIGTERM<br />[Tue Jun 3 16:35:10 2003] [warn] child process 427 still did not exit, <br />sending<br />a SIGTERM<br />[Tue Jun 3 16:35:14 2003] [notice] caught SIGTERM, shutting down<br /><br /><br />We can hint them towards perl and flock/spawn handling, people with out high <br />load would never have this problems, so they might not notice this stuff.

Well, threads is written in php and doesn't start child processes, so it can't apply to threads...<br /><br />Have you checked your cron jobs so that subscriptions isn't run every minute or something like that?<br /><br />A good thing to do is to set up /server-status and /server-info so that you can see what pages all these httpd processes are stuck at. That's how I managed to track down a problem with similar symptoms (which didn't have anything to do with threads).

The more you post, the less it sounds like a threads specific problem. I would almost guess it is a problem with the mysql. But I don't know enough to even make a good guess as to what though. He had said you we're hacked, but why did he think you might be? Hmmmm.

Maybe apache needs rebuilt?