Site issues Wed. 9/11/13-Fri. 9/13/13
Posted: Fri Sep 13, 2013 6:06 pm
I don't know this is worth an "announcement". I posted about it to Twitter and Google+ on Wednesday night, when the site went down for the 2nd time. It went down 3 times total, the last being Friday morning.
The Issue: We had a corruption in the database. This caused error logs twice the size of the entire website itself to be written to the server! This filled up our disk on the server. Whenever the log files got big enough to fill up the disk, it would shut down the website. I would delete them, of course. The 4 people who get emails sent to octeam at Opencaching.us know there were about 150 error emails sent to that address from the API!!
I went right to the top, to the founder of opencaching.pl, wrygiel. He gave some advice, and eventually told me we should recreate the database from http://opencaching.pl/okapi/devel/dbstruct (It's a big file, but you can look at it). I told him I would need a little guidance, and he wrote back, "no problem, I did it for you"! Nice guy eh? Thanks to him. Oh, and of course it worked.
It was quite strange the very first day the site went down to see the disk full. That stuck out like a sore thumb, so I knew something was wrong. By the way, the disk on our server is about 1/3 full after 3 years and 1 month of existence. I think we can use that thing at least another 3 years and 1 month. That way we get to keep our amazingly cheap web hosting. It's certainly not free, but if you would have asked me before I became an admin, I would have guessed it was a lot more than it is.
The Issue: We had a corruption in the database. This caused error logs twice the size of the entire website itself to be written to the server! This filled up our disk on the server. Whenever the log files got big enough to fill up the disk, it would shut down the website. I would delete them, of course. The 4 people who get emails sent to octeam at Opencaching.us know there were about 150 error emails sent to that address from the API!!
I went right to the top, to the founder of opencaching.pl, wrygiel. He gave some advice, and eventually told me we should recreate the database from http://opencaching.pl/okapi/devel/dbstruct (It's a big file, but you can look at it). I told him I would need a little guidance, and he wrote back, "no problem, I did it for you"! Nice guy eh? Thanks to him. Oh, and of course it worked.
It was quite strange the very first day the site went down to see the disk full. That stuck out like a sore thumb, so I knew something was wrong. By the way, the disk on our server is about 1/3 full after 3 years and 1 month of existence. I think we can use that thing at least another 3 years and 1 month. That way we get to keep our amazingly cheap web hosting. It's certainly not free, but if you would have asked me before I became an admin, I would have guessed it was a lot more than it is.