Page 1 of 2

Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Fri Sep 13, 2013 6:06 pm
by Mr.Yuck
I don't know this is worth an "announcement". I posted about it to Twitter and Google+ on Wednesday night, when the site went down for the 2nd time. It went down 3 times total, the last being Friday morning.

The Issue: We had a corruption in the database. This caused error logs twice the size of the entire website itself to be written to the server! This filled up our disk on the server. Whenever the log files got big enough to fill up the disk, it would shut down the website. I would delete them, of course. The 4 people who get emails sent to octeam at Opencaching.us know there were about 150 error emails sent to that address from the API!! :lol:

I went right to the top, to the founder of opencaching.pl, wrygiel. He gave some advice, and eventually told me we should recreate the database from http://opencaching.pl/okapi/devel/dbstruct (It's a big file, but you can look at it). I told him I would need a little guidance, and he wrote back, "no problem, I did it for you"! Nice guy eh? Thanks to him. Oh, and of course it worked.

It was quite strange the very first day the site went down to see the disk full. That stuck out like a sore thumb, so I knew something was wrong. By the way, the disk on our server is about 1/3 full after 3 years and 1 month of existence. I think we can use that thing at least another 3 years and 1 month. That way we get to keep our amazingly cheap web hosting. It's certainly not free, but if you would have asked me before I became an admin, I would have guessed it was a lot more than it is. :P

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Sat Sep 14, 2013 7:54 am
by Mr.Yuck
Oh boy. The twice the size of the website error log was there again Saturday morning, and the website down. Deleted and server rebooted. This log is only written once a day, so if we're lucky, it was just a remnant from the earlier problems. If we're not lucky, we're back to square one. Only time will tell, I suppose.

EDIT: After 2 error emails from the API, it's confirmed we're back to where we started. :mrgreen: <== That means I still have a fairly good attitude.

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Sun Sep 15, 2013 6:27 am
by Mr.Yuck
Still happening on Sunday, but it doesn't seem as bad, with the huge error logs. By the way, I should probably give a heads up to our ISP, eh? Otherwise the web hosting bill will be about $1,000 next month. :o

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Mon Sep 16, 2013 12:51 pm
by Mr.Yuck
Well that's good news, we can write all we want to our server, and it doesn't cost anything. And believe me, we're writing to it BIG TIME every day. :mrgreen: Wrygiel from opencaching.pl is actively working with us on the problem again.

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Mon Sep 16, 2013 1:39 pm
by DudleyGrunt
Great news x 2.

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Fri Sep 20, 2013 7:42 pm
by Mr.Yuck
I have finally patched the error log writing problem as of Thursday morning, 9/19. It's been almost 48 hours, I think we're good there. We continue to receive 100+ API error emails per day at octeam at opencaching.us. There is one feature that doesn't work on this website, but I don't want to say what it is, because I'm of the opinion when people try it, it increases the size of the error log file. :) We need the folks at opencaching.pl to reinstall our database again, because I sure don't want to mess with it. Not without step by step idiot proof instructions, that is.

Re: AW: Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Sat Sep 21, 2013 3:05 am
by flopp
Two of my sites are using OKAPI to retrieve geocaches from OCUS: http://www.flopp.net/okapi/ and http://www.flopp.net/

I can disable these requests for a while, if you think it helps.

Re: AW: Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Sat Sep 21, 2013 7:09 am
by Mr.Yuck
flopp wrote:Two of my sites are using OKAPI to retrieve geocaches from OCUS: http://www.flopp.net/okapi/ and http://www.flopp.net/

I can disable these requests for a while, if you think it helps.
Interesting, I had no idea. :D Yes, if it's not a problem, disable them. I certainly can't hurt.

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Sat Sep 21, 2013 6:00 pm
by Mr.Yuck
Ok, new development here. Opencaching.org.uk is the only other opencaching site that uses the original Opencaching.pl code like ourselves. Opencaching.pl (of course) and Opencaching.nl use the "new" OCPL code. I wrote to Paul from Opencaching.org.uk, as I noticed his site was down on Friday, 9/20, and asked him if he had error log problems. Although he didn't think so originally, his site went down again today, and yup, huge error logs. :o So there, I didn't do anything to mess up our website. :lol: I've mentioned this to DG a little, but let this be the first public acknowledgement that we are strongly considering moving to the new OCPL code at OCNA.

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Posted: Sat Sep 21, 2013 8:27 pm
by DudleyGrunt
Yeah, think that'll be a good idea. Other than helping with the log issue, do you know of any other specific new features that come with the new code?