Site issues Wed. 9/11/13-Fri. 9/13/13

For reporting issues with the OCNA site.
Mr.Yuck
Site Admin
Posts:2161
Joined:Wed Aug 18, 2010 2:09 pm
Location:York County, Va.
Contact:
Site issues Wed. 9/11/13-Fri. 9/13/13

Post by Mr.Yuck » Fri Sep 13, 2013 6:06 pm

I don't know this is worth an "announcement". I posted about it to Twitter and Google+ on Wednesday night, when the site went down for the 2nd time. It went down 3 times total, the last being Friday morning.

The Issue: We had a corruption in the database. This caused error logs twice the size of the entire website itself to be written to the server! This filled up our disk on the server. Whenever the log files got big enough to fill up the disk, it would shut down the website. I would delete them, of course. The 4 people who get emails sent to octeam at Opencaching.us know there were about 150 error emails sent to that address from the API!! :lol:

I went right to the top, to the founder of opencaching.pl, wrygiel. He gave some advice, and eventually told me we should recreate the database from http://opencaching.pl/okapi/devel/dbstruct (It's a big file, but you can look at it). I told him I would need a little guidance, and he wrote back, "no problem, I did it for you"! Nice guy eh? Thanks to him. Oh, and of course it worked.

It was quite strange the very first day the site went down to see the disk full. That stuck out like a sore thumb, so I knew something was wrong. By the way, the disk on our server is about 1/3 full after 3 years and 1 month of existence. I think we can use that thing at least another 3 years and 1 month. That way we get to keep our amazingly cheap web hosting. It's certainly not free, but if you would have asked me before I became an admin, I would have guessed it was a lot more than it is. :P
ImageImage

Mr.Yuck
Site Admin
Posts:2161
Joined:Wed Aug 18, 2010 2:09 pm
Location:York County, Va.
Contact:

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Post by Mr.Yuck » Sat Sep 14, 2013 7:54 am

Oh boy. The twice the size of the website error log was there again Saturday morning, and the website down. Deleted and server rebooted. This log is only written once a day, so if we're lucky, it was just a remnant from the earlier problems. If we're not lucky, we're back to square one. Only time will tell, I suppose.

EDIT: After 2 error emails from the API, it's confirmed we're back to where we started. :mrgreen: <== That means I still have a fairly good attitude.
ImageImage

Mr.Yuck
Site Admin
Posts:2161
Joined:Wed Aug 18, 2010 2:09 pm
Location:York County, Va.
Contact:

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Post by Mr.Yuck » Sun Sep 15, 2013 6:27 am

Still happening on Sunday, but it doesn't seem as bad, with the huge error logs. By the way, I should probably give a heads up to our ISP, eh? Otherwise the web hosting bill will be about $1,000 next month. :o
ImageImage

Mr.Yuck
Site Admin
Posts:2161
Joined:Wed Aug 18, 2010 2:09 pm
Location:York County, Va.
Contact:

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Post by Mr.Yuck » Mon Sep 16, 2013 12:51 pm

Well that's good news, we can write all we want to our server, and it doesn't cost anything. And believe me, we're writing to it BIG TIME every day. :mrgreen: Wrygiel from opencaching.pl is actively working with us on the problem again.
ImageImage

User avatar
DudleyGrunt
Site Admin
Posts:2038
Joined:Wed Aug 18, 2010 1:27 pm
Location:Jessup, MD
Contact:

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Post by DudleyGrunt » Mon Sep 16, 2013 1:39 pm

Great news x 2.
Dave, OCNA Team Member
For the smiles, not the smilies.
Maryland Geocaching Society

Image Image

Mr.Yuck
Site Admin
Posts:2161
Joined:Wed Aug 18, 2010 2:09 pm
Location:York County, Va.
Contact:

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Post by Mr.Yuck » Fri Sep 20, 2013 7:42 pm

I have finally patched the error log writing problem as of Thursday morning, 9/19. It's been almost 48 hours, I think we're good there. We continue to receive 100+ API error emails per day at octeam at opencaching.us. There is one feature that doesn't work on this website, but I don't want to say what it is, because I'm of the opinion when people try it, it increases the size of the error log file. :) We need the folks at opencaching.pl to reinstall our database again, because I sure don't want to mess with it. Not without step by step idiot proof instructions, that is.
ImageImage

flopp
Posts:3
Joined:Thu Sep 30, 2010 4:20 am

Re: AW: Site issues Wed. 9/11/13-Fri. 9/13/13

Post by flopp » Sat Sep 21, 2013 3:05 am

Two of my sites are using OKAPI to retrieve geocaches from OCUS: http://www.flopp.net/okapi/ and http://www.flopp.net/

I can disable these requests for a while, if you think it helps.

Mr.Yuck
Site Admin
Posts:2161
Joined:Wed Aug 18, 2010 2:09 pm
Location:York County, Va.
Contact:

Re: AW: Site issues Wed. 9/11/13-Fri. 9/13/13

Post by Mr.Yuck » Sat Sep 21, 2013 7:09 am

flopp wrote:Two of my sites are using OKAPI to retrieve geocaches from OCUS: http://www.flopp.net/okapi/ and http://www.flopp.net/

I can disable these requests for a while, if you think it helps.
Interesting, I had no idea. :D Yes, if it's not a problem, disable them. I certainly can't hurt.
ImageImage

Mr.Yuck
Site Admin
Posts:2161
Joined:Wed Aug 18, 2010 2:09 pm
Location:York County, Va.
Contact:

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Post by Mr.Yuck » Sat Sep 21, 2013 6:00 pm

Ok, new development here. Opencaching.org.uk is the only other opencaching site that uses the original Opencaching.pl code like ourselves. Opencaching.pl (of course) and Opencaching.nl use the "new" OCPL code. I wrote to Paul from Opencaching.org.uk, as I noticed his site was down on Friday, 9/20, and asked him if he had error log problems. Although he didn't think so originally, his site went down again today, and yup, huge error logs. :o So there, I didn't do anything to mess up our website. :lol: I've mentioned this to DG a little, but let this be the first public acknowledgement that we are strongly considering moving to the new OCPL code at OCNA.
ImageImage

User avatar
DudleyGrunt
Site Admin
Posts:2038
Joined:Wed Aug 18, 2010 1:27 pm
Location:Jessup, MD
Contact:

Re: Site issues Wed. 9/11/13-Fri. 9/13/13

Post by DudleyGrunt » Sat Sep 21, 2013 8:27 pm

Yeah, think that'll be a good idea. Other than helping with the log issue, do you know of any other specific new features that come with the new code?
Dave, OCNA Team Member
For the smiles, not the smilies.
Maryland Geocaching Society

Image Image

Post Reply