Panic Mode On (90) Server Problems?

Message boards : Number crunching : Panic Mode On (90) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 24 · Next

AuthorMessage
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11414
Credit: 29,581,041
RAC: 66
United States
Message 1580242 - Posted: 30 Sep 2014, 21:47:36 UTC

All AP channels have been split and 688 MB channels to split. It's gonna be a long time before fresh APs arrive.
ID: 1580242 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1580248 - Posted: 30 Sep 2014, 22:02:04 UTC - in response to Message 1580242.  

All AP channels have been split and 688 MB channels to split. It's gonna be a long time before fresh APs arrive.

Well there are still 3 channels being split t the moment, but they should be done shortly.
Given that there are normally 400-450 MB channels left to do when AP is finished. A few hundred more should not take very long to whittle down. Given that more and more machines will switch over to do MB as their cache of AP run out. Without a maintenance cycle & catchup the MB might get burned through a bit faster than normal too.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1580248 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1580310 - Posted: 1 Oct 2014, 0:28:40 UTC

Looks like the AP validator has gone OTL. Looking at the haveland stats it looks like it did this last year at the September/October transition as well. That is probably just an odd coincidence, but kind of amusing.
Let's see if the auto server recovery bits kick in and it fixes itself while everyone it out.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1580310 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36656
Credit: 261,360,520
RAC: 489
Australia
Message 1580388 - Posted: 1 Oct 2014, 7:34:20 UTC
Last modified: 1 Oct 2014, 7:34:44 UTC

I wouldn't be to sure about that HAL as my 2500K has gone way down in AP pendings while my 3570K has way gone up since last night. ;-)

Cheers.
ID: 1580388 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1580458 - Posted: 1 Oct 2014, 16:20:35 UTC - in response to Message 1580388.  

And we're back...So I guess it was just a delay of 1 day for the weekly maintenance.
ID: 1580458 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1580460 - Posted: 1 Oct 2014, 16:21:21 UTC - in response to Message 1580388.  
Last modified: 1 Oct 2014, 16:34:46 UTC

I wouldn't be to sure about that HAL as my 2500K has gone way down in AP pendings while my 3570K has way gone up since last night. ;-)

Cheers.

My MB pending to valid is still around 1:1, but my AP have shot up to about 4:1. With the # of valid dropping to about 40% of the normal number.

There are some tasks that are getting validated. So perhaps the validator is bouncing off/on.

And I just had ~100 go valid all at once.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1580460 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1580526 - Posted: 1 Oct 2014, 18:09:43 UTC - in response to Message 1580460.  

It sure would be nice if someone could get ap validate 4 working again. About 200 of those ~20,000 AP Workunits waiting for validation are mine. Since last night two of my machines have only had a couple APs validate.
ID: 1580526 · Report as offensive
Profile Oz
Avatar

Send message
Joined: 6 Jun 99
Posts: 233
Credit: 200,655,462
RAC: 212
United States
Message 1580733 - Posted: 1 Oct 2014, 23:59:06 UTC

Valid - 8
Validation pending - 170
wow.
Member of the 20 Year Club



ID: 1580733 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1580736 - Posted: 2 Oct 2014, 0:10:31 UTC - in response to Message 1580526.  
Last modified: 2 Oct 2014, 0:15:22 UTC

Well, it's been about a day since the validations stopped. The AP Workunits waiting for validation is now up to 29,130.
My "Validation pending" is up to 317 APs. 200+ APs is a lot of points. This is what it looks like when the validations stop for a day;

The bad part is, I made 2nd place with that daily total, http://stats.free-dc.org/stats.php?page=team&proj=sah&team=145273
not good...
ID: 1580736 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1580748 - Posted: 2 Oct 2014, 0:28:47 UTC - in response to Message 1580736.  

Well, it's been about a day since the validations stopped. The AP Workunits waiting for validation is now up to 29,130.
My "Validation pending" is up to 317 APs. 200+ APs is a lot of points. This is what it looks like when the validations stop for a day;
[img ]http://stats.free-dc.org/charts/hostdaily.php?proj=sah&hostid=6796479[/img]
The bad part is, I made 2nd place with that daily total, http://stats.free-dc.org/stats.php?page=team&proj=sah&team=145273
not good...

The stats were also generated several hours earlier than normal today. So that would just compound the issue of a really low stat day.

Once the validator starts working correctly again there will be an equally large stat day and throw the averages off again for a while. I kind of hope it takes a little while to get going again. I have an old top stat day of 840,794 from 2010 I an hoping to beat.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1580748 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11414
Credit: 29,581,041
RAC: 66
United States
Message 1580755 - Posted: 2 Oct 2014, 0:53:37 UTC

Yep APs are not validating and it will be a long time before they are split again.
ID: 1580755 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1580813 - Posted: 2 Oct 2014, 4:33:38 UTC - in response to Message 1577814.  

And he's just fixed it. I sincerely hope that was the last batch.

I don't know if you're still collecting these, but I just stumbled into the Science Status Page (when I was aiming for the Server Status Page) and got

Notice: Undefined variable: prefix in /disks/carolyn/b/home/boincadm/projects/sah/html/seti_boinc_html/sci_status.php on line 92 

along with
Notice: A non well formed numeric value encountered in /disks/carolyn/b/home/boincadm/projects/sah/html/seti_boinc_html/sci_status.php on line 201

which is repeated for 12 additional line numbers.

Just thought I'd pass it along!
ID: 1580813 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13851
Credit: 208,696,464
RAC: 304
Australia
Message 1580866 - Posted: 2 Oct 2014, 8:38:39 UTC - in response to Message 1580813.  

AP validators still deceased.
Grant
Darwin NT
ID: 1580866 · Report as offensive
Profile Oz
Avatar

Send message
Joined: 6 Jun 99
Posts: 233
Credit: 200,655,462
RAC: 212
United States
Message 1580912 - Posted: 2 Oct 2014, 12:59:51 UTC - in response to Message 1580748.  

Well, it's been about a day since the validations stopped. The AP Workunits waiting for validation is now up to 29,130.
My "Validation pending" is up to 317 APs. 200+ APs is a lot of points. This is what it looks like when the validations stop for a day;
[img ]http://stats.free-dc.org/charts/hostdaily.php?proj=sah&hostid=6796479[/img]
The bad part is, I made 2nd place with that daily total, http://stats.free-dc.org/stats.php?page=team&proj=sah&team=145273
not good...

The stats were also generated several hours earlier than normal today. So that would just compound the issue of a really low stat day.

Once the validator starts working correctly again there will be an equally large stat day and throw the averages off again for a while. I kind of hope it takes a little while to get going again. I have an old top stat day of 840,794 from 2010 I an hoping to beat.



I looked at the server status page, AP's awaiting validation is currently 291,250. Then I checked my account totals - valid 77, validation pending 1043. You are too right, there will be quite the bump when the validators catch up.
Member of the 20 Year Club



ID: 1580912 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1580929 - Posted: 2 Oct 2014, 13:33:21 UTC - in response to Message 1580912.  

Well, it's been about a day since the validations stopped. The AP Workunits waiting for validation is now up to 29,130.
My "Validation pending" is up to 317 APs. 200+ APs is a lot of points. This is what it looks like when the validations stop for a day;
[img ]http://stats.free-dc.org/charts/hostdaily.php?proj=sah&hostid=6796479[/img]
The bad part is, I made 2nd place with that daily total, http://stats.free-dc.org/stats.php?page=team&proj=sah&team=145273
not good...

The stats were also generated several hours earlier than normal today. So that would just compound the issue of a really low stat day.

Once the validator starts working correctly again there will be an equally large stat day and throw the averages off again for a while. I kind of hope it takes a little while to get going again. I have an old top stat day of 840,794 from 2010 I an hoping to beat.



I looked at the server status page, AP's awaiting validation is currently 291,250. Then I checked my account totals - valid 77, validation pending 1043. You are too right, there will be quite the bump when the validators catch up.

I was mostly looking at the value for Workunits waiting for validation. Normally this is a fairly low number < 100, but we are at ~45,000 & probably still rising. MB on the other hand is ~50.
I didn't pay much attention to Results returned and awaiting validation. As it is normally a pretty high value in the 100-200K range, but looking at the graph it took off at the same time.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1580929 · Report as offensive
Darth Beaver Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 20 Aug 99
Posts: 6728
Credit: 21,443,075
RAC: 3
Australia
Message 1580952 - Posted: 2 Oct 2014, 14:21:22 UTC

Validators servers are back up and working but 293,358 now aproxx 2,000 more than 1:20 hrs ago so all not well in SETIland still no wonder i only got 7,000 so far should be 20,000 oh well time for bed then
ID: 1580952 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1580957 - Posted: 2 Oct 2014, 14:48:59 UTC - in response to Message 1580952.  

Validators servers are back up and working but 293,358 now aproxx 2,000 more than 1:20 hrs ago so all not well in SETIland still no wonder i only got 7,000 so far should be 20,000 oh well time for bed then

The apparent only AP validator ap validate 4 has said running since yesterday, but the column on the left displaying the status is known to lie.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1580957 · Report as offensive
Profile ivan
Volunteer tester
Avatar

Send message
Joined: 5 Mar 01
Posts: 783
Credit: 348,560,338
RAC: 223
United Kingdom
Message 1580980 - Posted: 2 Oct 2014, 15:46:30 UTC - in response to Message 1580957.  
Last modified: 2 Oct 2014, 15:46:59 UTC

Validators servers are back up and working but 293,358 now aproxx 2,000 more than 1:20 hrs ago so all not well in SETIland still no wonder i only got 7,000 so far should be 20,000 oh well time for bed then

The apparent only AP validator ap validate 4 has said running since yesterday, but the column on the left displaying the status is known to lie.

The Astropulse "awaiting validation" graphs still show a nearly-constant rate of increase, with no sign of anything reducing the pool.
ID: 1580980 · Report as offensive
David S
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 18352
Credit: 27,761,924
RAC: 12
United States
Message 1581013 - Posted: 2 Oct 2014, 16:44:14 UTC

The crickets continue to fascinate. Never bottomed out during the outage yesterday, and there's a step up today that I thought must mean APs were back.
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

ID: 1581013 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11414
Credit: 29,581,041
RAC: 66
United States
Message 1581017 - Posted: 2 Oct 2014, 16:47:30 UTC - in response to Message 1580986.  

Mmmmm, hundreds of pending tasks now that the AP validator caught Ebola. Pendings are like money in the bank, great stuff for the future :-)

They don't get better with age.
ID: 1581017 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 24 · Next

Message boards : Number crunching : Panic Mode On (90) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.