False alarm, but please read: GTX 1660 Super, driver 441.12, Windows 7

Message boards : Number crunching : False alarm, but please read: GTX 1660 Super, driver 441.12, Windows 7
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14653
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2021556 - Posted: 3 Dec 2019, 16:10:05 UTC

Does the stock Cuda50 app work for you?

I activated my Beta account by mistake, and got a bunch of Arecibo VLAR tasks. It looks as if the app has stalled, and is displaying 'pseudo progress': 22 minutes elapsed, 79.232% done, zero CPU time since last checkpoint. I'll run a bench test later.
ID: 2021556 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 2021611 - Posted: 3 Dec 2019, 23:46:50 UTC - in response to Message 2021556.  

Does the stock Cuda50 app work for you?

I activated my Beta account by mistake, and got a bunch of Arecibo VLAR tasks. It looks as if the app has stalled, and is displaying 'pseudo progress': 22 minutes elapsed, 79.232% done, zero CPU time since last checkpoint. I'll run a bench test later.



I would swear over on the Windows thread as well as elsewhere there is a problem with Windows video drivers if they are super current. You have to backgrade a fair amount to get Nvidia to work on Windows....

Tom
A proud member of the OFA (Old Farts Association).
ID: 2021611 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34822
Credit: 261,360,520
RAC: 489
Australia
Message 2021612 - Posted: 3 Dec 2019, 23:51:04 UTC - in response to Message 2021611.  

I would swear over on the Windows thread as well as elsewhere there is a problem with Windows video drivers if they are super current. You have to backgrade a fair amount to get Nvidia to work on Windows....

Tom
That's a Win10 OpenCL problem Tom, Richard's problem is using an old Cuda app with Win7. ;-)

Cheers.
ID: 2021612 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14653
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2021663 - Posted: 4 Dec 2019, 14:05:16 UTC
Last modified: 4 Dec 2019, 14:05:53 UTC

As the post title will shortly say, false alarm.

I fumbled and got the CUDA 5.0 tasks while prepping my cache for the outage. By the time that started, the 1660 had snagged a GPUGrid task, and then it was time for bed.

But I've completed the tasks now, and all seems good - they resumed from checkpoints, completed and validated.

Task 35117108 ran on the 1660 Super in 1,772 seconds
Task 35116737 ran on a 750 Ti in 2,597 seconds

Both are from the same part of the same tape, and neither overflowed - so the extra runtime of roughly 50% seems a true measure of speed. By comparison, when running the SoG app, the speed ratio is closer to 3x, not 1.5x.

So, not only is the best-available CUDA app slower than SoG (we knew that): the CUDA app fails to scale proportionately on newer hardware, and frankly on the 1660 it's dire. That's what prompted my panic yesterday.

The threads that Tom alluded to remind us that NVidia's support for OpenCL is also pretty dire, as is their response time when problems are reported.

The CUDA v5.0 app is now very long in the tooth - it was tweaked slightly in 2016 to conform to SETI v8 standards, but the basic algorithms were left unchanged. At the moment, the SETI project doesn't have an active CUDA developer for Windows. I think we run a very real risk of not being able to use our NVidia cards on this project in the future, if a complete OpenCL breakdown coincides with an application upgrade to, say, process the Parkes data.

[At this point, somebody will mutter 'special sauce', but I will counter 'Linux only'. My interest is in apps usable by the many, not the few.]

The old Lunatics team has effectively disbanded, and Joe Segur, for one, is most certainly not coming back. So, time to hand the baton on to you youngsters. Anybody willing and able to develop the next generation of CUDA apps for SETI to stock application standards, or to nominate/recruit someone else who can?
ID: 2021663 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2021673 - Posted: 4 Dec 2019, 15:12:46 UTC - in response to Message 2021663.  

The threads that Tom alluded to remind us that NVidia's support for OpenCL is also pretty dire, as is their response time when problems are reported.


The problem only occurs on Windows 10; use windows 8 or 7 and everything works
The problem only occurs on the SoG app; use the sah app and everything works
The problem only occurs on a certain type of task; other task type work properly even with the SoG app

openCL does work. I won't be surprised if Nvidia decides the app needs an update, not their drivers.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2021673 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14653
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2021677 - Posted: 4 Dec 2019, 15:48:20 UTC - in response to Message 2021673.  

Windows 7 and 8 are 'end of life'.
The sah app wasn't deployed as stock, because of comparative inefficiency.
Neither the project, nor users, can choose the AR of data recorded at Arecibo.

Increasingly, as new users come along and bring new, 'off the shelf', hardware with them, Windows 10 will be the only version pre-installed for them. New GPU models will have a higher minimum driver requirement. If the project is offered new data sources, new application versions may be needed, and in that case every app will need to be re-written.

As I got older, I became more interested in reading about history, and seeing the parallels between errors that were made in the past, and errors that continue to be made today. I don't want SETI@Home to turn into another slice of history, because it failed to learn the necessary lesson of preparing for entirely foreseeable, nay inevitable, change.
ID: 2021677 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2021679 - Posted: 4 Dec 2019, 15:54:11 UTC

It is not all "wine and roses" over in the CUDA Developer forum. I see page after page of threads with issues with memory issues and FFT local array size issues with the latest drivers.
This goes to the heart of the same issue with the SoG application. The Nvidia response is "the issues are being worked on"
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2021679 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14653
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2021680 - Posted: 4 Dec 2019, 16:01:13 UTC - in response to Message 2021679.  

Which is why one lesson we could learn from history is 'keep one foot in both camps', and another is 'keep an eye open for new third, fourth, fifth options riding up over the horizon'. I get the impression that Apple is dropping both CUDA and OpenCL, though I don't have an immediate feel for what it replacing them.
ID: 2021680 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2021682 - Posted: 4 Dec 2019, 16:26:57 UTC - in response to Message 2021677.  
Last modified: 4 Dec 2019, 16:33:16 UTC

Windows 7 and 8 are 'end of life'.
The sah app wasn't deployed as stock, because of comparative inefficiency.
Neither the project, nor users, can choose the AR of data recorded at Arecibo.


I'm aware of this. I was just giving more context to the problem. it's not just that nvidia openCL is broken, it works in the majority of cases. it's possible they have made some other change to their driver handling, and do not want to revert it to placate one old (and probably obscure to them) app. at some point the applications need an update to accommodate new software/hardware techniques. I was just saying the problem solution may not rest solely on Nvidia's shoulders.

re: sah app inefficiency, looking at the results from This System on Beta which is running both the sah and SoG app on Win10, with "bad" drivers. it doesn't appear to be that much slower, only marginally.

Examples:
Arecibo VLAR (AR=.008604) sah app Crunch time = 811s
Arecibo VLAR (AR=.008885) SoG app Crunch time = 755s (7% faster)

Arecibo (AR=.802573) sah app Crunch time = 326s
Arecibo (AR=.802573) SoG app Crunch time = 293s (10% faster)

Arecibo VHAR (AR=19.863169) sah app
(SoG app can't process this one)

Personally I'd take an app that was 10% slower that works better with all tasks over the app that hangs for an hour waiting to time out when it gets tasks it doesn't like.

alternatively, the project does have some capability to decide to whom certain tasks are distributed. when Arecibo VLAR tasks were having issues with nvidia GPUs, they blocked them from going to them. The same could be done in this case. they may have to create a new classification for .vhar and append it to the WU file name like they do with .vlars

Don't send Arecibo VHAR tasks to Windows 10 systems with nvidia GPUs, or something like that. You'll ostracize the folks working around the problem with older drivers, or some people running an unaffected app, but you'll temporarily solve the problem with the least number of constraints.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2021682 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14653
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2021696 - Posted: 4 Dec 2019, 18:05:07 UTC - in response to Message 2021682.  

All of which are true, and fair points. I'm aware of the work-rounds: I've run many of them, and even written some. I'm running the special sauce on Linux: I prevented my Windows 7 machines automatically updating to Windows 10: for several years, I managed, maintained, and enhanced the Lunatics installer. But I'm not, myself, a qualified CUDA programmer, and I stopped maintaining the installer when the flow of new applications dried up, and the emphasis switched to yard-long command lines.

My point wasn't about that. I'm not interested in sticking band-aids on the SETI project until it looks like an Egyptian mummy. And runs about as fast. And as often.

I want the project to be looking forward, into the future. SETI@Home started as a pioneer of volunteer, distributed, computing. We could convert the whole thing into a data-center run by robots: we could continue as we are, with added band-aids: or we could anticipate the future and prepare for it.

Come that elusive day when ET finds us, I think she'll only respect us if we take the third way.
ID: 2021696 · Report as offensive
Profile TimeLord04
Volunteer tester
Avatar

Send message
Joined: 9 Mar 06
Posts: 21140
Credit: 33,933,039
RAC: 23
United States
Message 2021699 - Posted: 4 Dec 2019, 18:12:03 UTC - in response to Message 2021680.  

Which is why one lesson we could learn from history is 'keep one foot in both camps', and another is 'keep an eye open for new third, fourth, fifth options riding up over the horizon'. I get the impression that Apple is dropping both CUDA and OpenCL, though I don't have an immediate feel for what it replacing them.

Richard,

TBar and I have talked endlessly about issues I've had with both my Hackintosh and my REAL Mac Pro 5,1. In the course of our discussions, out came Mojave and now Catalina.

Mojave DEPRECATED OpenCL and OpenGL and is the absolute LAST OS that will run them. High Sierra is the LAST MacOS to run Maxwell and Pascal Cards, (NO Turing), AND is the LAST MacOS to run CUDA - per NVIDIA's release of news on November 22, 2019.

The future for Apple Computers seems to be in "Metal" and "Metal 2" processing. BOINC and SETI MUST find an Apple Programmer/Developer willing to Volunteer to develop Apps for Metal and Metal 2.

Due to my use of NVIDIA Maxwell and Pascal Cards in both of my MacOS Systems, I WILL remain on High Sierra for as long as I can. HOWEVER; I plan in the next three years to save up to get TWO MacVidCards' Radeon Sapphire Pulse RX-580 Cards one for each System. This way, I can move the Mac Pro up to Mojave and the Hackintosh will be free and clear to even go to Catalina - IF I choose to do so... Catalina is a WHOLE NEW ANIMAL, and I may decide NOT to go there on the Hackintosh... Too soon to tell, yet.

EVEN TBar says NOT to go to Catalina with his New OpenCL App. AND, his CUDA90 is BEST utilized in Sierra, NOT High Sierra - though it CAN run in High Sierra WITH ISSUES.

Good Luck to ALL the Mac Users out there. I hope that BOINC and SETI@Home CAN eventually pick up a good Mac Developer.


TL
TimeLord04
Have TARDIS, will travel...
Come along K-9!
Join Calm Chaos
ID: 2021699 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2021700 - Posted: 4 Dec 2019, 18:16:35 UTC - in response to Message 2021696.  

My point wasn't about that. I'm not interested in sticking band-aids on the SETI project until it looks like an Egyptian mummy. And runs about as fast. And as often.

I want the project to be looking forward, into the future. SETI@Home started as a pioneer of volunteer, distributed, computing. We could convert the whole thing into a data-center run by robots: we could continue as we are, with added band-aids: or we could anticipate the future and prepare for it.


Then I think the project should focus on recruiting the talent that can port the special sauce CUDA code to Windows.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2021700 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 2021702 - Posted: 4 Dec 2019, 18:43:01 UTC - in response to Message 2021700.  

Then I think the project should focus on recruiting the talent that can port the special sauce CUDA code to Windows.

Her's an idea from left field. How about a special fundraiser to hire one such as but not necessarily Jason Gee?
ID: 2021702 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 2021704 - Posted: 4 Dec 2019, 18:50:14 UTC - in response to Message 2021699.  

...EVEN TBar says NOT to go to Catalina with his New OpenCL App. AND, his CUDA90 is BEST utilized in Sierra, NOT High Sierra - though it CAN run in High Sierra WITH ISSUES....
I don't remember saying that, in fact, I believe I stated that OpenCL would probably work for a few more OS versions. Here's what it says at C.A., The Optimized CPU Apps Don't Work in Catalina 10.15, for Catalina use the ATi5-SoG_r4008 & CPU-Graphics package which has the Stock CPU App 8.05 with CPU Graphics The OpenCL Apps work fine in Catalina, it's the CPU Apps that have trouble. I wouldn't be surprised to see Apple make sure OpenCL is compatible with their OS for quite some time since they Own OpenCL. Just because something is listed as Deprecated doesn't mean it won't work, it means they prefer something else. I've seen code be deprecated for years and still work fine. Apple doesn't own CUDA, which is a direct competitor to OpenCL and Metal, so, I don't see them making any effort to support CUDA. Supporting CUDA on macOS is solely up to it's owner nVidia, and they could support it if they wished.
ID: 2021704 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2021723 - Posted: 4 Dec 2019, 21:22:03 UTC - in response to Message 2021680.  

Which is why one lesson we could learn from history is 'keep one foot in both camps', and another is 'keep an eye open for new third, fourth, fifth options riding up over the horizon'. I get the impression that Apple is dropping both CUDA and OpenCL, though I don't have an immediate feel for what it replacing them.

Apple is dropping support for both OpenGL, OpenCL and CUDA and going with their in-house Metal API. Pretty much a closed platform now. You either run Apple applications on your Apple hardware or nothing else.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2021723 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2021725 - Posted: 4 Dec 2019, 21:26:48 UTC - in response to Message 2021700.  

Then I think the project should focus on recruiting the talent that can port the special sauce CUDA code to Windows.

+100
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2021725 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14653
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2021739 - Posted: 4 Dec 2019, 22:32:45 UTC - in response to Message 2021700.  

Then I think the project should focus on recruiting the talent that can port the special sauce CUDA code to Windows.
I made that offer to Eric some years ago. At the time, he felt it right not to proceed, but later contacted me directly to participate in the Parkes server fundraiser. I was happy to do so, and I was introduced to 'Amigos' in her rack in July.

Maybe the time has come to revitalise that idea. I'm willing, but it has to fit into Eric's timetable. The responsibility for employing new staff is far greater than for purchasing an inanimate lump of metal and stuffing it into a data centre: the last thing either of us would have wanted would be to hire somebody whose primary task was to raise the funds for their next year's stipend. We have to do this properly - with an endowment - or not at all.
ID: 2021739 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65755
Credit: 55,293,173
RAC: 49
United States
Message 2022602 - Posted: 10 Dec 2019, 7:35:22 UTC - in response to Message 2021725.  

Then I think the project should focus on recruiting the talent that can port the special sauce CUDA code to Windows.

+100

+101
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 2022602 · Report as offensive
Profile Bill Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 30 Nov 05
Posts: 282
Credit: 6,916,194
RAC: 60
United States
Message 2022939 - Posted: 13 Dec 2019, 4:07:11 UTC

Richard, I think you're correct that for the project to survive we need to be pass responsibility from the people that have worked on the project to the next group of people who are willing and able to work on it. It reminds me a bit of Carl Sagan's chapter in Cosmos, "The Persistence of Memory", where he describes literacy and writing books as a good way for many people to remember something, not just one person.

I also like the idea of having a new cuda developer to port the cuda90 application. However, if we are having trouble finding someone that is 'able' to be 'willing', what about someone that is 'willing' but not 'able'? I have coded before, but unfortunately that was fifteen years ago in college, and most of that knowledge has faded. I'm sure there are other people here who have similar levels of programming experience. Is there a way those of us that have a minimal amount of coding experience could help? Even if it takes them time to understand the existing code and programming language, wouldn't that help be better than none at all?

Granted, I can imagine many reasons NOT to want inexperienced help. However, I would think that having more involvement with those that aren't professionals (or even semi-professionals) could still provide some benefit.

As for myself, I would love to spend time to get a better understanding of the science and code of it all to make a contribution. Unfortunately, I don't have much free time at this moment...perhaps in a few months.
Seti@home classic: 1,456 results, 1.613 years CPU time
ID: 2022939 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 2023789 - Posted: 20 Dec 2019, 0:02:19 UTC - in response to Message 2021702.  

Then I think the project should focus on recruiting the talent that can port the special sauce CUDA code to Windows.

Her's an idea from left field. How about a special fundraiser to hire one such as but not necessarily Jason Gee?


I would 2nd that Idea. And if someone will post a link I will start throwing money at it since I don't have the talent or skills to.

Tom
A proud member of the OFA (Old Farts Association).
ID: 2023789 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : False alarm, but please read: GTX 1660 Super, driver 441.12, Windows 7


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.