Problem with PPD and GPU

Human-Shield · 18 Jul 2013 at 15:10

Im hoping you can help.

Ive got an xeon running at 2.5gig and a FX3800 gpu. I was hoping that running the GPU would be a lot faster than the CPU.

I can only get the GPU to run if the client is set to full. is this right?

Running at full the predicted PPD drops from about 10000 to 6000 why is this?

Ive also got the GPU stuck at 99.99% complete not for a day and its not budging. Any idea how to fix it? the log says its an unstable client and shuts down?

cheers

HS

Biffa · 18 Jul 2013 at 16:13

Hiya, welcome to the team.

Couple of things, which Xeon do you have? That will make baselining its performance easier to work out.

Quadro FX3800 is effectively a GTX260 desktop equivalent card with 192 cores/shaders

Now I haven't run a card of that era for a while on Folding@Home so I'm not sure what you are likely to get on it, the latest cores are for Fermi or later GPU's so it would probably be Core 16 or older on that one.

I'd try pausing the CPU slot and try running just the GPU slot and see what PPD you get, you may have to use something like HFM.NET to check the output performance esp if you are using the v7 Folding client.

So try isolating the clients first, see what performance you get on just the CPU, then on just the GPU. It may be that you are better off just folding on one or the other rather than trying to do both.

Also do you use the machine for anything else? If so then when you are using it PPD will drop quite a lot.

Lastly, post some of the Log file from the folding client (you can use [ code ] tags in the forum for that) it may help diagnose any problems you have.

Human-Shield · 19 Jul 2013 at 09:04

Hi,

Cheers for that. the xeon is an old E5645 @2.4

Im having issues. the gpu stopped yesterday at 99.99% done. it doesnt run when im using the computer normally. I tried seting it to full on the slider and the gpu kicked up but in the log file it says that it was unstable and shutdown.

So ive removed the gpu slot to try and get it going again and its downloaded a WU but its been sat at 0% for a day now and the unstable bit hasnt cropped up again though. It shows its running.

Not sure whats happened!

Human-Shield · 19 Jul 2013 at 09:05

15:54:44:Adding folding slot 01: READY cpu:12
15:54:44:WU00:FS01:Connecting to assign3.stanford.edu:8080
15:54:45:Removing old file 'configs/config-20130718-095141.xml'
15:54:45:Saving configuration to config.xml
15:54:45:<config>
15:54:45: 
15:54:45: <checkpoint v='10'/>
15:54:45: <cpu-affinity v='true'/>
15:54:45:
15:54:45: 
15:54:45: <power v='full'/>
15:54:45:
15:54:45: 
15:54:45: <proxy v=':8080'/>
15:54:45:
15:54:45: 
15:54:45: <pause-on-battery v='false'/>
15:54:45:
15:54:45: 
15:54:45: <passkey v='********************************'/>
15:54:45: <team v='10'/>
15:54:45: <user v='Human-Shield'/>
15:54:45:
15:54:45: 
15:54:45: <slot id='0' type='GPU'/>
15:54:45: <slot id='1' type='CPU'>
15:54:45: <cpus v='-1'/>
15:54:45: </slot>
15:54:45:</config>
15:55:06:WARNING:WU00:FS01:Failed to get assignment from 'assign3.stanford.edu:8080': Failed to connect to assign3.stanford.edu:8080: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
15:55:06:WU00:FS01:Connecting to assign4.stanford.edu:80
15:55:06:WU00:FS01:News: Welcome to Folding@Home
15:55:06:WU00:FS01:Assigned to work server 171.64.65.101
15:55:06:WU00:FS01:Requesting new work unit for slot 01: READY cpu:12 from 171.64.65.101
15:55:06:WU00:FS01:Connecting to 171.64.65.101:8080
15:55:27:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
15:55:27:WU00:FS01:Connecting to 171.64.65.101:80
15:55:29:WU00:FS01

ownloading 536.84KiB
15:55:29:WU00:FS01

ownload complete
15:55:30:WU00:FS01:Received Unit: id:00 state

OWNLOAD error:NO_ERROR project:7647 run:178 clone:0 gen:140 core:0xa4 unit:0x000000b7664f2dcd4fa7fe957e673231
15:55:30:WU00:FS01:Starting
15:55:30:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -np 12
15:55:30:WU00:FS01:Started FahCore on PID 4832
15:55:30:WU00:FS01:Core PID:4864
15:55:30:WU00:FS01:FahCore 0xa4 started
15:55:30:WU00:FS01:0xa4:
15:55:30:WU00:FS01:0xa4:*------------------------------*
15:55:30:WU00:FS01:0xa4:Folding@Home Gromacs GB Core
15:55:30:WU00:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
15:55:30:WU00:FS01:0xa4:
15:55:30:WU00:FS01:0xa4

reparing to commence simulation
15:55:30:WU00:FS01:0xa4:- Looking at optimizations...
15:55:30:WU00:FS01:0xa4:- Created dyn
15:55:30:WU00:FS01:0xa4:- Files status OK
15:55:30:WU00:FS01:0xa4:- Expanded 549214 -> 848748 (decompressed 154.5 percent)
15:55:30:WU00:FS01:0xa4:Called DecompressByteArray: compressed_data_size=549214 data_size=848748, decompressed_data_size=848748 diff=0
15:55:30:WU00:FS01:0xa4:- Digital signature verified
15:55:30:WU00:FS01:0xa4:
15:55:30:WU00:FS01:0xa4

roject: 7647 (Run 178, Clone 0, Gen 140)
15:55:30:WU00:FS01:0xa4:
15:55:30:WU00:FS01:0xa4:Assembly optimizations on if available.
15:55:30:WU00:FS01:0xa4:Entering M.D.
15:55:36:WU00:FS01:0xa4:Mapping NT from 12 to 12
15:55:36:WU00:FS01:0xa4:Completed 0 out of 2500000 steps (0%)
16:17:04:WU00:FS01:0xa4:Completed 25000 out of 2500000 steps (1%)
16:37:37:WU00:FS01:0xa4:Completed 50000 out of 2500000 steps (2%)
16:58:14:WU00:FS01:0xa4:Completed 75000 out of 2500000 steps (3%)
17:18:47:WU00:FS01:0xa4:Completed 100000 out of 2500000 steps (4%)
17:39:24:WU00:FS01:0xa4:Completed 125000 out of 2500000 steps (5%)
18:00:06:WU00:FS01:0xa4:Completed 150000 out of 2500000 steps (6%)
18:20:43:WU00:FS01:0xa4:Completed 175000 out of 2500000 steps (7%)
18:41:21:WU00:FS01:0xa4:Completed 200000 out of 2500000 steps (8%)
19:01:56:WU00:FS01:0xa4:Completed 225000 out of 2500000 steps (9%)
19:22:32:WU00:FS01:0xa4:Completed 250000 out of 2500000 steps (10%)
19:43:06:WU00:FS01:0xa4:Completed 275000 out of 2500000 steps (11%)
20:04:02:WU00:FS01:0xa4:Completed 300000 out of 2500000 steps (12%)
20:24:39:WU00:FS01:0xa4:Completed 325000 out of 2500000 steps (13%)
20:45:17:WU00:FS01:0xa4:Completed 350000 out of 2500000 steps (14%)
21:06:04:WU00:FS01:0xa4:Completed 375000 out of 2500000 steps (15%)
21:26:38:WU00:FS01:0xa4:Completed 400000 out of 2500000 steps (16%)
21:47:15:WU00:FS01:0xa4:Completed 425000 out of 2500000 steps (17%)
******************************* Date: 2013-07-18 *******************************
22:07:52:WU00:FS01:0xa4:Completed 450000 out of 2500000 steps (18%)
22:28:29:WU00:FS01:0xa4:Completed 475000 out of 2500000 steps (19%)
22:49:06:WU00:FS01:0xa4:Completed 500000 out of 2500000 steps (20%)
23:09:35:WU00:FS01:0xa4:Completed 525000 out of 2500000 steps (21%)
23:30:41:WU00:FS01:0xa4:Completed 550000 out of 2500000 steps (22%)
23:55:20:WU00:FS01:0xa4:Completed 575000 out of 2500000 steps (23%)
00:16:28:WU00:FS01:0xa4:Completed 600000 out of 2500000 steps (24%)
00:37:01:WU00:FS01:0xa4:Completed 625000 out of 2500000 steps (25%)
01:01:54:WU00:FS01:0xa4:Completed 650000 out of 2500000 steps (26%)
01:22:08:WU00:FS01:0xa4:Completed 675000 out of 2500000 steps (27%)
01:43:24:WU00:FS01:0xa4:Completed 700000 out of 2500000 steps (28%)
02:04:05:WU00:FS01:0xa4:Completed 725000 out of 2500000 steps (29%)
02:24:40:WU00:FS01:0xa4:Completed 750000 out of 2500000 steps (30%)
02:45:13:WU00:FS01:0xa4:Completed 775000 out of 2500000 steps (31%)
03:05:52:WU00:FS01:0xa4:Completed 800000 out of 2500000 steps (32%)
03:26:29:WU00:FS01:0xa4:Completed 825000 out of 2500000 steps (33%)
03:47:09:WU00:FS01:0xa4:Completed 850000 out of 2500000 steps (34%)
04:07:46:WU00:FS01:0xa4:Completed 875000 out of 2500000 steps (35%)
******************************* Date: 2013-07-19 *******************************
04:28:18:WU00:FS01:0xa4:Completed 900000 out of 2500000 steps (36%)
04:48:48:WU00:FS01:0xa4:Completed 925000 out of 2500000 steps (37%)
05:09:23:WU00:FS01:0xa4:Completed 950000 out of 2500000 steps (38%)
05:30:03:WU00:FS01:0xa4:Completed 975000 out of 2500000 steps (39%)
05:50:45:WU00:FS01:0xa4:Completed 1000000 out of 2500000 steps (40%)
06:11:26:WU00:FS01:0xa4:Completed 1025000 out of 2500000 steps (41%)
06:32:02:WU00:FS01:0xa4:Completed 1050000 out of 2500000 steps (42%)
06:52:44:WU00:FS01:0xa4:Completed 1075000 out of 2500000 steps (43%)
07:13:31:WU00:FS01:0xa4:Completed 1100000 out of 2500000 steps (44%)
07:34:15:WU00:FS01:0xa4:Completed 1125000 out of 2500000 steps (45%)
07:55:02:WU00:FS01:0xa4:Completed 1150000 out of 2500000 steps (46%)
08:02:31:FS00:Shutting core down
08:02:31:FS01:Shutting core down
08:02:39:WU00:FS01:0xa4:Client no longer detected. Shutting down core
08:02:40:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
08:02:40:WU00:FS01:Starting
08:02:40:WARNING:WU00:FS01:Changed SMP threads from 12 to 11 this can cause some work units to fail
08:02:40:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users//AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -np 11
08:02:40:WU00:FS01:Started FahCore on PID 1520
08:02:40:WU00:FS01:Core PID:5352
08:02:40:WU00:FS01:FahCore 0xa4 started
08:02:40:WU00:FS01:0xa4:
08:02:40:WU00:FS01:0xa4:*------------------------------*
08:02:40:WU00:FS01:0xa4:Folding@Home Gromacs GB Core
08:02:40:WU00:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
08:02:40:WU00:FS01:0xa4:
08:02:40:WU00:FS01:0xa4

reparing to commence simulation
08:02:40:WU00:FS01:0xa4:- Looking at optimizations...
08:02:40:WU00:FS01:0xa4:- Files status OK
08:02:40:WU00:FS01:0xa4:- Expanded 549214 -> 848748 (decompressed 154.5 percent)
08:02:40:WU00:FS01:0xa4:Called DecompressByteArray: compressed_data_size=549214 data_size=848748, decompressed_data_size=848748 diff=0
08:02:40:WU00:FS01:0xa4:- Digital signature verified
08:02:40:WU00:FS01:0xa4:
08:02:40:WU00:FS01:0xa4

roject: 7647 (Run 178, Clone 0, Gen 140)
08:02:40:WU00:FS01:0xa4:
08:02:40:WU00:FS01:0xa4:Assembly optimizations on if available.
08:02:40:WU00:FS01:0xa4:Entering M.D.
08:02:46:WU00:FS01:0xa4:Using Gromacs checkpoints
08:02:46:WU00:FS01:0xa4:Mapping NT from 11 to 10
08:02:46:WU00:FS01:0xa4:Resuming from checkpoint
08:02:46:WU00:FS01:0xa4:Verified 00/wudata_01.log
08:02:47:WU00:FS01:0xa4:Verified 00/wudata_01.trr
08:02:47:WU00:FS01:0xa4:Verified 00/wudata_01.xtc
08:02:47:WU00:FS01:0xa4:Verified 00/wudata_01.edr
08:02:47:WU00:FS01:0xa4:Completed 1150690 out of 2500000 steps (46%)
08:03:32:WARNING:FS00:Killing WU01
08:03:32:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
08:03:33:Removing old file 'configs/config-20130718-130705.xml'
08:03:33:Saving configuration to config.xml
08:03:33:<config>
08:03:33: 
08:03:33: <checkpoint v='10'/>
08:03:33: <cpu-affinity v='true'/>
08:03:33:
08:03:33: 
08:03:33: <proxy v=':8080'/>
08:03:33:
08:03:33: 
08:03:33: <pause-on-battery v='false'/>
08:03:33:
08:03:33: 
08:03:33: <passkey v='********************************'/>
08:03:33: <team v='10'/>
08:03:33: <user v='Human-Shield'/>
08:03:33:
08:03:33: 
08:03:33: <slot id='0' type='GPU'/>
08:03:33: <slot id='1' type='CPU'>
08:03:33: <cpus v='-1'/>
08:03:33: </slot>
08:03:33:</config>

Biffa · 19 Jul 2013 at 12:59

What drivers are you using for the GPU? The log file only shows FoldingSlot01 which is the CPU. Try running GPUz and post a screenshot of what it shows on the first screen.

Human-Shield · 19 Jul 2013 at 13:32

*********************** Log Started 2013-07-18T15:53:28Z ***********************
15:53:28:************************* Folding@home Client *************************
15:53:28: Website: http://folding.stanford.edu/
15:53:28: Copyright: (c) 2009-2013 Stanford University
15:53:28: Author: Joseph Coffland <[email protected]>
15:53:28: Args:
15:53:28: Config: C:/Users/AppData/Roaming/FAHClient/config.xml
15:53:28:******************************** Build ********************************
15:53:28: Version: 7.3.6
15:53:28: Date: Feb 18 2013
15:53:28: Time: 15:25:17
15:53:28: SVN Rev: 3923
15:53:28: Branch: fah/trunk/client
15:53:28: Compiler: Intel(R) C++ MSVC 1500 mode 1200
15:53:28: Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
15:53:28: /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
15:53:28: Platform: win32 XP
15:53:28: Bits: 32
15:53:28: Mode: Release
15:53:28:******************************* System ********************************
15:53:28: CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
15:53:28: CPU ID: GenuineIntel Family 6 Model 44 Stepping 2
15:53:28: CPUs: 12
15:53:28: Memory: 5.98GiB
15:53:28: Free Memory: 4.74GiB
15:53:28: Threads: WINDOWS_THREADS
15:53:28: Has Battery: false
15:53:28: On Battery: false
15:53:28: UTC offset: 1
15:53:28: PID: 4112
15:53:28: CWD: C://AppData/Roaming/FAHClient
15:53:28: OS: Windows 7 Professional
15:53:28: OS Arch: AMD64
15:53:28: GPUs: 1
15:53:28: GPU 0: NVIDIA:1 GT200GL [Quadro FX 3800]
15:53:28: CUDA: 1.3
15:53:28: CUDA Driver: 5050
15:53:28:Win32 Service: false
15:53:28:***********************************************************************
15:53:28:<config>
15:53:28: 
15:53:28: <checkpoint v='10'/>
15:53:28: <cpu-affinity v='true'/>
15:53:28:
15:53:28: 
15:53:28: <power v='full'/>
15:53:28:
15:53:28: 
15:53:28: <proxy v=':8080'/>
15:53:28:
15:53:28: 
15:53:28: <pause-on-battery v='false'/>
15:53:28:
15:53:28: 
15:53:28: <passkey v='********************************'/>
15:53:28: <team v='10'/>
15:53:28: <user v='Human-Shield'/>
15:53:28:
15:53:28: 
15:53:28: <slot id='1' type='CPU'>
15:53:28: <cpus v='-1'/>
15:53:28: </slot>
15:53:28: <slot id='0' type='GPU'/>
15:53:28:</config>
15:53:28:Connecting to assign-GPU.stanford.edu:80
15:53:29:Connecting to assign-GPU.stanford.edu:8080
15:53:50:WARNING:Attempting to update GPUs.txt: Failed to connect to assign-GPU.stanford.edu:8080: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
15:53:50:Trying to access database...
15:53:50:Successfully acquired database lock
15:53:50:Enabled folding slot
15:53:50:ERROR:Exception: Option 'gpu-index' has no default and is not set.
15:53:50:Enabled folding slot 00: READY gpu:0:GT200GL [Quadro FX 3800]
15:53:50:WARNING:WU00:Slot ID 1 no longer exists and there are no other matching slots, dumping
15:53:50:WARNING:WU02:Slot ID 1 no longer exists and there are no other matching slots, dumping
15:53:50:WU00:FS01:Sending unit results: id:00 state:SEND error

UMPED project:7610 run:403 clone:0 gen:265 core:0xa4 unit:0x00000160664f2dd04de6d420430f72ca
15:53:50:WU00:FS01:Connecting to 171.64.65.104:8080
15:53:50:WU01:FS00:Starting
15:53:50:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users//AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
15:53:50:WU01:FS00:Started FahCore on PID 5060
15:53:50:WU01:FS00:Core PID:5072
15:53:50:WU01:FS00:FahCore 0x11 started
15:53:50:WU02:Sending unit results: id:02 state:SEND error

UMPED project:7809 run:7 clone:205 gen:112 core:0xa4 unit:0x000000a60a3b1e874e31149c093a8b2d
15:53:50:WU02:Connecting to 171.64.65.99:8080
15:53:50:WU01:FS00:0x11:
15:53:50:WU01:FS00:0x11:*------------------------------*
15:53:50:WU01:FS00:0x11:Folding@Home GPU Core
15:53:50:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
15:53:50:WU01:FS00:0x11:
15:53:50:WU01:FS00:0x11:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
15:53:50:WU01:FS00:0x11:Build host: amoeba
15:53:50:WU01:FS00:0x11:Board Type: Nvidia
15:53:50:WU01:FS00:0x11:Core :
15:53:50:WU01:FS00:0x11

reparing to commence simulation
15:53:50:WU01:FS00:0x11:- Looking at optimizations...
15:53:50:WU01:FS00:0x11

eleteFrameFiles: successfully deleted file=01/wudata_01.ckp
15:53:50:WU01:FS00:0x11:- Created dyn
15:53:50:WU01:FS00:0x11:- Files status OK
15:53:50:WU01:FS00:0x11:- Expanded 62894 -> 336763 (decompressed 535.4 percent)
15:53:50:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=62894 data_size=336763, decompressed_data_size=336763 diff=0
15:53:50:WU01:FS00:0x11:- Digital signature verified
15:53:50:WU01:FS00:0x11:
15:53:50:WU01:FS00:0x11

roject: 10501 (Run 32, Clone 0, Gen 1333)
15:53:50:WU01:FS00:0x11:
15:53:50:WU01:FS00:0x11:Assembly optimizations on if available.
15:53:50:WU01:FS00:0x11:Entering M.D.
15:53:56:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr: 1629719815 3940959380 699480707 98139461 3779876519
15:53:56:WU01:FS00:0x11:
15:53:56:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
15:53:56:WU01:FS00:0x11:
15:53:57:WU01:FS00:0x11:Working on Protein
15:53:58:WU01:FS00:0x11:Client config unavailable.
15:53:58:WU01:FS00:0x11:Starting GUI Server
15:53:58:WU01:FS00:0x11:mdrun_gpu returned
15:53:58:WU01:FS00:0x11:NANs detected on GPU
15:53:58:WU01:FS00:0x11:
15:53:58:WU01:FS00:0x11:Folding@home Core Shutdown: UNSTABLE_MACHINE
15:54:11:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
15:54:11:WU00:FS01:Connecting to 171.64.65.104:80
15:54:11:WARNING:WU02:WorkServer connection failed on port 8080 trying 80
15:54:11:WU02:Connecting to 171.64.65.99:80
15:54:11:WU00:FS01:Server responded WORK_ACK (400)
15:54:11:WU00:FS01:Cleaning up
15:54:12:WU02:Server responded WORK_ACK (400)
15:54:12:WU02:Cleaning up
15:54:44:Adding folding slot 01: READY cpu:12
15:54:44:WU00:FS01:Connecting to assign3.stanford.edu:8080
15:54:45:Removing old file 'configs/config-20130718-095141.xml'
15:54:45:Saving configuration to config.xml
15:54:45:<config>
15:54:45: 
15:54:45: <checkpoint v='10'/>
15:54:45: <cpu-affinity v='true'/>
15:54:45:
15:54:45: 
15:54:45: <power v='full'/>
15:54:45:
15:54:45: 
15:54:45: <proxy v=':8080'/>
15:54:45:
15:54:45: 
15:54:45: <pause-on-battery v='false'/>
15:54:45:
15:54:45: 
15:54:45: <passkey v='********************************'/>
15:54:45: <team v='10'/>
15:54:45: <user v='Human-Shield'/>
15:54:45:
15:54:45: 
15:54:45: <slot id='0' type='GPU'/>
15:54:45: <slot id='1' type='CPU'>
15:54:45: <cpus v='-1'/>
15:54:45: </slot>
15:54:45:</config>
15:55:06:WARNING:WU00:FS01:Failed to get assignment from 'assign3.stanford.edu:8080': Failed to connect to assign3.stanford.edu:8080: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
15:55:06:WU00:FS01:Connecting to assign4.stanford.edu:80
15:55:06:WU00:FS01:News: Welcome to Folding@Home
15:55:06:WU00:FS01:Assigned to work server 171.64.65.101
15:55:06:WU00:FS01:Requesting new work unit for slot 01: READY cpu:12 from 171.64.65.101
15:55:06:WU00:FS01:Connecting to 171.64.65.101:8080
15:55:27:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
15:55:27:WU00:FS01:Connecting to 171.64.65.101:80
15:55:29:WU00:FS01

ownloading 536.84KiB
15:55:29:WU00:FS01

ownload complete
15:55:30:WU00:FS01:Received Unit: id:00 state

OWNLOAD error:NO_ERROR project:7647 run:178 clone:0 gen:140 core:0xa4 unit:0x000000b7664f2dcd4fa7fe957e673231
15:55:30:WU00:FS01:Starting
15:55:30:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users//AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -np 12
15:55:30:WU00:FS01:Started FahCore on PID 4832
15:55:30:WU00:FS01:Core PID:4864
15:55:30:WU00:FS01:FahCore 0xa4 started
15:55:30:WU00:FS01:0xa4:
15:55:30:WU00:FS01:0xa4:*------------------------------*
15:55:30:WU00:FS01:0xa4:Folding@Home Gromacs GB Core
15:55:30:WU00:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
15:55:30:WU00:FS01:0xa4:
15:55:30:WU00:FS01:0xa4

reparing to commence simulation
15:55:30:WU00:FS01:0xa4:- Looking at optimizations...
15:55:30:WU00:FS01:0xa4:- Created dyn
15:55:30:WU00:FS01:0xa4:- Files status OK
15:55:30:WU00:FS01:0xa4:- Expanded 549214 -> 848748 (decompressed 154.5 percent)
15:55:30:WU00:FS01:0xa4:Called DecompressByteArray: compressed_data_size=549214 data_size=848748, decompressed_data_size=848748 diff=0
15:55:30:WU00:FS01:0xa4:- Digital signature verified
15:55:30:WU00:FS01:0xa4:
15:55:30:WU00:FS01:0xa4

roject: 7647 (Run 178, Clone 0, Gen 140)
15:55:30:WU00:FS01:0xa4:
15:55:30:WU00:FS01:0xa4:Assembly optimizations on if available.
15:55:30:WU00:FS01:0xa4:Entering M.D.
15:55:36:WU00:FS01:0xa4:Mapping NT from 12 to 12
15:55:36:WU00:FS01:0xa4:Completed 0 out of 2500000 steps (0%)
16:17:04:WU00:FS01:0xa4:Completed 25000 out of 2500000 steps (1%)
16:37:37:WU00:FS01:0xa4:Completed 50000 out of 2500000 steps (2%)
16:58:14:WU00:FS01:0xa4:Completed 75000 out of 2500000 steps (3%)
17:18:47:WU00:FS01:0xa4:Completed 100000 out of 2500000 steps (4%)
17:39:24:WU00:FS01:0xa4:Completed 125000 out of 2500000 steps (5%)
18:00:06:WU00:FS01:0xa4:Completed 150000 out of 2500000 steps (6%)
18:20:43:WU00:FS01:0xa4:Completed 175000 out of 2500000 steps (7%)
18:41:21:WU00:FS01:0xa4:Completed 200000 out of 2500000 steps (8%)
19:01:56:WU00:FS01:0xa4:Completed 225000 out of 2500000 steps (9%)
19:22:32:WU00:FS01:0xa4:Completed 250000 out of 2500000 steps (10%)
19:43:06:WU00:FS01:0xa4:Completed 275000 out of 2500000 steps (11%)
20:04:02:WU00:FS01:0xa4:Completed 300000 out of 2500000 steps (12%)
20:24:39:WU00:FS01:0xa4:Completed 325000 out of 2500000 steps (13%)
20:45:17:WU00:FS01:0xa4:Completed 350000 out of 2500000 steps (14%)
21:06:04:WU00:FS01:0xa4:Completed 375000 out of 2500000 steps (15%)
21:26:38:WU00:FS01:0xa4:Completed 400000 out of 2500000 steps (16%)
21:47:15:WU00:FS01:0xa4:Completed 425000 out of 2500000 steps (17%)
******************************* Date: 2013-07-18 *******************************
22:07:52:WU00:FS01:0xa4:Completed 450000 out of 2500000 steps (18%)
22:28:29:WU00:FS01:0xa4:Completed 475000 out of 2500000 steps (19%)
22:49:06:WU00:FS01:0xa4:Completed 500000 out of 2500000 steps (20%)
23:09:35:WU00:FS01:0xa4:Completed 525000 out of 2500000 steps (21%)
23:30:41:WU00:FS01:0xa4:Completed 550000 out of 2500000 steps (22%)
23:55:20:WU00:FS01:0xa4:Completed 575000 out of 2500000 steps (23%)
00:16:28:WU00:FS01:0xa4:Completed 600000 out of 2500000 steps (24%)
00:37:01:WU00:FS01:0xa4:Completed 625000 out of 2500000 steps (25%)
01:01:54:WU00:FS01:0xa4:Completed 650000 out of 2500000 steps (26%)
01:22:08:WU00:FS01:0xa4:Completed 675000 out of 2500000 steps (27%)
01:43:24:WU00:FS01:0xa4:Completed 700000 out of 2500000 steps (28%)
02:04:05:WU00:FS01:0xa4:Completed 725000 out of 2500000 steps (29%)
02:24:40:WU00:FS01:0xa4:Completed 750000 out of 2500000 steps (30%)
02:45:13:WU00:FS01:0xa4:Completed 775000 out of 2500000 steps (31%)
03:05:52:WU00:FS01:0xa4:Completed 800000 out of 2500000 steps (32%)
03:26:29:WU00:FS01:0xa4:Completed 825000 out of 2500000 steps (33%)
03:47:09:WU00:FS01:0xa4:Completed 850000 out of 2500000 steps (34%)
04:07:46:WU00:FS01:0xa4:Completed 875000 out of 2500000 steps (35%)
******************************* Date: 2013-07-19 *******************************
04:28:18:WU00:FS01:0xa4:Completed 900000 out of 2500000 steps (36%)
04:48:48:WU00:FS01:0xa4:Completed 925000 out of 2500000 steps (37%)
05:09:23:WU00:FS01:0xa4:Completed 950000 out of 2500000 steps (38%)
05:30:03:WU00:FS01:0xa4:Completed 975000 out of 2500000 steps (39%)
05:50:45:WU00:FS01:0xa4:Completed 1000000 out of 2500000 steps (40%)
06:11:26:WU00:FS01:0xa4:Completed 1025000 out of 2500000 steps (41%)
06:32:02:WU00:FS01:0xa4:Completed 1050000 out of 2500000 steps (42%)
06:52:44:WU00:FS01:0xa4:Completed 1075000 out of 2500000 steps (43%)
07:13:31:WU00:FS01:0xa4:Completed 1100000 out of 2500000 steps (44%)
07:34:15:WU00:FS01:0xa4:Completed 1125000 out of 2500000 steps (45%)
07:55:02:WU00:FS01:0xa4:Completed 1150000 out of 2500000 steps (46%)
08:02:31:FS00:Shutting core down
08:02:31:FS01:Shutting core down
08:02:39:WU00:FS01:0xa4:Client no longer detected. Shutting down core
08:02:40:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
08:02:40:WU00:FS01:Starting
08:02:40:WARNING:WU00:FS01:Changed SMP threads from 12 to 11 this can cause some work units to fail
08:02:40:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -np 11
08:02:40:WU00:FS01:Started FahCore on PID 1520
08:02:40:WU00:FS01:Core PID:5352
08:02:40:WU00:FS01:FahCore 0xa4 started
08:02:40:WU00:FS01:0xa4:
08:02:40:WU00:FS01:0xa4:*------------------------------*
08:02:40:WU00:FS01:0xa4:Folding@Home Gromacs GB Core
08:02:40:WU00:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
08:02:40:WU00:FS01:0xa4:
08:02:40:WU00:FS01:0xa4

reparing to commence simulation
08:02:40:WU00:FS01:0xa4:- Looking at optimizations...
08:02:40:WU00:FS01:0xa4:- Files status OK
08:02:40:WU00:FS01:0xa4:- Expanded 549214 -> 848748 (decompressed 154.5 percent)
08:02:40:WU00:FS01:0xa4:Called DecompressByteArray: compressed_data_size=549214 data_size=848748, decompressed_data_size=848748 diff=0
08:02:40:WU00:FS01:0xa4:- Digital signature verified
08:02:40:WU00:FS01:0xa4:
08:02:40:WU00:FS01:0xa4

roject: 7647 (Run 178, Clone 0, Gen 140)
08:02:40:WU00:FS01:0xa4:
08:02:40:WU00:FS01:0xa4:Assembly optimizations on if available.
08:02:40:WU00:FS01:0xa4:Entering M.D.
08:02:46:WU00:FS01:0xa4:Using Gromacs checkpoints
08:02:46:WU00:FS01:0xa4:Mapping NT from 11 to 10
08:02:46:WU00:FS01:0xa4:Resuming from checkpoint
08:02:46:WU00:FS01:0xa4:Verified 00/wudata_01.log
08:02:47:WU00:FS01:0xa4:Verified 00/wudata_01.trr
08:02:47:WU00:FS01:0xa4:Verified 00/wudata_01.xtc
08:02:47:WU00:FS01:0xa4:Verified 00/wudata_01.edr
08:02:47:WU00:FS01:0xa4:Completed 1150690 out of 2500000 steps (46%)
08:03:32:WARNING:FS00:Killing WU01
08:03:32:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
08:03:33:Removing old file 'configs/config-20130718-130705.xml'
08:03:33:Saving configuration to config.xml
08:03:33:<config>
08:03:33: 
08:03:33: <checkpoint v='10'/>
08:03:33: <cpu-affinity v='true'/>
08:03:33:
08:03:33: 
08:03:33: <proxy v=':8080'/>
08:03:33:
08:03:33: 
08:03:33: <pause-on-battery v='false'/>
08:03:33:
08:03:33: 
08:03:33: <passkey v='********************************'/>
08:03:33: <team v='10'/>
08:03:33: <user v='Human-Shield'/>
08:03:33:
08:03:33: 
08:03:33: <slot id='0' type='GPU'/>
08:03:33: <slot id='1' type='CPU'>
08:03:33: <cpus v='-1'/>
08:03:33: </slot>
08:03:33:</config>
08:22:56:WU00:FS01:0xa4:Completed 1175000 out of 2500000 steps (47%)
08:31:02:WU01:FS00:Starting
08:31:02:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
08:31:02:WU01:FS00:Started FahCore on PID 5432
08:31:02:WU01:FS00:Core PID:1552
08:31:02:WU01:FS00:FahCore 0x11 started
08:31:03:WU01:FS00:0x11:
08:31:03:WU01:FS00:0x11:*------------------------------*
08:31:03:WU01:FS00:0x11:Folding@Home GPU Core
08:31:03:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
08:31:03:WU01:FS00:0x11:
08:31:03:WU01:FS00:0x11:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
08:31:03:WU01:FS00:0x11:Build host: amoeba
08:31:03:WU01:FS00:0x11:Board Type: Nvidia
08:31:03:WU01:FS00:0x11:Core :
08:31:03:WU01:FS00:0x11

reparing to commence simulation
08:31:03:WU01:FS00:0x11:- Looking at optimizations...
08:31:03:WU01:FS00:0x11

eleteFrameFiles: successfully deleted file=01/wudata_01.ckp
08:31:03:WU01:FS00:0x11:- Created dyn
08:31:03:WU01:FS00:0x11:- Files status OK
08:31:03:WU01:FS00:0x11:- Expanded 62894 -> 336763 (decompressed 535.4 percent)
08:31:03:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=62894 data_size=336763, decompressed_data_size=336763 diff=0
08:31:03:WU01:FS00:0x11:- Digital signature verified
08:31:03:WU01:FS00:0x11:
08:31:03:WU01:FS00:0x11

roject: 10501 (Run 32, Clone 0, Gen 1333)
08:31:03:WU01:FS00:0x11:
08:31:03:WU01:FS00:0x11:Assembly optimizations on if available.
08:31:03:WU01:FS00:0x11:Entering M.D.
08:31:09:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr: 1629719815 3940959380 699480707 98139461 3779876519
08:31:09:WU01:FS00:0x11:
08:31:09:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
08:31:09:WU01:FS00:0x11:
08:31:09:WU01:FS00:0x11:Working on Protein
08:31:12:WU01:FS00:0x11:Client config unavailable.
08:31:12:WU01:FS00:0x11:Starting GUI Server
08:31:12:WU01:FS00:0x11:mdrun_gpu returned
08:31:12:WU01:FS00:0x11:NANs detected on GPU
08:31:12:WU01:FS00:0x11:
08:31:12:WU01:FS00:0x11:Folding@home Core Shutdown: UNSTABLE_MACHINE
08:32:40:FS00:Shutting core down
08:33:41:WARNING:FS00:Killing WU01
08:33:41:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
08:43:47:WU00:FS01:0xa4:Completed 1200000 out of 2500000 steps (48%)
08:52:36:WU01:FS00:Starting
08:52:36:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
08:52:36:WU01:FS00:Started FahCore on PID 1892
08:52:36:WU01:FS00:Core PID:6048
08:52:36:WU01:FS00:FahCore 0x11 started
08:52:36:WU01:FS00:0x11:
08:52:36:WU01:FS00:0x11:*------------------------------*
08:52:36:WU01:FS00:0x11:Folding@Home GPU Core
08:52:36:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
08:52:36:WU01:FS00:0x11:
08:52:36:WU01:FS00:0x11:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
08:52:36:WU01:FS00:0x11:Build host: amoeba
08:52:36:WU01:FS00:0x11:Board Type: Nvidia
08:52:36:WU01:FS00:0x11:Core :
08:52:36:WU01:FS00:0x11

reparing to commence simulation
08:52:36:WU01:FS00:0x11:- Looking at optimizations...
08:52:36:WU01:FS00:0x11

eleteFrameFiles: successfully deleted file=01/wudata_01.ckp
08:52:36:WU01:FS00:0x11:- Created dyn
08:52:36:WU01:FS00:0x11:- Files status OK
08:52:36:WU01:FS00:0x11:- Expanded 62894 -> 336763 (decompressed 535.4 percent)
08:52:36:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=62894 data_size=336763, decompressed_data_size=336763 diff=0
08:52:36:WU01:FS00:0x11:- Digital signature verified
08:52:36:WU01:FS00:0x11:
08:52:36:WU01:FS00:0x11

roject: 10501 (Run 32, Clone 0, Gen 1333)
08:52:36:WU01:FS00:0x11:
08:52:36:WU01:FS00:0x11:Assembly optimizations on if available.
08:52:36:WU01:FS00:0x11:Entering M.D.
08:52:39:FS00:Shutting core down
08:52:42:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr: 1629719815 3940959380 699480707 98139461 3779876519
08:52:42:WU01:FS00:0x11:
08:52:42:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
08:52:42:WU01:FS00:0x11:
08:52:42:WU01:FS00:0x11:Working on Protein
08:52:44:WU01:FS00:0x11:Client config unavailable.
08:52:44:WU01:FS00:0x11:Starting GUI Server
08:52:44:WU01:FS00:0x11:mdrun_gpu returned
08:52:44:WU01:FS00:0x11:NANs detected on GPU
08:52:44:WU01:FS00:0x11:
08:52:44:WU01:FS00:0x11:Folding@home Core Shutdown: UNSTABLE_MACHINE
08:53:40:WARNING:FS00:Killing WU01
08:53:40:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
09:04:46:WU00:FS01:0xa4:Completed 1225000 out of 2500000 steps (49%)
09:25:01:WU00:FS01:0xa4:Completed 1250000 out of 2500000 steps (50%)
09:45:24:WU00:FS01:0xa4:Completed 1275000 out of 2500000 steps (51%)
10:05:49:WU00:FS01:0xa4:Completed 1300000 out of 2500000 steps (52%)
10:11:07:WU01:FS00:Starting
10:11:07:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
10:11:07:WU01:FS00:Started FahCore on PID 5544
10:11:07:WU01:FS00:Core PID:3932
10:11:07:WU01:FS00:FahCore 0x11 started
10:11:08:WU01:FS00:0x11:
10:11:08:WU01:FS00:0x11:*------------------------------*
10:11:08:WU01:FS00:0x11:Folding@Home GPU Core
10:11:08:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
10:11:08:WU01:FS00:0x11:
10:11:08:WU01:FS00:0x11:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
10:11:08:WU01:FS00:0x11:Build host: amoeba
10:11:08:WU01:FS00:0x11:Board Type: Nvidia
10:11:08:WU01:FS00:0x11:Core :
10:11:08:WU01:FS00:0x11

reparing to commence simulation
10:11:08:WU01:FS00:0x11:- Looking at optimizations...
10:11:08:WU01:FS00:0x11

eleteFrameFiles: successfully deleted file=01/wudata_01.ckp
10:11:08:WU01:FS00:0x11:- Created dyn
10:11:08:WU01:FS00:0x11:- Files status OK
10:11:08:WU01:FS00:0x11:- Expanded 62894 -> 336763 (decompressed 535.4 percent)
10:11:08:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=62894 data_size=336763, decompressed_data_size=336763 diff=0
10:11:08:WU01:FS00:0x11:- Digital signature verified
10:11:08:WU01:FS00:0x11:
10:11:08:WU01:FS00:0x11

roject: 10501 (Run 32, Clone 0, Gen 1333)
10:11:08:WU01:FS00:0x11:
10:11:08:WU01:FS00:0x11:Assembly optimizations on if available.
10:11:08:WU01:FS00:0x11:Entering M.D.
10:11:14:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr: 1629719815 3940959380 699480707 98139461 3779876519
10:11:14:WU01:FS00:0x11:
10:11:14:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
10:11:14:WU01:FS00:0x11:
10:11:15:WU01:FS00:0x11:Working on Protein
10:11:17:WU01:FS00:0x11:Client config unavailable.
10:11:17:WU01:FS00:0x11:Starting GUI Server
10:11:17:WU01:FS00:0x11:mdrun_gpu returned
10:11:17:WU01:FS00:0x11:NANs detected on GPU
10:11:17:WU01:FS00:0x11:
10:11:17:WU01:FS00:0x11:Folding@home Core Shutdown: UNSTABLE_MACHINE
10:18:08:FS00:Shutting core down
10:19:09:WARNING:FS00:Killing WU01
10:19:09:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
10:26:05:WU00:FS01:0xa4:Completed 1325000 out of 2500000 steps (53%)
******************************* Date: 2013-07-19 *******************************
10:46:24:WU00:FS01:0xa4:Completed 1350000 out of 2500000 steps (54%)
10:46:43:WU01:FS00:Starting
10:46:43:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users//AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
10:46:43:WU01:FS00:Started FahCore on PID 5364
10:46:43:WU01:FS00:Core PID:2044
10:46:43:WU01:FS00:FahCore 0x11 started
10:46:43:WU01:FS00:0x11:
10:46:43:WU01:FS00:0x11:*------------------------------*
10:46:43:WU01:FS00:0x11:Folding@Home GPU Core
10:46:43:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
10:46:43:WU01:FS00:0x11:
10:46:43:WU01:FS00:0x11:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
10:46:43:WU01:FS00:0x11:Build host: amoeba
10:46:43:WU01:FS00:0x11:Board Type: Nvidia
10:46:43:WU01:FS00:0x11:Core :
10:46:43:WU01:FS00:0x11

reparing to commence simulation
10:46:43:WU01:FS00:0x11:- Looking at optimizations...
10:46:43:WU01:FS00:0x11

eleteFrameFiles: successfully deleted file=01/wudata_01.ckp
10:46:43:WU01:FS00:0x11:- Created dyn
10:46:43:WU01:FS00:0x11:- Files status OK
10:46:43:WU01:FS00:0x11:- Expanded 62894 -> 336763 (decompressed 535.4 percent)
10:46:43:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=62894 data_size=336763, decompressed_data_size=336763 diff=0
10:46:43:WU01:FS00:0x11:- Digital signature verified
10:46:43:WU01:FS00:0x11:
10:46:43:WU01:FS00:0x11

roject: 10501 (Run 32, Clone 0, Gen 1333)
10:46:43:WU01:FS00:0x11:
10:46:43:WU01:FS00:0x11:Assembly optimizations on if available.
10:46:43:WU01:FS00:0x11:Entering M.D.
10:46:49:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr: 1629719815 3940959380 699480707 98139461 3779876519
10:46:49:WU01:FS00:0x11:
10:46:49:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
10:46:49:WU01:FS00:0x11:
10:46:50:WU01:FS00:0x11:Working on Protein
10:46:52:WU01:FS00:0x11:Client config unavailable.
10:46:52:WU01:FS00:0x11:Starting GUI Server
10:46:52:WU01:FS00:0x11:mdrun_gpu returned
10:46:52:WU01:FS00:0x11:NANs detected on GPU
10:46:52:WU01:FS00:0x11:
10:46:52:WU01:FS00:0x11:Folding@home Core Shutdown: UNSTABLE_MACHINE
10:47:32:FS00:Shutting core down
10:48:33:WARNING:FS00:Killing WU01
10:48:33:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
11:06:42:WU00:FS01:0xa4:Completed 1375000 out of 2500000 steps (55%)
11:27:06:WU00:FS01:0xa4:Completed 1400000 out of 2500000 steps (56%)
11:38:39:WU01:FS00:Starting
11:38:39:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users//AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
11:38:39:WU01:FS00:Started FahCore on PID 3044
11:38:39:WU01:FS00:Core PID:416
11:38:39:WU01:FS00:FahCore 0x11 started
11:38:40:WU01:FS00:0x11:
11:38:40:WU01:FS00:0x11:*------------------------------*
11:38:40:WU01:FS00:0x11:Folding@Home GPU Core
11:38:40:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
11:38:40:WU01:FS00:0x11:
11:38:40:WU01:FS00:0x11:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
11:38:40:WU01:FS00:0x11:Build host: amoeba
11:38:40:WU01:FS00:0x11:Board Type: Nvidia
11:38:40:WU01:FS00:0x11:Core :
11:38:40:WU01:FS00:0x11

reparing to commence simulation
11:38:40:WU01:FS00:0x11:- Looking at optimizations...
11:38:40:WU01:FS00:0x11

eleteFrameFiles: successfully deleted file=01/wudata_01.ckp
11:38:40:WU01:FS00:0x11:- Created dyn
11:38:40:WU01:FS00:0x11:- Files status OK
11:38:40:WU01:FS00:0x11:- Expanded 62894 -> 336763 (decompressed 535.4 percent)
11:38:40:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=62894 data_size=336763, decompressed_data_size=336763 diff=0
11:38:40:WU01:FS00:0x11:- Digital signature verified
11:38:40:WU01:FS00:0x11:
11:38:40:WU01:FS00:0x11

roject: 10501 (Run 32, Clone 0, Gen 1333)
11:38:40:WU01:FS00:0x11:
11:38:40:WU01:FS00:0x11:Assembly optimizations on if available.
11:38:40:WU01:FS00:0x11:Entering M.D.
11:38:46:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr: 1629719815 3940959380 699480707 98139461 3779876519
11:38:46:WU01:FS00:0x11:
11:38:46:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
11:38:46:WU01:FS00:0x11:
11:38:47:WU01:FS00:0x11:Working on Protein
11:38:49:WU01:FS00:0x11:Client config unavailable.
11:38:49:WU01:FS00:0x11:Starting GUI Server
11:38:49:WU01:FS00:0x11:mdrun_gpu returned
11:38:49:WU01:FS00:0x11:NANs detected on GPU
11:38:49:WU01:FS00:0x11:
11:38:49:WU01:FS00:0x11:Folding@home Core Shutdown: UNSTABLE_MACHINE
11:42:26:FS00:Shutting core down
11:43:27:WARNING:FS00:Killing WU01
11:43:27:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
11:47:25:WU00:FS01:0xa4:Completed 1425000 out of 2500000 steps (57%)
11:52:12:WU01:FS00:Starting
11:52:12:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users//AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
11:52:12:WU01:FS00:Started FahCore on PID 1052
11:52:12:WU01:FS00:Core PID:2612
11:52:12:WU01:FS00:FahCore 0x11 started
11:52:12:WU01:FS00:0x11:
11:52:12:WU01:FS00:0x11:*------------------------------*
11:52:12:WU01:FS00:0x11:Folding@Home GPU Core
11:52:12:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
11:52:12:WU01:FS00:0x11:
11:52:12:WU01:FS00:0x11:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
11:52:12:WU01:FS00:0x11:Build host: amoeba
11:52:12:WU01:FS00:0x11:Board Type: Nvidia
11:52:12:WU01:FS00:0x11:Core :
11:52:12:WU01:FS00:0x11

reparing to commence simulation
11:52:12:WU01:FS00:0x11:- Looking at optimizations...
11:52:12:WU01:FS00:0x11

eleteFrameFiles: successfully deleted file=01/wudata_01.ckp
11:52:12:WU01:FS00:0x11:- Created dyn
11:52:12:WU01:FS00:0x11:- Files status OK
11:52:12:WU01:FS00:0x11:- Expanded 62894 -> 336763 (decompressed 535.4 percent)
11:52:12:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=62894 data_size=336763, decompressed_data_size=336763 diff=0
11:52:12:WU01:FS00:0x11:- Digital signature verified
11:52:12:WU01:FS00:0x11:
11:52:12:WU01:FS00:0x11

roject: 10501 (Run 32, Clone 0, Gen 1333)
11:52:12:WU01:FS00:0x11:
11:52:12:WU01:FS00:0x11:Assembly optimizations on if available.
11:52:12:WU01:FS00:0x11:Entering M.D.
11:52:18:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr: 1629719815 3940959380 699480707 98139461 3779876519
11:52:18:WU01:FS00:0x11:
11:52:18:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
11:52:18:WU01:FS00:0x11:
11:52:19:WU01:FS00:0x11:Working on Protein
11:52:21:WU01:FS00:0x11:Client config unavailable.
11:52:21:WU01:FS00:0x11:mdrun_gpu returned
11:52:21:WU01:FS00:0x11:NANs detected on GPU
11:52:21:WU01:FS00:0x11:
11:52:21:WU01:FS00:0x11:Folding@home Core Shutdown: UNSTABLE_MACHINE
12:07:12:FS00:Shutting core down
12:07:43:WU00:FS01:0xa4:Completed 1450000 out of 2500000 steps (58%)
12:08:13:WARNING:FS00:Killing WU01
12:08:13:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
12:14:22:WU01:FS00:Starting
12:14:22:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users//AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
12:14:22:WU01:FS00:Started FahCore on PID 5424
12:14:22:WU01:FS00:Core PID:6516
12:14:22:WU01:FS00:FahCore 0x11 started
12:14:22:WU01:FS00:0x11:
12:14:22:WU01:FS00:0x11:*------------------------------*
12:14:22:WU01:FS00:0x11:Folding@Home GPU Core
12:14:22:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
12:14:22:WU01:FS00:0x11:
12:14:22:WU01:FS00:0x11:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
12:14:22:WU01:FS00:0x11:Build host: amoeba
12:14:22:WU01:FS00:0x11:Board Type: Nvidia
12:14:22:WU01:FS00:0x11:Core :
12:14:22:WU01:FS00:0x11

reparing to commence simulation
12:14:22:WU01:FS00:0x11:- Looking at optimizations...
12:14:22:WU01:FS00:0x11

eleteFrameFiles: successfully deleted file=01/wudata_01.ckp
12:14:22:WU01:FS00:0x11:- Created dyn
12:14:22:WU01:FS00:0x11:- Files status OK
12:14:22:WU01:FS00:0x11:- Expanded 62894 -> 336763 (decompressed 535.4 percent)
12:14:22:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=62894 data_size=336763, decompressed_data_size=336763 diff=0
12:14:22:WU01:FS00:0x11:- Digital signature verified
12:14:22:WU01:FS00:0x11:
12:14:22:WU01:FS00:0x11

roject: 10501 (Run 32, Clone 0, Gen 1333)
12:14:22:WU01:FS00:0x11:
12:14:22:WU01:FS00:0x11:Assembly optimizations on if available.
12:14:22:WU01:FS00:0x11:Entering M.D.
12:14:28:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr: 1629719815 3940959380 699480707 98139461 3779876519
12:14:28:WU01:FS00:0x11:
12:14:28:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
12:14:28:WU01:FS00:0x11:
12:14:29:WU01:FS00:0x11:Working on Protein
12:14:31:WU01:FS00:0x11:Client config unavailable.
12:14:31:WU01:FS00:0x11:Starting GUI Server
12:14:31:WU01:FS00:0x11:mdrun_gpu returned
12:14:31:WU01:FS00:0x11:NANs detected on GPU
12:14:31:WU01:FS00:0x11:
12:14:31:WU01:FS00:0x11:Folding@home Core Shutdown: UNSTABLE_MACHINE
12:24:22:FS00:Shutting core down
12:28:00:WU00:FS01:0xa4:Completed 1475000 out of 2500000 steps (59%)
12:31:02:WARNING:FS00:Killing WU01
12:31:02:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
12:31:41:13:127.0.0.1:New Web connection
12:31:52:WARNING:Exception: 18:127.0.0.1: Send error: 10053: An established connection was aborted by the software in your host machine.
12:31:56:FS01:Shutting core down
12:31:56:WU01:FS00:Starting
12:31:56:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/L/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/NVIDIA/G80/Core_11.fah/FahCore_11.exe -dir 01 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -gpu 0 -gpu-vendor nvidia
12:31:56:WU01:FS00:Started FahCore on PID 964
12:31:56:WU01:FS00:Core PID:5824
12:31:56:WU01:FS00:FahCore 0x11 started
12:31:57:WU01:FS00:0x11:
12:31:57:WU01:FS00:0x11:*------------------------------*
12:31:57:WU01:FS00:0x11:Folding@Home GPU Core
12:31:57:WU01:FS00:0x11:Version 1.31 (Tue Sep 15 10:57:42 PDT 2009)
12:31:57:WU01:FS00:0x11:
12:31:57:WU01:FS00:0x11:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
12:31:57:WU01:FS00:0x11:Build host: amoeba
12:31:57:WU01:FS00:0x11:Board Type: Nvidia
12:31:57:WU01:FS00:0x11:Core :
12:31:57:WU01:FS00:0x11

reparing to commence simulation
12:31:57:WU01:FS00:0x11:- Looking at optimizations...
12:31:57:WU01:FS00:0x11

eleteFrameFiles: successfully deleted file=01/wudata_01.ckp
12:31:57:WU01:FS00:0x11:- Created dyn
12:31:57:WU01:FS00:0x11:- Files status OK
12:31:57:WU01:FS00:0x11:- Expanded 62894 -> 336763 (decompressed 535.4 percent)
12:31:57:WU01:FS00:0x11:Called DecompressByteArray: compressed_data_size=62894 data_size=336763, decompressed_data_size=336763 diff=0
12:31:57:WU01:FS00:0x11:- Digital signature verified
12:31:57:WU01:FS00:0x11:
12:31:57:WU01:FS00:0x11

roject: 10501 (Run 32, Clone 0, Gen 1333)
12:31:57:WU01:FS00:0x11:
12:31:57:WU01:FS00:0x11:Assembly optimizations on if available.
12:31:57:WU01:FS00:0x11:Entering M.D.
12:32:02:WU01:FS00:0x11:Tpr hash 01/wudata_01.tpr: 1629719815 3940959380 699480707 98139461 3779876519
12:32:02:WU01:FS00:0x11:
12:32:02:WU01:FS00:0x11:Calling fah_main args: 14 usage=100
12:32:02:WU01:FS00:0x11:
12:32:03:WU01:FS00:0x11:Working on Protein
12:32:03:WU00:FS01:0xa4:Client no longer detected. Shutting down core
12:32:03:WU00:FS01:0xa4:
12:32:03:WU00:FS01:0xa4:Folding@home Core Shutdown: CLIENT_DIED
12:32:03:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
12:32:04:WU00:FS01:Starting
12:32:04:WARNING:WU00:FS01:Changed SMP threads from 11 to 12 this can cause some work units to fail
12:32:04:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 703 -lifeline 4112 -checkpoint 10 -np 12
12:32:04:WU00:FS01:Started FahCore on PID 5752
12:32:04:WU00:FS01:Core PID:6708
12:32:04:WU00:FS01:FahCore 0xa4 started
12:32:04:WU01:FS00:0x11:Client config unavailable.
12:32:04:WU00:FS01:0xa4:
12:32:04:WU00:FS01:0xa4:*------------------------------*
12:32:04:WU00:FS01:0xa4:Folding@Home Gromacs GB Core
12:32:04:WU00:FS01:0xa4:Version 2.27 (Dec. 15, 2010)
12:32:04:WU00:FS01:0xa4:
12:32:04:WU00:FS01:0xa4

reparing to commence simulation
12:32:04:WU00:FS01:0xa4:- Looking at optimizations...
12:32:04:WU00:FS01:0xa4:- Files status OK
12:32:04:WU00:FS01:0xa4:- Expanded 549214 -> 848748 (decompressed 154.5 percent)
12:32:04:WU00:FS01:0xa4:Called DecompressByteArray: compressed_data_size=549214 data_size=848748, decompressed_data_size=848748 diff=0
12:32:04:WU00:FS01:0xa4:- Digital signature verified
12:32:04:WU00:FS01:0xa4:
12:32:04:WU00:FS01:0xa4

roject: 7647 (Run 178, Clone 0, Gen 140)
12:32:04:WU00:FS01:0xa4:
12:32:04:WU00:FS01:0xa4:Assembly optimizations on if available.
12:32:04:WU00:FS01:0xa4:Entering M.D.
12:32:04:WU01:FS00:0x11:Starting GUI Server
12:32:04:WU01:FS00:0x11:mdrun_gpu returned
12:32:04:WU01:FS00:0x11:NANs detected on GPU
12:32:04:WU01:FS00:0x11:
12:32:04:WU01:FS00:0x11:Folding@home Core Shutdown: UNSTABLE_MACHINE
12:32:10:WU00:FS01:0xa4:Using Gromacs checkpoints
12:32:10:WU00:FS01:0xa4:Mapping NT from 12 to 12
12:32:10:WU00:FS01:0xa4:Resuming from checkpoint
12:32:10:WU00:FS01:0xa4:Verified 00/wudata_01.log
12:32:10:WU00:FS01:0xa4:Verified 00/wudata_01.trr
12:32:10:WU00:FS01:0xa4:Verified 00/wudata_01.xtc
12:32:10:WU00:FS01:0xa4:Verified 00/wudata_01.edr
12:32:10:WU00:FS01:0xa4:Completed 1468570 out of 2500000 steps (58%)

Hope that helps

Latest drivers as of yesterday.

Biffa · 19 Jul 2013 at 13:42

Hi yes the card is definitely crashing, it might be better if you don't use it, as I doubt it will do more than take away from the CPU PPD at any rate.

However if you do want to try.

Is the GPU overclocked?
What drivers are you using?
What temps is the GPU running at?
Make sure you stop the CPU folding while you test the GPU

Human-Shield · 19 Jul 2013 at 15:26

OK CHEERS,

its not clocked at all, using latest nvidia drivers. rinning at 65-70 deg