Project

General

Profile

Bug #2295 ยป MemLeakLog.txt

/var/log/message file contents - Amber Herold, 03/25/2013 04:24 PM

 
Mar 24 06:28:39 guppy-14 pbs_mom: LOG_ERROR::req_cpyfile, #012#012Unable to copy
file /var/spool/torque/spool/122731.guppy.ami.scripps.edu.OU to dlyumkis@guppy.
ami.scripps.edu:/ami/archive/18/appion/11jun03a/syntheticData/fullstack_emdb_179
8_1799_5030_10k_new/frealign/perturbed_alignment_classification/0.013/scratch/re
construct_1_2.o122731#012*** error from copy#012Host key verification failed.#01
5#012lost connection#012*** end error output#012Output retained on that host in:
/var/spool/torque/undelivered/122731.guppy.ami.scripps.edu.OU
Mar 24 06:30:41 guppy-14 kernel: xmipp_mpi_class invoked oom-killer: gfp_mask=0x
201da, order=0, oom_adj=0, oom_score_adj=0
Mar 24 06:30:41 guppy-14 kernel: xmipp_mpi_class cpuset=/ mems_allowed=0
Mar 24 06:30:41 guppy-14 kernel: Pid: 19343, comm: xmipp_mpi_class Not tainted 2
.6.32-279.11.1.el6.x86_64 #1
Mar 24 06:30:41 guppy-14 kernel: Call Trace:
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff810c4c81>] ? cpuset_print_task_mems_a
llowed+0x91/0xb0
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff811173f0>] ? dump_header+0x90/0x1b0
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8128017b>] ? __bitmap_intersects+0xb/
0xa0
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81214a3c>] ? security_real_capable_no
audit+0x3c/0x70
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81117872>] ? oom_kill_process+0x82/0x
2a0
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff811177b1>] ? select_bad_process+0xe1/
0x120
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81117cb0>] ? out_of_memory+0x220/0x3c
0
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff811279ce>] ? __alloc_pages_nodemask+0
x89e/0x940
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8115c54a>] ? alloc_pages_current+0xaa
/0x110
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff811147f7>] ? __page_cache_alloc+0x87/
0x90
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81012bd9>] ? read_tsc+0x9/0x20
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8112a41b>] ? __do_page_cache_readahea
d+0xdb/0x210
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8112a571>] ? ra_submit+0x21/0x30
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81115b23>] ? filemap_fault+0x4c3/0x50
0
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81170d63>] ? swap_cgroup_record+0xa3/
0xc0
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8113ef14>] ? __do_fault+0x54/0x510
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8116d55b>] ? mem_cgroup_uncharge_swap
cache+0x2b/0x60
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8115337d>] ? swapcache_free+0x4d/0x70
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8113f4c7>] ? handle_pte_fault+0xf7/0xb50
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81140104>] ? handle_mm_fault+0x1e4/0x2b0
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8105b4e3>] ? perf_event_task_sched_out+0x33/0x80
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff810444c9>] ? __do_page_fault+0x139/0x480
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81054c90>] ? __dequeue_entity+0x30/0x50
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff814fde00>] ? thread_return+0x4e/0x76e
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81012b89>] ? sched_clock+0x9/0x10
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81099155>] ? sched_clock_local+0x25/0x90
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff8150383e>] ? do_page_fault+0x3e/0xa0
Mar 24 06:30:41 guppy-14 kernel: [<ffffffff81500bf5>] ? page_fault+0x25/0x30
Mar 24 06:30:41 guppy-14 kernel: Mem-Info:
Mar 24 06:30:41 guppy-14 kernel: Node 0 DMA per-cpu:
Mar 24 06:30:41 guppy-14 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 4: hi: 0, btch: 1 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 5: hi: 0, btch: 1 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 6: hi: 0, btch: 1 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 7: hi: 0, btch: 1 usd: 0
Mar 24 06:30:41 guppy-14 kernel: Node 0 DMA32 per-cpu:
Mar 24 06:30:41 guppy-14 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 1: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 5: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: Node 0 Normal per-cpu:
Mar 24 06:30:41 guppy-14 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 1: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 5: hi: 186, btch: 31 usd: 27
Mar 24 06:30:41 guppy-14 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Mar 24 06:30:41 guppy-14 kernel: active_anon:3523036 inactive_anon:445695 isolated_anon:32
Mar 24 06:30:41 guppy-14 kernel: active_file:146 inactive_file:183 isolated_file:0
Mar 24 06:30:41 guppy-14 kernel: unevictable:7036 dirty:0 writeback:1741 unstable:0
Mar 24 06:30:41 guppy-14 kernel: free:33624 slab_reclaimable:2927 slab_unreclaimable:8906
Mar 24 06:30:41 guppy-14 kernel: mapped:760 shmem:0 pagetables:14276 bounce:0
Mar 24 06:30:41 guppy-14 kernel: Node 0 DMA free:15688kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB i
nactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15304kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB sh
mem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scan
ned:0 all_unreclaimable? yes
Mar 24 06:30:41 guppy-14 kernel: lowmem_reserve[]: 0 3255 16133 16133
Mar 24 06:30:41 guppy-14 kernel: Node 0 DMA32 free:65056kB min:13624kB low:17028kB high:20436kB active_anon:2329392kB inactive_anon:5453
40kB active_file:384kB inactive_file:404kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334016kB mlocked:0kB dirty:0kB
writeback:0kB mapped:260kB shmem:0kB slab_reclaimable:832kB slab_unreclaimable:232kB kernel_stack:72kB pagetables:2228kB unstable:0kB b
ounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 24 06:30:41 guppy-14 kernel: lowmem_reserve[]: 0 0 12877 12877
Mar 24 06:30:41 guppy-14 kernel: Node 0 Normal free:53752kB min:53892kB low:67364kB high:80836kB active_anon:11762752kB inactive_anon:12
37440kB active_file:200kB inactive_file:328kB unevictable:28144kB isolated(anon):128kB isolated(file):0kB present:13186560kB mlocked:117
92kB dirty:0kB writeback:6964kB mapped:2780kB shmem:0kB slab_reclaimable:10876kB slab_unreclaimable:35392kB kernel_stack:1984kB pagetabl
es:54876kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no
Mar 24 06:30:41 guppy-14 kernel: lowmem_reserve[]: 0 0 0 0
Mar 24 06:30:41 guppy-14 kernel: Node 0 DMA: 2*4kB 2*8kB 1*16kB 1*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15688
kB
Mar 24 06:30:41 guppy-14 kernel: Node 0 DMA32: 237*4kB 192*8kB 140*16kB 139*32kB 101*64kB 70*128kB 36*256kB 27*512kB 17*1024kB 0*2048kB
0*4096kB = 65044kB
Mar 24 06:30:41 guppy-14 kernel: Node 0 Normal: 998*4kB 648*8kB 410*16kB 278*32kB 198*64kB 93*128kB 19*256kB 0*512kB 0*1024kB 0*2048kB 0
*4096kB = 54072kB
Mar 24 06:30:41 guppy-14 kernel: 8205 total pagecache pages
Mar 24 06:30:41 guppy-14 kernel: 7232 pages in swap cache
Mar 24 06:30:41 guppy-14 kernel: Swap cache stats: add 3268939, delete 3261707, find 503826/572895
Mar 24 06:30:41 guppy-14 kernel: Free swap = 0kB
Mar 24 06:30:41 guppy-14 kernel: Total swap = 8232952kB
Mar 24 06:30:41 guppy-14 kernel: 4194303 pages RAM
Mar 24 06:30:41 guppy-14 kernel: 111350 pages reserved
Mar 24 06:30:41 guppy-14 kernel: 12977 pages shared
Mar 24 06:30:41 guppy-14 kernel: 4044025 pages non-shared
Mar 24 06:30:41 guppy-14 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Mar 24 06:30:41 guppy-14 kernel: [ 519] 0 519 2688 64 3 -17 -1000 udevd
Mar 24 06:30:41 guppy-14 kernel: [ 1295] 0 1295 2280 72 3 0 0 dhclient
Mar 24 06:30:41 guppy-14 kernel: [ 1343] 0 1343 62797 207 4 0 0 rsyslogd
Mar 24 06:30:41 guppy-14 kernel: [ 1375] 0 1375 2286 120 4 0 0 irqbalance
Mar 24 06:30:41 guppy-14 kernel: [ 1389] 32 1389 4743 155 1 0 0 rpcbind
Mar 24 06:30:41 guppy-14 kernel: [ 1407] 29 1407 5836 175 4 0 0 rpc.statd
Mar 24 06:30:41 guppy-14 kernel: [ 1446] 0 1446 5801 112 7 0 0 rpc.idmapd
Mar 24 06:30:41 guppy-14 kernel: [ 5263] 81 5263 6410 83 0 0 0 dbus-daemon
Mar 24 06:30:41 guppy-14 kernel: [ 5307] 0 5307 1019 127 4 0 0 acpid
Mar 24 06:30:41 guppy-14 kernel: [ 5316] 68 5316 7497 329 7 0 0 hald
Mar 24 06:30:41 guppy-14 kernel: [ 5317] 0 5317 4526 127 6 0 0 hald-runner
Mar 24 06:30:41 guppy-14 kernel: [ 5346] 0 5346 5055 116 0 0 0 hald-addon-inpu
Mar 24 06:30:41 guppy-14 kernel: [ 5357] 68 5357 4451 162 6 0 0 hald-addon-acpi
Mar 24 06:30:41 guppy-14 kernel: [ 5396] 0 5396 16018 83 1 -17 -1000 sshd
Mar 24 06:30:41 guppy-14 kernel: [ 5404] 38 5404 8600 256 0 0 0 ntpd
Mar 24 06:30:41 guppy-14 kernel: [ 5480] 0 5480 19653 247 5 0 0 master
Mar 24 06:30:41 guppy-14 kernel: [ 5495] 89 5495 20239 244 0 0 0 qmgr
Mar 24 06:30:41 guppy-14 kernel: [ 5504] 0 5504 27543 139 0 0 0 abrtd
Mar 24 06:30:41 guppy-14 kernel: [ 5512] 0 5512 27018 145 2 0 0 abrt-dump-oops
Mar 24 06:30:41 guppy-14 kernel: [ 5520] 0 5520 29302 154 1 0 0 crond
Mar 24 06:30:41 guppy-14 kernel: [ 5531] 0 5531 5363 70 1 0 0 atd
Mar 24 06:30:41 guppy-14 kernel: [ 5567] 0 5567 1015 117 6 0 0 mingetty
Mar 24 06:30:41 guppy-14 kernel: [ 5569] 0 5569 1015 117 5 0 0 mingetty
Mar 24 06:30:41 guppy-14 kernel: [ 5571] 0 5571 1015 117 4 0 0 mingetty
Mar 24 06:30:41 guppy-14 kernel: [ 5573] 0 5573 1015 117 4 0 0 mingetty
Mar 24 06:30:41 guppy-14 kernel: [ 5575] 0 5575 1015 117 7 0 0 mingetty
Mar 24 06:30:41 guppy-14 kernel: [ 5577] 0 5577 1015 117 0 0 0 mingetty
Mar 24 06:30:41 guppy-14 kernel: [ 5580] 0 5580 2687 60 3 -17 -1000 udevd
Mar 24 06:30:41 guppy-14 kernel: [ 5581] 0 5581 2687 57 1 -17 -1000 udevd
Mar 24 06:30:41 guppy-14 kernel: [ 5599] 0 5599 6909 141 1 -17 -1000 auditd
Mar 24 06:30:41 guppy-14 kernel: [ 5832] 0 5832 11676 7037 3 0 0 pbs_mom
Mar 24 06:30:41 guppy-14 kernel: [32725] 0 32725 53345 112 3 0 0 ypbind
Mar 24 06:30:41 guppy-14 kernel: [18218] 22365 18218 28145 208 6 0 0 bash
Mar 24 06:30:41 guppy-14 kernel: [18219] 22365 18219 3395 113 5 0 0 pbs_demux
Mar 24 06:30:41 guppy-14 kernel: [18250] 22365 18250 29295 189 5 0 0 121073.guppy.am
Mar 24 06:30:41 guppy-14 kernel: [18263] 22365 18263 32023 177 1 0 0 webcaller.py
Mar 24 06:30:41 guppy-14 kernel: [18264] 22365 18264 32055 174 7 0 0 appion
Mar 24 06:30:41 guppy-14 kernel: [18309] 22365 18309 150134 276 0 0 0 runXmippCL2D.py
Mar 24 06:30:41 guppy-14 kernel: [19339] 22365 19339 15178 238 0 0 0 mpirun
Mar 24 06:30:41 guppy-14 kernel: [19340] 22365 19340 438247 215934 7 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19341] 22365 19341 438245 216404 5 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19342] 22365 19342 438259 195965 4 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19343] 22365 19343 383864 178077 4 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19344] 22365 19344 438259 194609 2 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19345] 22365 19345 438241 200602 0 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19346] 22365 19346 438244 238041 1 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19347] 22365 19347 438269 235774 7 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19348] 22365 19348 438270 199895 1 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19349] 22365 19349 438273 145695 2 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19350] 22365 19350 438254 181728 6 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19351] 22365 19351 438258 244556 3 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19352] 22365 19352 438256 191033 5 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19353] 22365 19353 438267 231240 3 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19354] 22365 19354 438295 237957 6 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [19355] 22365 19355 438299 226895 0 0 0 xmipp_mpi_class
Mar 24 06:30:41 guppy-14 kernel: [27168] 89 27168 20196 246 5 0 0 pickup
Mar 24 06:30:41 guppy-14 kernel: [27793] 19565 27793 28143 301 0 0 0 bash
Mar 24 06:30:41 guppy-14 kernel: [27794] 19565 27794 28143 300 2 0 0 bash
Mar 24 06:30:41 guppy-14 kernel: [27795] 19565 27795 28143 299 2 0 0 bash
Mar 24 06:30:41 guppy-14 kernel: [27796] 19565 27796 28143 297 3 0 0 bash
Mar 24 06:30:41 guppy-14 kernel: [27949] 19565 27949 29252 320 5 0 0 122793.guppy.am
Mar 24 06:30:41 guppy-14 kernel: [27950] 19565 27950 29252 320 0 0 0 122791.guppy.am
Mar 24 06:30:41 guppy-14 kernel: [27951] 19565 27951 29252 320 0 0 0 122790.guppy.am
Mar 24 06:30:41 guppy-14 kernel: [27952] 19565 27952 29252 318 5 0 0 122792.guppy.am
Mar 24 06:30:41 guppy-14 kernel: [27991] 19565 27991 27691 252 1 0 0 mult_refine_n.c
Mar 24 06:30:41 guppy-14 kernel: [27992] 19565 27992 27691 247 1 0 0 mult_refine_n.c
Mar 24 06:30:41 guppy-14 kernel: [27998] 19565 27998 27691 249 2 0 0 mult_refine_n.c
Mar 24 06:30:41 guppy-14 kernel: [27999] 19565 27999 27691 245 1 0 0 mult_refine_n.c
Mar 24 06:30:41 guppy-14 kernel: [28208] 19565 28208 263026 139159 4 0 0 frealign_v9.exe
Mar 24 06:30:41 guppy-14 kernel: [28210] 19565 28210 263040 204096 5 0 0 frealign_v9.exe
Mar 24 06:30:41 guppy-14 kernel: [28211] 19565 28211 263026 137559 5 0 0 frealign_v9.exe
Mar 24 06:30:41 guppy-14 kernel: [28212] 19565 28212 263026 149053 1 0 0 frealign_v9.exe
Mar 24 06:30:41 guppy-14 kernel: Out of memory: Kill process 19344 (xmipp_mpi_class) score 57 or sacrifice child
Mar 24 06:30:41 guppy-14 kernel: Killed process 19344, UID 22365, (xmipp_mpi_class) total-vm:1753036kB, anon-rss:777412kB, file-rss:1024
kB
Mar 24 06:30:47 guppy-14 kernel: Pid 19345(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:47 guppy-14 kernel: Skipping core dump
Mar 24 06:30:47 guppy-14 kernel: Pid 19346(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:47 guppy-14 kernel: Skipping core dump
Mar 24 06:30:47 guppy-14 kernel: Pid 19347(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:47 guppy-14 kernel: Skipping core dump
Mar 24 06:30:48 guppy-14 kernel: Pid 19348(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:48 guppy-14 kernel: Skipping core dump
Mar 24 06:30:48 guppy-14 kernel: Pid 19349(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:48 guppy-14 kernel: Skipping core dump
Mar 24 06:30:48 guppy-14 kernel: Pid 19350(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:48 guppy-14 kernel: Skipping core dump
Mar 24 06:30:48 guppy-14 kernel: Pid 19351(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:48 guppy-14 kernel: Skipping core dump
Mar 24 06:30:48 guppy-14 kernel: Pid 19352(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:48 guppy-14 kernel: Skipping core dump
Mar 24 06:30:48 guppy-14 kernel: Pid 19353(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:48 guppy-14 kernel: Skipping core dump
Mar 24 06:30:48 guppy-14 kernel: Pid 19354(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:48 guppy-14 kernel: Skipping core dump
Mar 24 06:30:48 guppy-14 kernel: Pid 19355(xmipp_mpi_class) over core_pipe_limit
Mar 24 06:30:48 guppy-14 kernel: Skipping core dump
Mar 24 06:30:49 guppy-14 abrt[28219]: Not dumping repeating crash in '/home/export/sw/opt/Xmipp/bin/xmipp_mpi_class_averages'
Mar 24 06:30:49 guppy-14 abrt[28220]: Not dumping repeating crash in '/home/export/sw/opt/Xmipp/bin/xmipp_mpi_class_averages'
Mar 24 06:30:49 guppy-14 abrt[28217]: Not dumping repeating crash in '/home/export/sw/opt/Xmipp/bin/xmipp_mpi_class_averages'
Mar 24 06:31:52 guppy-14 abrt[28218]: Saved core dump of pid 19341 (/home/export/sw/opt/Xmipp/bin/xmipp_mpi_class_averages) to /var/spoo
l/abrt/ccpp-2013-03-24-06:30:49-19341 (1680769024 bytes)
Mar 24 06:31:52 guppy-14 abrtd: Directory 'ccpp-2013-03-24-06:30:49-19341' creation detected
Mar 24 06:31:54 guppy-14 abrtd: Executable '/home/export/sw/opt/Xmipp/bin/xmipp_mpi_class_averages' doesn't belong to any package
Mar 24 06:31:54 guppy-14 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-03-24-06:30:49-19341' exited with 1
Mar 24 06:31:54 guppy-14 abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2013-03-24-06:30:49-19341, deleting
Mar 24 06:35:45 guppy-14 pbs_mom: LOG_ERROR::sys_copy, command '/usr/bin/scp -rpB /var/spool/torque/spool/122790.guppy.ami.scripps.edu.O
U dlyumkis@guppy.ami.scripps.edu:/ami/archive/18/appion/11jun03a/syntheticData/fullstack_emdb_1798_1799_5030_10k_new/frealign/perturbed_
alignment_classification/0.006/scratch/refine_8001_1.o122790' failed with status=1, giving up after 4 attempts
Mar 24 06:35:45 guppy-14 pbs_mom: LOG_ERROR::req_cpyfile, Unable to copy file /var/spool/torque/spool/122790.guppy.ami.scripps.edu.OU to
dlyumkis@guppy.ami.scripps.edu:/ami/archive/18/appion/11jun03a/syntheticData/fullstack_emdb_1798_1799_5030_10k_new/frealign/perturbed_a
lignment_classification/0.006/scratch/refine_8001_1.o122790
Mar 24 06:35:45 guppy-14 pbs_mom: LOG_ERROR::req_cpyfile, #012#012Unable to copy file /var/spool/torque/spool/122790.guppy.ami.scripps.e
du.OU to dlyumkis@guppy.ami.scripps.edu:/ami/archive/18/appion/11jun03a/syntheticData/fullstack_emdb_1798_1799_5030_10k_new/frealign/per
turbed_alignment_classification/0.006/scratch/refine_8001_1.o122790#012*** error from copy#012Host key verification failed.#015#012lost
connection#012*** end error output#012Output retained on that host in: /var/spool/torque/undelivered/122790.guppy.ami.scripps.edu.OU
Mar 24 10:06:28 guppy-14 pbs_mom: LOG_ERROR::req_cpyfile, Unable to copy file /var/spool/torque/spool/123464.guppy.ami.scripps.edu.OU to
dlyumkis@guppy.ami.scripps.edu:/ami/archive/18/appion/11jun03a/syntheticData/fullstack_emdb_1798_1799_5030_10k_new/frealign/perturbed_a
lignment_classification/0.013/scratch/reconstruct_1_3.o123464

...

Mar 24 10:06:28 guppy-14 pbs_mom: LOG_ERROR::req_cpyfile, #012#012Unable to copy file /var/spool/torque/spool/123464.guppy.ami.scripps.e
du.OU to dlyumkis@guppy.ami.scripps.edu:/ami/archive/18/appion/11jun03a/syntheticData/fullstack_emdb_1798_1799_5030_10k_new/frealign/per
turbed_alignment_classification/0.013/scratch/reconstruct_1_3.o123464#012*** error from copy#012Host key verification failed.#015#012los
t connection#012*** end error output#012Output retained on that host in: /var/spool/torque/undelivered/123464.guppy.ami.scripps.edu.OU
Mar 24 10:10:09 guppy-14 abrt: detected unhandled Python exception in '/opt/myamisnap/bin/dogPicker.py'
Mar 24 10:10:09 guppy-14 abrtd: New client connected
Mar 24 10:10:09 guppy-14 abrt-server[8319]: Saved Python crash dump of pid 6809 to /var/spool/abrt/pyhook-2013-03-24-10:10:09-6809
Mar 24 10:10:09 guppy-14 abrtd: Directory 'pyhook-2013-03-24-10:10:09-6809' creation detected
Mar 24 10:10:10 guppy-14 abrtd: Executable '/opt/myamisnap/bin/dogPicker.py' doesn't belong to any package
Mar 24 10:10:10 guppy-14 abrtd: 'post-create' on '/var/spool/abrt/pyhook-2013-03-24-10:10:09-6809' exited with 1
    (1-1/1)