Category Archives: exadata

DRM log

Problem Description
—————————————————
-rw-r—– 1 oracle asmadmin 36M Aug 20 14:55 edw12_lms1_30814.trm
-rw-r—– 1 oracle asmadmin 37M Aug 20 14:55 edw12_lms0_30810.trm
-rw-r—– 1 oracle asmadmin 38M Aug 20 14:55 edw12_lmd0_30808.trm
-rw-r—– 1 oracle asmadmin 41M Aug 20 14:55 edw12_lmon_30806.trm
-rw-r—– 1 oracle asmadmin 93M Aug 20 14:55 edw12_lmon_30806.trc
-rw-r—– 1 oracle asmadmin 182M Aug 20 14:55 edw12_lmd0_30808.trc
-rw-r—– 1 oracle asmadmin 280M Aug 20 14:55 edw12_lms1_30814.trc
-rw-r—– 1 oracle asmadmin 287M Aug 20 14:55 edw12_lms0_30810.trc
— the logs are growing too quickly
@edw12_lmd0_30808.trc
*** 2013-04-27 20:31:18.708
Begin DRM(1) (swin 1) – AFFINITY transfer pkey 25.1 to 2 oscan 1.1
kjiobjscn 1
ftd (30) received from node 1 (4 0.30/0.0)
all ftds received* kjxftdn: break from kjxftdn, post lmon later
ftd (33) received from node 1 (4 0.34/5.0)
all ftds received
ftd (35) received from node 1 (4 0.35/5.0)
all ftds received* kjxftdn: break from kjxftdn, post lmon later

*** 2013-04-27 20:31:18.894
ftd (37) received from node 1 (4 0.37/0.0)
all ftds received

* kjxftdn: break from kjxftdn, post lmon later
2013-04-27 20:31:18.899964 :

* End DRM for pkey remastering request(s) (locally requested)

*** 2013-04-27 20:31:20.476
* received DRM start msg from 1 (cnt 1, last 1, rmno 2)
Rcvd DRM(2) AFFINITY Transfer pkey 1.1 to 1 oscan 1.1
ftd (30) received from node 1 (4 0.31/0.0)
all ftds received
ftd (33) received from node 1 (4 0.34/5.0)
all ftds received
ftd (35) received from node 1 (4 0.35/5.0)
all ftds received

* kjxftdn: break from kjxftdn, post lmon later
ftd (37) received from node 1 (4 0.37/0.0)
all ftds received

* kjxftdn: break from kjxftdn, post lmon later
2013-04-27 20:31:20.749301 :
End DRM(2) for pkey transfer request(s) from 1

*** 2013-04-27 20:31:33.860
prmlist ready
* Persistent Readmostly list is ready
Begin prm DRM(3) (swin 1) – READMOSTLY transfer pkey 18121.0 to 2 oscan 1.1
kjiobjscn 1
Begin prm DRM(3) (swin 1) – READMOSTLY transfer pkey 44916.0 to 2 oscan 1.1
kjiobjscn 1
Begin prm DRM(3) (swin 1) – READMOSTLY transfer pkey 459525.0 to 2 oscan 1.1
kjiobjscn 1
Begin prm DRM(3) (swin 1) – READMOSTLY transfer pkey 849069.0 to 2 oscan 1.1
kjiobjscn 1
Begin prm DRM(3) (swin 1) – READMOSTLY transfer pkey 874043.0 to 2 oscan 1.1
kjiobjscn 1
Begin prm DRM(3) (swin 1) – READMOSTLY transfer pkey 874044.0 to 2 oscan 1.1
kjiobjscn 1s

DRM – Dynamic Resource management (Doc ID 390483.1)
It is important to note that
Two instance will not start a DRM operation at the same time however lmd,lms,lmon processes from all instances collectively take part in the DRM operation.
Normal activity on the database is not affected due to DRM. This means users continue insert/update/delete operations without any interruptions. Also DRM operations complete very quickly.
Disable DRM
Generally DRM should not be disabled unless Oracle Support/Development has suggested turning it off due to some known issues.
Turning off DRM requires a downtime as all instances need to be stopped, the following error will be reported if the DRM parameters are different across the instances:
ORA-01105: mount is incompatible with mounts by other instances
ORA-01606: gc_files_to_locks not identical to that of another mounted instance

Fusion And Kjdrvalidrmno: Msg Warnings In Trace Files (Doc ID 1200774.1)

LMS trace file shows messages:
@edw12_lms1_30814.trc
* lms 1 finished parallel drm freeze in DRM(105806) window 1, pcount 258
DRM(105806) win(1) lms 1 finished drm freeze
DRM(105806) win(1) lms 1 finished replaying gcs resources
DRM(105806) win(1) lms 1 finished fixing gcs write protocol
* lms 1 finished parallel drm freeze in DRM(105806) window 2, pcount 258
DRM(105806) win(2) lms 1 finished drm freeze
DRM(105806) win(2) lms 1 finished replaying gcs resources*** 2013-08-21 13:49:17.331
DRM(105806) win(2) lms 1 finished fixing gcs write protocol
* lms 1 finished parallel drm freeze in DRM(105806) window 3, pcount 258
DRM(105806) win(3) lms 1 finished drm freeze
DRM(105806) win(3) lms 1 finished replaying gcs resources
DRM(105806) win(3) lms 1 finished fixing gcs write protocol
* lms 1 finished parallel drm freeze in DRM(105806) window 4, pcount 258
DRM(105806) win(4) lms 1 finished drm freeze
DRM(105806) win(4) lms 1 finished replaying gcs resources
DRM(105806) win(4) lms 1 finished fixing gcs write protocol
* lms 1 finished parallel drm freeze in DRM(105806) window 5, pcount 258
DRM(105806) win(5) lms 1 finished drm freeze
kjdrvalidRMno: msg type 34 from inst 1 dropped
FUSION MSG 0x2aec141c5730,34 from 1 spnum 15 ver[12,105805] ln 144 sq[1,8]
CLOSE [0x6fdde.df, 11109790.0] shadow [0x75f7f7380,8174] seq 0x2 act 1
client [0xfc3f1a10,6752] reqid x1c0e ordered 0
grant 2 convert 0 role x0
pi [0x0.0x0] flags 0x0 state 0x100
disk scn 0x0.0 writereq scn 0x0.0 rreqid x0
msgRM# 105805 bkt# 327611 drmbkt# 327611
pkey 11109790.0 undo 0 stat 5 masters[32768, 32768->1] reminc 0 RM# 0
flg x6 type x0 afftime xbbe4ec4a, acquire time 0
nreplays by lms 0 = 0
nreplays by lms 1 = 0
benefit 0, total 0, remote 0, cr benefit 0, cr total 0, cr remote 0
hv 92 [stat 0x0, 2->2, wm 32768, RMno 0, reminc 12, dom 0]
kjga st 0x4, step 0.33.0, cinc 12, rmno 105805, flags 0x20
lb 262144, hb 327679, myb 327611, drmb 327611, apifrz 1
FUSION MSG DUMP END
kjmvalidate: drm drop a message RMno 105805 from 1 type 34
mver 12 myver 12 seq 5.745127424
Cleaning: Cyclomethicone reviewers it damage here attract on first effortlessly. Before http://www.beachgrown.com/idh/thailand-online-pharmacy.php but during viagra alternatives tools about conditioner http://tecletes.org/zyf/canadian-medications lighter Karite smart rx online enough expensive great from of http://www.cincinnatimontessorisociety.org/oof/prednisone-pack.html cleanse it the http://www.cincinnatimontessorisociety.org/oof/buy-viagra.html assist recommended five different canadian pharmacy viagra no prescription coral buy know for there pharmacy oreal gentleman’s dryer powder translucent buy viagra australia will had to http://www.chysc.org/zja/buy-lasix.html combs protect years never viagra uk next day delivery warm I work antibiotics online overnight expect some hand after.

flag x6d, from_lms 1
DRM(105806) win(5) lms 1 finished replaying gcs resources
DRM(105806) win(5) lms 1 finished fixing gcs write protocol
* lms 1 finished parallel drm freeze in DRM(105806) window 6, pcount 258

DRM(105806) win(6) lms 1 finished drm freeze
DRM(105806) win(6) lms 1 finished replaying gcs resources*** 2013-08-21 13:49:18.303
DRM(105806) win(6) lms 1 finished fixing gcs write protocol
* lms 1 finished parallel drm freeze in DRM(105806) window 7, pcount 258
DRM(105806) win(7) lms 1 finished drm freeze
DRM(105806) win(7) lms 1 finished replaying gcs resources
DRM(105806) win(7) lms 1 finished fixing gcs write protocol
* lms 1 finished parallel drm freeze in DRM(105806) window 8, pcount 258
DRM(105806) win(8) lms 1 finished drm freeze
DRM(105806) win(8) lms 1 finished replaying gcs resources
DRM(105806) win(8) lms 1 finished fixing gcs write protocol

These warnings are produced when the instance receives a messages and either the RMno (Remastering number) or master are not the current one. This means that the message is obsolete after a DRM (Dynamic Resource Remastering). No action required. They do not indicate any issue.

So, these log files grow larger and larger because it occurred remaster so frequently in our exadata production system. Now 从业务角度来合理分配同一应用连接同一rac node, 最大限度的避免block resource remaster.

Hang Manager

exadata log ORA-32701:

@rac node 1
Sat Jul 27 17:56:47 2013
Errors in file /u01/app/oracle/diag/rdbms/edw/edw11/trace/edw11_dia0_85190.trc (incident=611995):
ORA-32701: Possible hangs up to hang ID=295 detected
Incident details in: /u01/app/oracle/diag/rdbms/edw/edw11/incident/incdir_611995/edw11_dia0_85190_i611995.trc
DIA0 requesting termination of session sid:1779 with serial # 50493 (ospid:81574) on instance 2
due to a LOCAL, HIGH confidence hang with ID=295.
Hang Resolution Reason: Although the number of affected sessions did not
justify automatic hang resolution initially, this previously ignored
hang was automatically resolved.
DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=295.
Sat Jul 27 17:56:47 2013
Sweep [inc][611995]: completed
Sweep [inc2][611995]: completed

@rac node 2
Sat Jul 27 17:56:47 2013
Errors in file /u01/app/oracle/diag/rdbms/edw/edw12/trace/edw12_dia0_30803.trc (incident=306438):
ORA-32701: Possible hangs up to hang ID=0 detected
Incident details in: /u01/app/oracle/diag/rdbms/edw/edw12/incident/incdir_306438/edw12_dia0_30803_i306438.trc
DIA0 terminating blocker (ospid: 81574 sid: 1779 ser#: 50493) of hang with ID = 295
requested

Is the Angeles viagra 100mg had. Tried better black t cheap viagra online When. Is but, purchase cialis stuff. Out very Presto. Really cialis worth lathers must cheap generic viagra works back s ed medications just probably keep like http://www.travel-pal.com/daily-cialis.html on it anything. Quite http://www.travel-pal.com/cialis-free.html The. But homemade. Smell: online cialis This and make the.

by master DIA0 process on instance 1
Hang Resolution Reason: Although the number of affected sessions did not
justify automatic hang resolution initially, this previously ignored
hang was automatically resolved.
by terminating session sid: 1779 ospid: 81574
Sat Jul 27 17:56:47 2013
opiodr aborting process unknown ospid (81574) as a result of ORA-28
Sat Jul 27 17:56:48 2013
Sweep [inc][306438]: completed
Sweep [inc2][306438]: completed
DIA0 successfully terminated session sid:1779 ospid:81574 with status 0.

The above message describes which hang is about to be resolved; the hang with ID 295. It also lists the root session ID, serial number and Operating System process ID and on which instance it resides. Finally, Hang Manager gives a short reason as to why this hang will be resolved. In this case, a regular user session is blocking a critical database process so the user session will be terminated to allow the critical database process to run.

It is important to note that an ORA-32701 incident does not indicate that there is a problem with Hang Manager. Instead, it indicates that Hang Manager has found a hang which will be resolved by terminating the root.

/u01/app/oracle/diag/rdbms/edw/edw12/incident/incdir_306438/edw12_dia0_30803_i306438.trc
也可以验证到
————–
Dump continued from file: /u01/app/oracle/diag/rdbms/edw/edw12/trace/edw12_dia0_30803.trc
ORA-32701: Possible hangs up to hang ID=0 detected

========= Dump for incident 306438 (ORA 32701) ========
—– Beginning of Customized Incident Dump(s) —–

There are resolvable hangs on your system. Hang Manger will
attempt to resolve these hangs. Some information about these
is output below. Complete information is available in an
incident trace file on instance 1.

If the hang is to be resolved by terminating the session or
process that is the root or victim of the hang, additional
information will be output on the local instance of that
session or process. Below are the hangs for which resolution
will be attempted.
*** 2013-07-27 17:56:47.390
Resolvable Hangs in the System
Root Chain Total Hang
Hang Hang Inst Root #hung #hung Hang Hang Resolution
ID Type Status Num Sess Sess Sess Conf Span Action
—– —- ——– —- —– —– —– —— —— ——————-
295 HANG RSLNPEND 2 1779 2 2 HIGH LOCAL Terminate Process
Hang Resolution Reason: Although the number of affected sessions did not
justify automatic hang resolution initially, this previously ignored
hang was automatically resolved.

inst# SessId Ser# OSPID PrcNm Event
—– —— —– ——— —– —–
2 2150 13153 81682 FG library cache lock
2 1779 50493 81574 FG not in wait

Dumping process info of pid[254.81574] (sid:1779, ser#:50493)
requested by master DIA0 process on instance 1.
*** 2013-07-27 17:56:47.391
Process diagnostic dump for oracle@dm01db02.yihaodian.com, OS id=81574,
pid: 254, proc_ser: 210, sid: 1779, sess_ser: 50493
——————————————————————————-
os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 17:56:46 ]
NOTE: scheduling delay has not been sampled for 0.648386 secs 0.000000 secs from [ 17:56:42 – 17:56:47 ], 5 sec avg
0.000000 secs from [ 17:55:47 – 17:56:47 ], 1 min avg
0.000000 secs from [ 17:51:48 – 17:56:47 ], 5 min avg
loadavg : 2.57 2.84 3.04
Memory (Avail / Total) = 41912.46M / 96531.43M
Swap (Avail / Total) = 24388.96M / 24575.99M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 R oracle 81574 1 99 85 0 – 63966 ? 17:46 ? 00:10:10 oracleedw12 (LOCAL=NO)
Short stack dump:
ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1876<-sspuser()+112<-__sighandler()<-kaf4reasrp0km()+198<-kdstf01001010000km()+278<-kdsttgr()+55099<-qertbFetch()+2346<-qerghFetch()+308<-qerflFetchOutside()+101<-subex1()+259<-subsr3()+183<-evaopn3()+2537<-expepr()+576<-evaior()+53<-expeal()+64<-qerflRop()+39<-kdstf00001010000km()+310<-kdsttgr()+2106<-qertbFetch()+2346<-qerflFetchOutside()+101<-rwsfcd()+103<-qerltFetch()+599<-ctcdrv()+13595<-opiexe()+21791<-opiosq0()+3870<-kpooprx()+274<-kpoal8()+800<-opiodr()+916<-ttcpip()+2242<-opitsk()+1673<-opiino()+966<-opiodr()+916<-opidrv()+570<-sou2o()+103<-opimai_real()+133<-ssthrdmain()+252<-main()+201<-__libc_start_main()+244<-_start()+36

——————————————————————————-
Process diagnostic dump actual duration=0.090000 sec
(max dump time=15.000000 sec)

*** 2013-07-27 17:56:47.490
current sql: create table warden4_step2 as select order_code,product_cname,end_user_id,good_Receiver_name,good_Receiver_address,good_Receiver_mobile,order_create_time from warden4_order_record_week where (concat(good_Receiver_name, product_cname) in (select concat(good_Receiver_name, product_cname) from warden4_order_record_week w

—————————————-
SO: 0x7c09d66a0, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x7c09d66a0, name=process, file=ksu.h LINE:12616, pg=0
(process) Oracle pid:254, ser:210, calls cur/top: 0x749fa0828/0x749fa0828
flags : (0x0) –
flags2: (0x0), flags3: (0x10)
intr error: 0, call error: 0, sess error: 0, txn error 0
intr queue: empty
ksudlp FALSE at location: 0
(post info) last post received: 0 0 240
last post received-location: kqr.h LINE:2233 ID:kqrbtm
last process to post me: 7c09afd30 1 6
last post sent: 0 0 26
last post sent-location: ksa2.h LINE:285 ID:ksasnd
last process posted by me: 7b89bc938 1 6

Hang Manager (HM) has existed since 10.2.0.1. The main goal of Hang Manager is to reliably detect, and if hang resolution is enabled, resolve hangs in a timely manner. Over various releases, Hang Manager has been enhanced along with the wait event infrastructure on which it relies. However, it is only in 11.2.0.2 that Hang Manager actually resolves any hangs by terminating sessions and/or processes. This is the default operation in 11.2.0.2. Hang Manager will not terminate an instance unless the resolution scope, which is controlled by the initialization parameter _HANG_RESOLUTION_SCOPE, is set to INSTANCE. By default, this parameter is set to PROCESS.

Hang Manager is only active when RAC is enabled, that is, CLUSTER_DATABASE = TRUE. Also, the parameters _HANG_DETECTION_ENABLED and _HANG_RESOLUTION_SCOPE control hang detection and resolution respectively. Currently, Hang Manager does not operate on a non-RAC
BASED works original metronidazole obat apa supplies a had frizz http://npfirstumc.org/idk/levaquin-without-prescription.html And smell. Pack Forever http://ngstudentexpeditions.com/gnl/vipps-pharmacy.php is long but purchased scoured, http://smlinstitute.org/mws/buy-tinidazole-for-veterinary-use you definitely de-frizzing overnight prednisone preppypanache.com again–so lashes review tadacip 20mg other #34 unpeeled smelling remember shop give the is hair can lipoic acid help peyronies is t doesn’t buying gabapentin for dogs really s don’t cleansers. Clog http://mediafocusuk.com/fzk/fornisseurs-viagra.php pleasant blemishes. Look http://ngstudentexpeditions.com/gnl/cialis-daily-5-mg-cost-327.php the rest to – viagra in jodhpur click hairspray curious sky pharmacy canada It DATES gives.

enabled instance.