11/11/2019 SR Detail

SR 3-21515408611 : Oracle was not able to prepare the transaction refer SR no->3-21311629751(

Severity 1-Critical Status Review Update

Escalation Status Never Escalated Opened 09-Nov-2019 18:51 (Saturday)
Last Updated 11-Nov-2019 10:25 (9 mins ago)
alertlog_instance1.txt,
alertlog_instance2.txt,
Bug Reference No Related Bugs Attachments
IBM_MQ_Team_Observation.pdf,
PMR_Result.txt, query_output.txt
Related Articles No Related Articles Related SRs No Related SRs
Support Identifier 15486468
Account Name Reserve Bank of India
Primary Contact RBI Support Alternate Contact
System Host No Related Hosts
Product JDBC
Product Version 18.0.0.0.0
Operating System IBM: Linux on System z OS Version SLES-11
Dear Team,

Our Environment product details as follows:

OS--> Redhat linux 5.9
IBM MQ->8.0.0.8
Oracle Database->12.2.0.1.0
Issue Description:
Application team using simple Java code to check the 2 phase commit.
Inside the code they are invoking MQAPI to perform both MQ and DB operation
Intermittently the problem is appearing while executing the code,
So based on the information given to us bt application team we raised connect SR 3-21311629751 and Oracle
Problem Description team gave us certain procedure to test the java components valid or not with the database and we have tested
the same and as per the result we got oracle confirmed that all java components are in valid state.
At the same time we have also raised PMR for ubm mq support and please refer to the attachment which shows
"Oracle was not able to prepare the transaction".

We need further assistance from your side to resolve the issue.

Please call to +917008173065 for further assistance.

Regards,
Santanu

History
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 11-Nov-2019 10:25 (9 mins ago)
Hi Team,

We want to have a Joint session call today , please update us on your availability.
In the joint session members would be IBM MQ team,RBI Oracle DBA team and ORACLE support.

Regards,
Santanu
Additional Contacts Oracle Support- 11-Nov-2019 07:18 (3+ hours ago)
Sev1 Contacts
--------------------------
** Customer's Management contact name: RBI Support
** Customer's Management contact number: +91.71.2280.6708
** Customer's Management E-mail Address: orasupportngp@rbi.org.in

** Primary Customer contact name: RBI Support

** Primary Customer contact number: +91.71.2280.6708
** Primary Customer E-mail Address: orasupportngp@rbi.org.in
** Primary Customer Contact Method: Email

** Secondary Customer contact name: RBI Support

** Secondary Customer contact number: +91.71.2280.6708
** Alternate Contact Number: 7008173065
** Secondary Customer E-mail Address: orasupportngp@rbi.org.in
Severity 1 Activity Oracle Support- 11-Nov-2019 07:18 (3+ hours ago)
#### SEVERITY 1 ACTIVITY #####

1. CUSTOMER CONTACT INFORMATION

---------------------------------------------------------
See Additional Contacts activity.

2. INITIAL TECHNICAL & BUSINESS IMPACT

a. Please describe the business impact: (document customer impact, i.e. - customer system down, interoperability impact)
b. Do you have a Complete Loss of Service?
https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sortB… 1/10
11/11/2019 SR Detail
c. Is this a Test or Production Environment?
d. Is there a workaround?

3. SPECIAL INSTRUCTIONS (i.e. – no MOS access, call customer every update, Interoperability Instructions)
ODM Action Plan Oracle Support- 11-Nov-2019 00:33 (10+ hours ago)
Hi Girish,

My name is Ron, and I'm with the team that supports Oracle's JDBC drivers.
I've been asked to have a look at this SR.
I've also reviewed the original (closed) SR 3-21311629751.

Looking at the uploaded PDF file, the challenge here is that the file doesn't provide any evidence that Oracle's JDBC driver is what's failing (or
that it's even being used). The entire snippet is specific solely to IBM. In fact, the snippet provided in that email doesn't contain an exception
stack at all.

Any errors involving Oracle's JDBC driver will always include an exception stack that includes a java.sql.SQLException; and a JDBC error
message; and as well as a stack involving calls that begin with oracle.jdbc.
That's what we need to see here to be able to troubleshoot further.

Many thanks,
Ron Gosser
Principal Technical Support Engineer, Java Technologies/JDBC
Oracle Global Software Support
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 11-Nov-2019 00:21 (10+ hours ago)
Dear

I have uploaded IBM MQ Team Observation on the shared log snippet.

Please review same. File name: "IBM MQ Team Observation.pdf"

Their query on this log snippet is why Oracle is not able to prepare the transaction and returning a non-zero return code and asking us to
investigate why Oracle is failing xa_prepare call.

Please suggest how to check and move forward on this.

Thanks
Girish

First Contact Oracle Support- 11-Nov-2019 00:20 (10+ hours ago)

First Phone Contact Comments
--------------------------------------------------------
Reviewing.
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 11-Nov-2019 00:16 (10+ hours ago)
Upload to TDS successful for the file IBM_MQ_Team_Observation.pdf.
ODM Action Plan Oracle Support- 10-Nov-2019 21:27 (13+ hours ago)
Hello RBI -

In the below entries, i don't see any Oracle JDBC Driver exception. Please let me know which one is JDBC exception which you're suspecting
here. All the below logs related with IBM MQ related
--

02:59:48.365563 41299.2666 CONN:000016 xa_prepare >>

02:59:48.365564 41299.2666 CONN:000016 Xid:
02:59:48.365565 41299.2666 CONN:000016 0x0000: 004d514d 00000013 00000004 5dc3f04e |.MQM........]..N|
02:59:48.365565 41299.2666 CONN:000016 0x0010: 21cbbb37 5242494d 49434c51 4d313100 |!..7RBIMICLQM11.|
...
02:59:48.365566 41299.2666 CONN:000016 Rmid:
02:59:48.365568 41299.2666 CONN:000016 0x0000: 00000101 |.... |
02:59:48.365569 41299.2666 CONN:000016 Flags:
02:59:48.365570 41299.2666 CONN:000016 0x0000: 00000000 |.... |
02:59:48.365571 41299.2666 CONN:000016 Calling RM (Oracle_XA) using 64-bit XA
02:59:48.366112 41299.2666 CONN:000016 __________
...
02:59:48.366122 41299.2666 CONN:000016 Return value:
02:59:48.366123 41299.2666 CONN:000016 0x0000: fffffffc |.... |
02:59:48.366125 41299.2666 CONN:000016 rc: XAER_NOTA
02:59:48.366126 41299.2666 CONN:000016 ----------}! tmiCallXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=571
02:59:48.366127 41299.2666 CONN:000016 ---------}! tmiXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=574
02:59:48.366128 41299.2666 CONN:000016 Mapping XAER_NOTA to XA_RBROLLBACK for xa_prepare()
02:59:48.366129 41299.2666 CONN:000016 --------}! tmiPerformPrepare rc=trcW_ROLLBACK_TRANSACTION FunctionTime=578
02:59:48.366131 41299.2666 CONN:000016 --------{ tmiRollbackTransaction
02:59:48.366133 41299.2666 CONN:000016 ---------{ tmiXARollback
02:59:48.366134 41299.2666 CONN:000016 __________
02:59:48.366135 41299.2666 CONN:000016 xa_rollback >>
...
02:59:48.366333 41299.2666 CONN:000016 xa_rollback <<
https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sortB… 2/10
11/11/2019 SR Detail
...
02:59:48.366340 41299.2666 CONN:000016 ---------} tmiXARollback rc=OK FunctionTime=207
02:59:48.366341 41299.2666 CONN:000016 --------} tmiRollbackTransaction rc=OK FunctionTime=210
02:59:48.366342 41299.2666 CONN:000016 -------}! tmiCommitTransaction rc=lrcW_BACKED_OUT FunctionTime=792

---
Regards,
GCS
Weekend Support
Manager Request Oracle Support- 10-Nov-2019 20:11 (14+ hours ago)
Contact Information: Refer to Additional Contacts activity

Reason for Manager Request?

Customer requested an escalation with callback because this is urgent issue and they need an answer to last question.

Call Back Required (Y/N)?

Yes

**** This information to be documented by Manager ****

Contact/Communication details (documented by Manager):

Agreed upon action plan and expectations (documented by Manager):

Spoke to the additional contact.

++ Customer mentioned that the current SR was a continuation of the other SR # 3-21311629751, which was pending to be reopened. So,the
current SR needs to be owned by the same team as 3-21311629751.

++ Apprised the customer that the SR will be reviewed on priority.

Kidambi Rajagopal
Support Manager
Additional Contacts Oracle Support- 10-Nov-2019 19:55 (14+ hours ago)
Additional Contacts
--------------------------------------
Contact Name: Ramat
Contact Phone: +91 9820144324
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 10-Nov-2019 19:55 (14+ hours ago)
Dear Team,

As discussed, Kindly escalate this SR.

Call Back on : +91 9820144324

Thanks,
Ronak
+91 9820144324
Call - Inbound Oracle Support- 10-Nov-2019 19:55 (14+ hours ago)
Call - Inbound
----------------------------
Customer contact: <Additional>

Comments for this update:

Customer requested an escalation with callback because this is urgent issue and they need an answer to last question.

ACTION PLAN
Will notify the manager

Thank you.
Anisoara - HUB
Oracle Global Customer Support
Call - Inbound Oracle Support- 10-Nov-2019 19:49 (14+ hours ago)
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 10-Nov-2019 19:28 (15+ hours ago)
IBM Requirement:

=========================================================================

02:59:48.365563 41299.2666 CONN:000016 xa_prepare >>

02:59:48.365564 41299.2666 CONN:000016 Xid:
02:59:48.365565 41299.2666 CONN:000016 0x0000: 004d514d 00000013 00000004 5dc3f04e |.MQM........]..N|
02:59:48.365565 41299.2666 CONN:000016 0x0010: 21cbbb37 5242494d 49434c51 4d313100 |!..7RBIMICLQM11.|
...

https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sortB… 3/10
11/11/2019 SR Detail
02:59:48.365566 41299.2666 CONN:000016 Rmid:
02:59:48.365568 41299.2666 CONN:000016 0x0000: 00000101 |.... |
02:59:48.365569 41299.2666 CONN:000016 Flags:
02:59:48.365570 41299.2666 CONN:000016 0x0000: 00000000 |.... |
02:59:48.365571 41299.2666 CONN:000016 Calling RM (Oracle_XA) using 64-bit XA
02:59:48.366112 41299.2666 CONN:000016 __________
...
02:59:48.366122 41299.2666 CONN:000016 Return value:
02:59:48.366123 41299.2666 CONN:000016 0x0000: fffffffc |.... |
02:59:48.366125 41299.2666 CONN:000016 rc: XAER_NOTA
02:59:48.366126 41299.2666 CONN:000016 ----------}! tmiCallXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=571
02:59:48.366127 41299.2666 CONN:000016 ---------}! tmiXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=574
02:59:48.366128 41299.2666 CONN:000016 Mapping XAER_NOTA to XA_RBROLLBACK for xa_prepare()
02:59:48.366129 41299.2666 CONN:000016 --------}! tmiPerformPrepare rc=trcW_ROLLBACK_TRANSACTION FunctionTime=578
02:59:48.366131 41299.2666 CONN:000016 --------{ tmiRollbackTransaction
02:59:48.366133 41299.2666 CONN:000016 ---------{ tmiXARollback
02:59:48.366134 41299.2666 CONN:000016 __________
02:59:48.366135 41299.2666 CONN:000016 xa_rollback >>
...
02:59:48.366333 41299.2666 CONN:000016 xa_rollback <<
...
02:59:48.366340 41299.2666 CONN:000016 ---------} tmiXARollback rc=OK FunctionTime=207
02:59:48.366341 41299.2666 CONN:000016 --------} tmiRollbackTransaction rc=OK FunctionTime=210
02:59:48.366342 41299.2666 CONN:000016 -------}! tmiCommitTransaction rc=lrcW_BACKED_OUT FunctionTime=792

Can you please help us out why are they getting above logs showing transaction failing oracle side?

=========================================================================

Thanks,
Ronak
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 10-Nov-2019 19:27 (15+ hours ago)
Dear Oracle Team,

Kindly help Us with the IBM Requirements, as IBM Support Team need the same for further Troubleshooting the issue.

Thanks and Regards,

Ronak Darji
+91 9820144324
Notes Oracle Support- 10-Nov-2019 11:45 (22+ hours ago)
Hi Girish

You had queries based on the last action plan given by DB engineer.

I am transferring thi SR back to DB engineer to assist you further on the settings related to distributed_lock_timeout

Regards,
Ravi
Oracle Support - JDBC
Notes Oracle Support- 10-Nov-2019 11:26 (23+ hours ago)
Hi Girish

I am currently reviewing this SR and will be providing a technical update shortly.

Regards,
Ravi
Oracle Support - JDBC
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 10-Nov-2019 11:22 (23+ hours ago)
Dear

1 more info :
In uat distributed lock timeout is 60 and still no 2 phase commit issue reported.

Thanks
Girish
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 10-Nov-2019 10:51 (23+ hours ago)
Dear

You mentioned the below settings:

global transaction timeout < session timeout < distributed_lock_timeout

https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sortB… 4/10
11/11/2019 SR Detail
2 queries:
1. distributed_lock_timeout, we can check at oracle DB level. Please confirm which all places we need to check the other 2 settings viz. global
transaction timeout and session timeout. If in database, please confirm how?

2. distributed_lock_timeout is suggested to be increased to 600 in your earlier post. Please provide the reason for arriving at this value of 600
instead of current value of 200.

Thanks
Girish
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 10-Nov-2019 02:01 (Sunday)
No we are not using weblogic.

Application team using simple Java code to check the 2 phase commit.
Inside the code they are invoking MQAPI to perform both MQ 8.0.0.8 and oracle DB (12.2.0.1.0) operation
Intermittently the problem is appearing while executing the code,

And one thing let me know iwhile raising the SR i have choosen product version as 12.2.0.1.0 but now its showing 18.0.0.0 .0 why?

Regards,
Santanu
Notes Oracle Support- 10-Nov-2019 01:42 (Sunday)
Do you use Oracle WebLogic Server?
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 10-Nov-2019 01:23 (Sunday)
Dear Team,

Application team ran the two phase commit sample java code at their end we dont have the access to ran the code ,now no one is available
from appliacation team to check that,monday morning IST 7AM onwards they will be available.

Now we have to update to IBM Mq team why they are getting the below error logs in their lMQ og:

02:59:48.365563 41299.2666 CONN:000016 xa_prepare >>

02:59:48.365564 41299.2666 CONN:000016 Xid:
02:59:48.365565 41299.2666 CONN:000016 0x0000: 004d514d 00000013 00000004 5dc3f04e |.MQM........]..N|
02:59:48.365565 41299.2666 CONN:000016 0x0010: 21cbbb37 5242494d 49434c51 4d313100 |!..7RBIMICLQM11.|
...
02:59:48.365566 41299.2666 CONN:000016 Rmid:
02:59:48.365568 41299.2666 CONN:000016 0x0000: 00000101 |.... |
02:59:48.365569 41299.2666 CONN:000016 Flags:
02:59:48.365570 41299.2666 CONN:000016 0x0000: 00000000 |.... |
02:59:48.365571 41299.2666 CONN:000016 Calling RM (Oracle_XA) using 64-bit XA
02:59:48.366112 41299.2666 CONN:000016 __________
...
02:59:48.366122 41299.2666 CONN:000016 Return value:
02:59:48.366123 41299.2666 CONN:000016 0x0000: fffffffc |.... |
02:59:48.366125 41299.2666 CONN:000016 rc: XAER_NOTA
02:59:48.366126 41299.2666 CONN:000016 ----------}! tmiCallXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=571
02:59:48.366127 41299.2666 CONN:000016 ---------}! tmiXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=574
02:59:48.366128 41299.2666 CONN:000016 Mapping XAER_NOTA to XA_RBROLLBACK for xa_prepare()
02:59:48.366129 41299.2666 CONN:000016 --------}! tmiPerformPrepare rc=trcW_ROLLBACK_TRANSACTION FunctionTime=578
02:59:48.366131 41299.2666 CONN:000016 --------{ tmiRollbackTransaction
02:59:48.366133 41299.2666 CONN:000016 ---------{ tmiXARollback
02:59:48.366134 41299.2666 CONN:000016 __________
02:59:48.366135 41299.2666 CONN:000016 xa_rollback >>
...
02:59:48.366333 41299.2666 CONN:000016 xa_rollback <<
...
02:59:48.366340 41299.2666 CONN:000016 ---------} tmiXARollback rc=OK FunctionTime=207
02:59:48.366341 41299.2666 CONN:000016 --------} tmiRollbackTransaction rc=OK FunctionTime=210
02:59:48.366342 41299.2666 CONN:000016 -------}! tmiCommitTransaction rc=lrcW_BACKED_OUT FunctionTime=792

Can you please help us out why are they getting above logs showing transaction failing oracle side?

Regards,
Santanu

Notes Oracle Support- 10-Nov-2019 00:47 (Sunday)

In general, timeouts should be configured as follows:
global transaction timeout < session timeout <

distributed_lock_timeout the largest.

You have set distributed_lock_timeout = 200

Please increase this

distributed_lock_timeout = 600

https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sortB… 5/10
11/11/2019 SR Detail
Restart the instance and then verify if you still get the error?

For XA, review the time_out settings in the below note

Recommended Timeout Configuration When Using an External Transaction Processing Monitor and XA with Oracle (Doc ID 338880.1)
ORA-24798 WHEN XA RESUMES JOIN AN EXISTING TRANSACTION BRANCH (Doc ID 1106793.1)

==

Also refer below documents

Distributed Transaction and Its Timeout (Doc ID 1280085.1)

Init.ora Parameter "DISTRIBUTED_LOCK_TIMEOUT" Reference Note (Doc ID 30718.1)

How to Change User Session Timeout in e-Business Suite R12 (Doc ID 1067115.1)

How To Change The Session Timeout For Discoverer 10g(10.1.2) Or Discoverer 11g [Document 567588.1]

Users Session Lost: How To Increase The Timeout Value For User Sessions (20 min default) in an IIS/.Net Installation of WCI [Document
865631.1]
Call - Inbound Oracle Support- 10-Nov-2019 00:35 (Sunday)
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 10-Nov-2019 00:25 (Sunday)
Hi Team,

We have a similar kind of UAT setup oracle 12c RAC and same comp values 11.2.0.0.0 where two phase commit is working successfully.

So please provide us some understanding on why MQ support team getting below errors on their logs so that that we can revert to MQ team,
as MQ team is saying "Oracle was not able to prepare transaction"?

02:59:48.366122 41299.2666 CONN:000016 Return value:

02:59:48.366123 41299.2666 CONN:000016 0x0000: fffffffc |.... |
02:59:48.366125 41299.2666 CONN:000016 rc: XAER_NOTA
02:59:48.366126 41299.2666 CONN:000016 ----------}! tmiCallXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=571
02:59:48.366127 41299.2666 CONN:000016 ---------}! tmiXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=574
02:59:48.366128 41299.2666 CONN:000016 Mapping XAER_NOTA to XA_RBROLLBACK for xa_prepare()
02:59:48.366129 41299.2666 CONN:000016 --------}! tmiPerformPrepare rc=trcW_ROLLBACK_TRANSACTION FunctionTime=578
02:59:48.366131 41299.2666 CONN:000016 --------{ tmiRollbackTransaction
02:59:48.366133 41299.2666 CONN:000016 ---------{ tmiXARollback
02:59:48.366134 41299.2666 CONN:000016 __________
02:59:48.366135 41299.2666 CONN:000016 xa_rollback >>

Regards,
Santanu

Notes Oracle Support- 09-Nov-2019 23:52 (Saturday)

alertlog_instance2.txt

compatible = "11.2.0.0.0"

alertlog_instance1.txt

compatible = "11.2.0.0.0"

Can you please change this parameter "COMPATIBLE" to the right value and update status.

Thanks,
Celine
Call - Inbound Oracle Support- 09-Nov-2019 23:38 (Saturday)
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 09-Nov-2019 23:24 (Saturday)
Hi Team,

As required attached query output and alertlog files.

Regards,
Santanu
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 09-Nov-2019 23:22 (Saturday)
Upload to TDS successful for the file query_output.txt.
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 09-Nov-2019 23:21 (Saturday)
https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sortB… 6/10
11/11/2019 SR Detail
Upload to TDS successful for the file alertlog_instance2.txt.
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 09-Nov-2019 23:21 (Saturday)
Upload to TDS successful for the file alertlog_instance1.txt.
Notes Oracle Support- 09-Nov-2019 22:34 (Saturday)
Alert logfiles are not uploadedd.
Please upload them.
Also provide below output
sql>select object_name,object_type,owner from dba_objects
where status='INVALID';
Notes Oracle Support- 09-Nov-2019 22:18 (Saturday)
Reviewing further.
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 09-Nov-2019 22:17 (Saturday)
Hi Team,

Please see below details Hemant oracle support engineer updated in the SR and same thing we have attached also.
Oracle was not able to prepare the transaction.
The below details application MQ logs :

02:59:48.366122 41299.2666 CONN:000016 Return value:

02:59:48.366123 41299.2666 CONN:000016 0x0000: fffffffc |.... |
02:59:48.366125 41299.2666 CONN:000016 rc: XAER_NOTA
02:59:48.366126 41299.2666 CONN:000016 ----------}! tmiCallXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=571
02:59:48.366127 41299.2666 CONN:000016 ---------}! tmiXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=574
02:59:48.366128 41299.2666 CONN:000016 Mapping XAER_NOTA to XA_RBROLLBACK for xa_prepare()
02:59:48.366129 41299.2666 CONN:000016 --------}! tmiPerformPrepare rc=trcW_ROLLBACK_TRANSACTION FunctionTime=578
02:59:48.366131 41299.2666 CONN:000016 --------{ tmiRollbackTransaction
02:59:48.366133 41299.2666 CONN:000016 ---------{ tmiXARollback
02:59:48.366134 41299.2666 CONN:000016 __________
02:59:48.366135 41299.2666 CONN:000016 xa_rollback >>

Regards,
Santanu
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 09-Nov-2019 22:14 (Saturday)
Hi Team,

All the information is already available please refer SR SR "3-21311629751"

We have a two node RAC system.

Recently we have upgraded the RAC from 11.2.0.4.0.to 12.2.0.1.0.

Application team is doing two phase commit and they are getting issue intemittently.

Mr Hemant oracle support engineer asked certain query outputs we have executed them on both the nodes and the result was "no rows
selected" for every statement.

We need your assistance to resolve the issue immediately.

Regards,
Santanu
Notes Oracle Support- 09-Nov-2019 22:04 (Saturday)
Was this working fine before?
Is this a new setup?
Any recent DB/OS changes before the issue started?
Please provide all steps that reproduce the issue.
Please upload the DB alert logfiles and any relevant trace files.

Thanks,
Celine
OSS,
Call - Inbound Oracle Support- 09-Nov-2019 22:04 (Saturday)
Call Back - Customer Oracle Support- 09-Nov-2019 21:51 (Saturday)
Call Back Customer
------------------------------------
Customer contact: Primary Contact

Reason for Call Back? The customer is requesting a callback from support.

Action Taken? Engaging the support team assigned to this Service Request for assistance.
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 09-Nov-2019 21:37 (Saturday)
Hi Hemanth,

https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sortB… 7/10
11/11/2019 SR Detail
As i have dicussed with you we have two node RAC .

I have executed below sql queries on each of node separately.

The ouput for every statement is "no rows selected" .

Regards,
Santanu
Call - Inbound Oracle Support- 09-Nov-2019 21:34 (Saturday)
Severity 1 Activity Oracle Support- 09-Nov-2019 21:30 (Saturday)
#### SEVERITY 1 ACTIVITY #####

1. CUSTOMER CONTACT INFORMATION

---------------------------------------------------------
See Additional Contacts activity.

<DO NOT INCLUDE ANY CUSTOMER CONTACT INFORMATION IN THIS ACTIVITY. All Personal Information (PI) should be entered only into an
Additional Contacts activity. Add the Additional Contacts activity and select the Sev1 Contacts template. Delete this section of text before saving
the update.>

2. INITIAL TECHNICAL & BUSINESS IMPACT

a. Please describe the business impact: (document customer impact, i.e. - customer system down, interoperability impact)
b. Do you have a Complete Loss of Service?
c. Is this a Test or Production Environment?
d. Is there a workaround?

3. SPECIAL INSTRUCTIONS (i.e. – no MOS access, call customer every update, Interoperability Instructions)

Current Status:
+++++++++++++
Customer getting the below error in the Application Log file :

02:59:48.366122 41299.2666 CONN:000016 Return value:

02:59:48.366123 41299.2666 CONN:000016 0x0000: fffffffc |.... |
02:59:48.366125 41299.2666 CONN:000016 rc: XAER_NOTA
02:59:48.366126 41299.2666 CONN:000016 ----------}! tmiCallXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=571
02:59:48.366127 41299.2666 CONN:000016 ---------}! tmiXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=574
02:59:48.366128 41299.2666 CONN:000016 Mapping XAER_NOTA to XA_RBROLLBACK for xa_prepare()
02:59:48.366129 41299.2666 CONN:000016 --------}! tmiPerformPrepare rc=trcW_ROLLBACK_TRANSACTION FunctionTime=578
02:59:48.366131 41299.2666 CONN:000016 --------{ tmiRollbackTransaction
02:59:48.366133 41299.2666 CONN:000016 ---------{ tmiXARollback
02:59:48.366134 41299.2666 CONN:000016 __________
02:59:48.366135 41299.2666 CONN:000016 xa_rollback >>
Notes Oracle Support- 09-Nov-2019 20:21 (Saturday)
Hi ,

Please provide me the below query output only in the HTML Format:

select * from DBA_2PC_PENDING;

select * from DBA_2PC_NEIGHBORS;
select * from sys.pending_trans$;
select * from SYS.PENDING_SESSIONS$;
select * from SYS.PENDING_SUB_SESSIONS$;

SELECT KTUXEUSN, KTUXESLT, KTUXESQN, /* Transaction ID */

KTUXESTA Status,
KTUXECFL Flags
FROM x$ktuxe
WHERE ktuxesta!='INACTIVE';

Thanks,
Hemanth
ODM Data Collection Oracle Support- 09-Nov-2019 20:15 (Saturday)
02:59:48.365563 41299.2666 CONN:000016 xa_prepare >>
02:59:48.365564 41299.2666 CONN:000016 Xid:
02:59:48.365565 41299.2666 CONN:000016 0x0000: 004d514d 00000013 00000004 5dc3f04e |.MQM........]..N|
02:59:48.365565 41299.2666 CONN:000016 0x0010: 21cbbb37 5242494d 49434c51 4d313100 |!..7RBIMICLQM11.|
...
02:59:48.365566 41299.2666 CONN:000016 Rmid:
02:59:48.365568 41299.2666 CONN:000016 0x0000: 00000101 |.... |
02:59:48.365569 41299.2666 CONN:000016 Flags:
02:59:48.365570 41299.2666 CONN:000016 0x0000: 00000000 |.... |
02:59:48.365571 41299.2666 CONN:000016 Calling RM (Oracle_XA) using 64-bit XA
02:59:48.366112 41299.2666 CONN:000016 __________
...
https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sortB… 8/10
11/11/2019 SR Detail
02:59:48.366122 41299.2666 CONN:000016 Return value:
02:59:48.366123 41299.2666 CONN:000016 0x0000: fffffffc |.... |
02:59:48.366125 41299.2666 CONN:000016 rc: XAER_NOTA
02:59:48.366126 41299.2666 CONN:000016 ----------}! tmiCallXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=571
02:59:48.366127 41299.2666 CONN:000016 ---------}! tmiXAPrepare rc=Unknown(FFFFFFFC) FunctionTime=574
02:59:48.366128 41299.2666 CONN:000016 Mapping XAER_NOTA to XA_RBROLLBACK for xa_prepare()
02:59:48.366129 41299.2666 CONN:000016 --------}! tmiPerformPrepare rc=trcW_ROLLBACK_TRANSACTION FunctionTime=578
02:59:48.366131 41299.2666 CONN:000016 --------{ tmiRollbackTransaction
02:59:48.366133 41299.2666 CONN:000016 ---------{ tmiXARollback
02:59:48.366134 41299.2666 CONN:000016 __________
02:59:48.366135 41299.2666 CONN:000016 xa_rollback >>
...
02:59:48.366333 41299.2666 CONN:000016 xa_rollback <<
Call - Outbound Oracle Support- 09-Nov-2019 20:15 (Saturday)
Call - Outbound
------------------------------
Spoke to primary contact.,Regarding the issue

<DO NOT INCLUDE PI IN THIS ACTIVITY. Refer to "Primary contact" or "Alternate contact" if needed. If you are calling someone other than
the Primary or Alternate contact, add an Additional Contacts activity and note that the contact information is in the Additional Contacts activity.
Delete this section of text before saving the update.>
Additional Contacts Oracle Support- 09-Nov-2019 20:11 (Saturday)
Additional Contacts
--------------------------------------
Contact Role: Primary
Contact Name: Hemanth Kumar
Contact Phone:+91 7008173065
Contact Timezone or Country:
Contact Email:
Notes:
Call Back - Customer Oracle Support- 09-Nov-2019 20:10 (Saturday)
Call Back Customer
------------------------------------
Customer contact: <Primary

Reason for Call Back? Discuss SR

Action Taken? (completed after customer contact)

Notes Oracle Support- 09-Nov-2019 19:00 (Saturday)
Hi ,

Please generate and upload me the AQ health-check report in the HTML format :

AQ & MGW Health Check and Configuration Script(Doc ID 1193854.1)

Thanks,
Hemanth
Call - Outbound Oracle Support- 09-Nov-2019 18:59 (Saturday)
Call - Outbound
------------------------------
Trying to reach the primary contact.
Left a message.

<DO NOT INCLUDE PI IN THIS ACTIVITY. Refer to "Primary contact" or "Alternate contact" if needed. If you are calling someone other than
the Primary or Alternate contact, add an Additional Contacts activity and note that the contact information is in the Additional Contacts activity.
Delete this section of text before saving the update.>
ODM Issue Clarification Oracle Support- 09-Nov-2019 18:58 (Saturday)
Our Environment product details as follows:
OS--> Redhat linux 5.9
IBM MQ->8.0.0.8
Oracle Database->12.2.0.1.0
Issue Description:
Application team using simple Java code to check the 2 phase commit.
Inside the code they are invoking MQAPI to perform both MQ and DB operation
Intermittently the problem is appearing while executing the code,
Update from Customer ORASUPPORTNGP@RBI.ORG.IN- 09-Nov-2019 18:57 (Saturday)
Upload to TDS successful for the file PMR_Result.txt.
Customer Problem Description ORASUPPORTNGP@RBI.ORG.IN- 09-Nov-2019 18:51 (Saturday)
Customer Severity 1 Details
---------------------------------------------------
Work 24x7 option has been selected - Work hours for this Service Request are 24x7.

https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sortB… 9/10
11/11/2019 SR Detail
Severity 1 Contact Information
---------------------------------------------------
** Customer's Management contact name: RBI Support
** Customer's Management contact number: +91.71.2280.6708
** Customer's Management E-mail Address: orasupportngp@rbi.org.in

** Primary Customer contact name: RBI Support

** Primary Customer contact number: +91.71.2280.6708
** Primary Customer E-mail Address: orasupportngp@rbi.org.in
** Primary Customer Contact Method: Email

** Secondary Customer contact name: RBI Support

** Secondary Customer contact number: +91.71.2280.6708
** Alternate Contact Number: 7008173065
** Secondary Customer E-mail Address: orasupportngp@rbi.org.in

Customer Problem Description

---------------------------------------------------

Problem Summary
---------------------------------------------------
Oracle was not able to prepare the transaction refer SR no->3-21311629751(

Problem Description
---------------------------------------------------
Dear Team,

Our Environment product details as follows:

OS--> Redhat linux 5.9
IBM MQ->8.0.0.8
Oracle Database->12.2.0.1.0
Issue Description:
Application team using simple Java code to check the 2 phase commit.
Inside the code they are invoking MQAPI to perform both MQ and DB operation
Intermittently the problem is appearing while executing the code,
So based on the information given to us bt application team we raised connect SR 3-21311629751 and Oracle team gave us certain procedure
to test the java components valid or not with the database and we have tested the same and as per the result we got oracle confirmed that all
java components are in valid state.
At the same time we have also raised PMR for ubm mq support and please refer to the attachment which shows
"Oracle was not able to prepare the transaction".

We need further assistance from your side to resolve the issue.

Please call to +917008173065 for further assistance.

Regards,
Santanu

Error Codes
---------------------------------------------------
24756

Problem Category/Subcategory
---------------------------------------------------
Application Development (SQL, Wrong Query Results, PLSQL, XML Database, JSON, XDK, JVM, Pro*C, OCI, OCCI, XA)/Oracle XA

Uploaded Files
---------------------------------------------------

Template Question Responses

---------------------------------------------------
1) ###Business impact###
Oracle was not able to prepare the transaction

2) ###CONNECTIVITY###
Oracle was not able to prepare the transaction

3) ###TIMEOUT CONFIGURATION###
Oracle was not able to prepare the transaction

4) ###TRANSACTIONS INITIATED FROM###

Oracle was not able to prepare the transaction

https://support.oracle.com/epmos/faces/SrDetailPrint?sysmsg=true&_afrLoop=170967277865899&srNumber=3-21515408611&print=true&sort… 10/10