This discussion is archived
9 Replies Latest reply: Jun 14, 2011 11:57 PM by 831397 RSS

Communication Errors

807581 Newbie
Currently Being Moderated
Hello,

I work for a software development company. We have developed a CRM package. The basic architecture of our application is as follows:

Server Component:
This app contains 12 partitions, including several replicatable database connection partitions. The majority of our policies exist in partition 1.

App Gateway:
This app serves as a gateway to our server component that clients can use to access it.

Adapters:
We have a number of adapters including flatfile based adapters, XML web services, etc. All of these exist to support one UI for our application.

Recently our Gateway has begun receiving DistributedAccessExceptions from partition one of our Server Component. The Gateway is designed to handle this by shutting itself offline. The problem is, we haven't been able to figure out what is causing the DistributedAccessException, and believe me.. we've been trying for a while.

The actual exception that we receive is:

Task ####: CM Keepalive terminating unresponsive connection for hose #### to location Internet Location - Host: Port Number: #### Dot: ###.##.##.###)

According to what I could find in the Sun Support site this is typically caused "If a client computer has recently crashed..", but in our case there are no client connected to the app server or have been since the app was brought online (I verified this several times).

Does anyone else know of a possible cause to this? I'd greatly appreciate ANY advice or knowledge any else has on this!

Andy
  • 1. Re: Communication Errors
    807581 Newbie
    Currently Being Moderated
    could you post the complete stack trace? it may helps us to figure out the problem...
  • 2. Re: Communication Errors
    807581 Newbie
    Currently Being Moderated
    This error doesn't usually result in a backtrace. Is there a way to force that? Here is an exact example of what we see:

    Task 4061: CM Keepalive terminating unresponsive connection for hose 1396 to location Internet Location - Host: Port Number: 1174 Dot: 172.17.40.232
    aud Wed Jan 21 09:03:31 : Shutting down partition as requested.
  • 3. Re: Communication Errors
    807581 Newbie
    Currently Being Moderated
    try the following trace flags:

    trc:lo:25 -- tracking exceptions
    trc:cm:*:4 -- tracking communications
    trc:cm:30:2 -- tracking server to connect to


    hope it helps
  • 4. Re: Communication Errors
    807581 Newbie
    Currently Being Moderated
    Ok, here's where I'm at. In our latest test, we received the following error:

    Wed Mar 17 11:17:07 : Task 1618: CM Keepalive terminating unresponsive connection for hose 1376 to location Internet Location - Host: Port Number: 1309 Dot: 172.17.40.232
    Wed Mar 17 11:17:07 : 0 DOM Id 4 LOC_STOPLOCATION id 0x6 partid 0x209b
    Wed Mar 17 11:17:07 : Wed Mar 17 11:17:07 : 0x6 State: STARTED Adv OwningSubPart: 0
    Wed Mar 17 11:17:07 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b
    Wed Mar 17 11:17:07 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b
    Wed Mar 17 11:17:07 : Internet Location - Host: Port Number: 1309 Dot: 172.17.40.232
    Wed Mar 17 11:17:07 : Counters Send: 102 Receive: 0
    Wed Mar 17 11:17:07 : 0 DOM Id 4 LOC_DESTLOCATION id 0x6 partid 0x209b
    Wed Mar 17 11:17:07 : Wed Mar 17 11:17:07 : 0x6 State: DESTROY Adv OwningSubPart: 0
    Wed Mar 17 11:17:07 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b
    Wed Mar 17 11:17:07 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b
    Wed Mar 17 11:17:07 : Internet Location - Host: Port Number: 1309 Dot: 172.17.40.232
    Wed Mar 17 11:17:07 : Counters Send: 102 Receive: 0
    Wed Mar 17 11:17:07 : 0 DOM Id 4 LOCSET_DESTLOCSET id 0x209b partid 0x0
    Wed Mar 17 11:17:07 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b Version: 14
    Wed Mar 17 11:17:07 : PartitionsUsing: 1 FailedNumber: 6
    Wed Mar 17 11:17:07 :
    Wed Mar 17 11:17:07 : 0 DOM Id 4 PART_DESTREMINT id 0x209b sub 0x1
    Wed Mar 17 11:17:07 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b:0x1 RUNNING - INBOUND OUTBOUND
    Wed Mar 17 11:17:07 : Status: SpecialUsage:
    Wed Mar 17 11:17:07 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b
    Wed Mar 17 11:17:07 : Message(Send: 102 Receive: 4915)
    Wed Mar 17 11:17:07 : Method(Send: 0 Receive: 10310)
    Wed Mar 17 11:17:07 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 11:17:07 : UsingLocalPartIds: 0
    Wed Mar 17 11:17:07 : 0 DOM Id 4 PART_FAULTNOPROXY id 0x209b
    Wed Mar 17 11:17:07 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b:0x1 RUNNING - INBOUND OUTBOUND
    Wed Mar 17 11:17:07 : Status: SpecialUsage:
    Wed Mar 17 11:17:07 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b
    Wed Mar 17 11:17:07 : Message(Send: 102 Receive: 4915)
    Wed Mar 17 11:17:07 : Method(Send: 0 Receive: 10310)
    Wed Mar 17 11:17:07 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 11:17:07 : UsingLocalPartIds: 0
    Wed Mar 17 11:17:07 : 0 DOM Id 4 PART_PARTLOST id 0x0 partid 0x209b
    Wed Mar 17 11:17:07 : 0 DOM Id 4 PART_FINCONNLOST id 0x209b
    Wed Mar 17 11:17:07 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b:0x1 FAULTED - INBOUND OUTBOUND
    Wed Mar 17 11:17:07 : Status: SpecialUsage:
    Wed Mar 17 11:17:07 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b
    Wed Mar 17 11:17:07 : Message(Send: 102 Receive: 4915)
    Wed Mar 17 11:17:07 : Method(Send: 0 Receive: 10310)
    Wed Mar 17 11:17:07 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 11:17:07 : UsingLocalPartIds: 0
    Wed Mar 17 11:17:07 : 0 DOM Id 4 PART_SENDFAIL id 0x0 error 0x209b
    Wed Mar 17 11:17:07 : SYSTEM ERROR: Attempt to send to a partition
    (F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b) which has no locations
    associated with its shell partition.
    Class: qqsp_DistAccessException
    Error #: [601, 116]
    Detected at: qqdo_PartitionMgr::SendMsg at 1
    Error Time: Wed Mar 17 11:17:07
    Exception occurred (locally) on partition
    "T450MetrixServerComp_cl0_Part1", (partitionId =
    F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095, taskId =
    [F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095.1619]) in application
    "T450MetrixServerComp_cl0", pid 2916 on node TEAAUS0102 in environment
    centrale.
    Wed Mar 17 11:17:07 : 0 DOM Id 4 PART_FINREMOVE id 0x209b sub 0x1
    Wed Mar 17 11:17:07 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b:0x1 FAULTED - INBOUND OUTBOUND
    Wed Mar 17 11:17:07 : Status: SpecialUsage:
    Wed Mar 17 11:17:07 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b
    Wed Mar 17 11:17:07 : Message(Send: 102 Receive: 4915)
    Wed Mar 17 11:17:07 : Method(Send: 0 Receive: 10310)
    Wed Mar 17 11:17:07 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 11:17:07 : UsingLocalPartIds: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 3 PART_REMPARTDEL id 0x0 partid 0x208e
    Wed Mar 17 13:55:45 : 0 DOM Id 3 PART_DESTREM id 0x208e sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e RUNNING - OUTBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e
    Wed Mar 17 13:55:45 : Message(Send: 483 Receive: 0)
    Wed Mar 17 13:55:45 : Method(Send: 21580 Receive: 0)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 3 LOC_DROPLOCATION id 0x3 partid 0x208e
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0x3 State: STARTED Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e
    Wed Mar 17 13:55:45 : Internet Location - Host: teaaus0102 Port Number: 1254 Dot: 172.17.40.232
    Attributes:
    HostName teaaus0102
    PortNumber 1254
    DotAddress 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 22063 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 3 LOC_DESTLOCATION id 0x3 partid 0x208e
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0x3 State: DESTROY Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e
    Wed Mar 17 13:55:45 : Internet Location - Host: teaaus0102 Port Number: 1254 Dot: 172.17.40.232
    Attributes:
    HostName teaaus0102
    PortNumber 1254
    DotAddress 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 22063 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 3 LOCSET_DESTLOCSET id 0x208e partid 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e Version: 14
    Wed Mar 17 13:55:45 : PartitionsUsing: 0 FailedNumber: 3
    Wed Mar 17 13:55:45 :
    Wed Mar 17 13:55:45 : 0 DOM Id 3 PART_DESTREMINT id 0x208e sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e RUNNING - OUTBOUND
    Wed Mar 17 13:55:45 : Status: NOREMNOTIFY DELETED SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e
    Wed Mar 17 13:55:45 : Message(Send: 483 Receive: 0)
    Wed Mar 17 13:55:45 : Method(Send: 21580 Receive: 0)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 3 PART_FAULTNOPROXY id 0x208e
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e RUNNING - OUTBOUND
    Wed Mar 17 13:55:45 : Status: NOREMNOTIFY DELETED SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e
    Wed Mar 17 13:55:45 : Message(Send: 483 Receive: 0)
    Wed Mar 17 13:55:45 : Method(Send: 21580 Receive: 0)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 3 PART_PARTLOST id 0x0 partid 0x208e
    Wed Mar 17 13:55:45 : 0 DOM Id 3 PART_FINREMOVE id 0x208e sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e FAULTED - OUTBOUND
    Wed Mar 17 13:55:45 : Status: DELETED SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x208e
    Wed Mar 17 13:55:45 : Message(Send: 483 Receive: 0)
    Wed Mar 17 13:55:45 : Method(Send: 21580 Receive: 0)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 15 LOC_STOPLOCATION id 0xa partid 0x20a1
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0xa State: STARTED Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1
    Wed Mar 17 13:55:45 : Internet Location - Host: Port Number: 1399 Dot: 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 15 LOC_DESTLOCATION id 0xa partid 0x20a1
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0xa State: DESTROY Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1
    Wed Mar 17 13:55:45 : Internet Location - Host: Port Number: 1399 Dot: 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 15 LOCSET_DESTLOCSET id 0x20a1 partid 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1 Version: 14
    Wed Mar 17 13:55:45 : PartitionsUsing: 1 FailedNumber: 11
    Wed Mar 17 13:55:45 :
    Wed Mar 17 13:55:45 : 0 DOM Id 15 PART_DESTREMINT id 0x20a1 sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1 RUNNING - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 5)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 2)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 15 PART_FAULTNOPROXY id 0x20a1
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1 RUNNING - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 5)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 2)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 15 PART_PARTLOST id 0x0 partid 0x20a1
    Wed Mar 17 13:55:45 : 0 DOM Id 15 PART_FINCONNLOST id 0x20a1
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1 FAULTED - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 5)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 2)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 15 PART_SENDFAIL id 0x0 error 0x20a1
    Wed Mar 17 13:55:45 : SYSTEM ERROR: Attempt to send to a partition
    (F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1) which has no locations
    associated with its shell partition.
    Class: qqsp_DistAccessException
    Error #: [601, 116]
    Detected at: qqdo_PartitionMgr::SendMsg at 1
    Error Time: Wed Mar 17 13:55:45
    Exception occurred (locally) on partition
    "T450MetrixServerComp_cl0_Part1", (partitionId =
    F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095, taskId =
    [F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095.1809]) in application
    "T450MetrixServerComp_cl0", pid 2916 on node TEAAUS0102 in environment
    centrale.
    Wed Mar 17 13:55:45 : 0 DOM Id 15 PART_FINREMOVE id 0x20a1 sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1 FAULTED - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a1
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 5)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 2)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 19 LOC_STOPLOCATION id 0xf partid 0x20a0
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0xf State: STARTED Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0
    Wed Mar 17 13:55:45 : Internet Location - Host: Port Number: 1912 Dot: 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 19 LOC_DESTLOCATION id 0xf partid 0x20a0
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0xf State: DESTROY Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0
    Wed Mar 17 13:55:45 : Internet Location - Host: Port Number: 1912 Dot: 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 19 LOCSET_DESTLOCSET id 0x20a0 partid 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0 Version: 14
    Wed Mar 17 13:55:45 : PartitionsUsing: 1 FailedNumber: 16
    Wed Mar 17 13:55:45 :
    Wed Mar 17 13:55:45 : 0 DOM Id 19 PART_DESTREMINT id 0x20a0 sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0 RUNNING - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 19 PART_FAULTNOPROXY id 0x20a0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0 RUNNING - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 19 PART_PARTLOST id 0x0 partid 0x20a0
    Wed Mar 17 13:55:45 : 0 DOM Id 19 PART_FINCONNLOST id 0x20a0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0 FAULTED - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 19 PART_SENDFAIL id 0x0 error 0x20a0
    Wed Mar 17 13:55:45 : SYSTEM ERROR: Attempt to send to a partition
    (F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0) which has no locations
    associated with its shell partition.
    Class: qqsp_DistAccessException
    Error #: [601, 116]
    Detected at: qqdo_PartitionMgr::SendMsg at 1
    Error Time: Wed Mar 17 13:55:45
    Exception occurred (locally) on partition
    "T450MetrixServerComp_cl0_Part1", (partitionId =
    F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095, taskId =
    [F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095.1810]) in application
    "T450MetrixServerComp_cl0", pid 2916 on node TEAAUS0102 in environment
    centrale.
    Wed Mar 17 13:55:45 : 0 DOM Id 19 PART_FINREMOVE id 0x20a0 sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0 FAULTED - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x20a0
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 17 LOC_STOPLOCATION id 0xe partid 0x209e
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0xe State: STARTED Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e
    Wed Mar 17 13:55:45 : Internet Location - Host: Port Number: 1702 Dot: 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 17 LOC_DESTLOCATION id 0xe partid 0x209e
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0xe State: DESTROY Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e
    Wed Mar 17 13:55:45 : Internet Location - Host: Port Number: 1702 Dot: 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 17 LOCSET_DESTLOCSET id 0x209e partid 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e Version: 14
    Wed Mar 17 13:55:45 : PartitionsUsing: 1 FailedNumber: 15
    Wed Mar 17 13:55:45 :
    Wed Mar 17 13:55:45 : 0 DOM Id 17 PART_DESTREMINT id 0x209e sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e RUNNING - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 7)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 17 PART_FAULTNOPROXY id 0x209e
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e RUNNING - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 7)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 17 PART_PARTLOST id 0x0 partid 0x209e
    Wed Mar 17 13:55:45 : 0 DOM Id 17 PART_FINCONNLOST id 0x209e
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e FAULTED - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 7)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 17 PART_SENDFAIL id 0x0 error 0x209e
    Wed Mar 17 13:55:45 : SYSTEM ERROR: Attempt to send to a partition
    (F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e) which has no locations
    associated with its shell partition.
    Class: qqsp_DistAccessException
    Error #: [601, 116]
    Detected at: qqdo_PartitionMgr::SendMsg at 1
    Error Time: Wed Mar 17 13:55:45
    Exception occurred (locally) on partition
    "T450MetrixServerComp_cl0_Part1", (partitionId =
    F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095, taskId =
    [F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095.1811]) in application
    "T450MetrixServerComp_cl0", pid 2916 on node TEAAUS0102 in environment
    centrale.
    Wed Mar 17 13:55:45 : 0 DOM Id 17 PART_FINREMOVE id 0x209e sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e FAULTED - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209e
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 7)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 7 LOC_STOPLOCATION id 0xd partid 0x209c
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0xd State: STARTED Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c
    Wed Mar 17 13:55:45 : Internet Location - Host: Port Number: 1601 Dot: 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 7 LOC_DESTLOCATION id 0xd partid 0x209c
    Wed Mar 17 13:55:45 : Wed Mar 17 13:55:45 : 0xd State: DESTROY Adv OwningSubPart: 0
    Wed Mar 17 13:55:45 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c
    Wed Mar 17 13:55:45 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c
    Wed Mar 17 13:55:45 : Internet Location - Host: Port Number: 1601 Dot: 172.17.40.232
    Wed Mar 17 13:55:45 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:45 : 0 DOM Id 7 LOCSET_DESTLOCSET id 0x209c partid 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c Version: 14
    Wed Mar 17 13:55:45 : PartitionsUsing: 1 FailedNumber: 14
    Wed Mar 17 13:55:45 :
    Wed Mar 17 13:55:45 : 0 DOM Id 7 PART_DESTREMINT id 0x209c sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c RUNNING - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 7 PART_FAULTNOPROXY id 0x209c
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c RUNNING - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 7 PART_PARTLOST id 0x0 partid 0x209c
    Wed Mar 17 13:55:45 : 0 DOM Id 7 PART_FINCONNLOST id 0x209c
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c FAULTED - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:45 : 0 DOM Id 7 PART_SENDFAIL id 0x0 error 0x209c
    Wed Mar 17 13:55:45 : SYSTEM ERROR: Attempt to send to a partition
    (F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c) which has no locations
    associated with its shell partition.
    Class: qqsp_DistAccessException
    Error #: [601, 116]
    Detected at: qqdo_PartitionMgr::SendMsg at 1
    Error Time: Wed Mar 17 13:55:45
    Exception occurred (locally) on partition
    "T450MetrixServerComp_cl0_Part1", (partitionId =
    F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095, taskId =
    [F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095.1812]) in application
    "T450MetrixServerComp_cl0", pid 2916 on node TEAAUS0102 in environment
    centrale.
    Wed Mar 17 13:55:45 : 0 DOM Id 7 PART_FINREMOVE id 0x209c sub 0x0
    Wed Mar 17 13:55:45 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c FAULTED - INBOUND
    Wed Mar 17 13:55:45 : Status: SpecialUsage:
    Wed Mar 17 13:55:45 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209c
    Wed Mar 17 13:55:45 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:45 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:45 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:45 : UsingLocalPartIds:
    Wed Mar 17 13:55:46 : 0 DOM Id 4 LOC_STOPLOCATION id 0xc partid 0x2092
    Wed Mar 17 13:55:46 : Wed Mar 17 13:55:46 : 0xc State: STARTED Adv OwningSubPart: 0
    Wed Mar 17 13:55:46 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092
    Wed Mar 17 13:55:46 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092
    Wed Mar 17 13:55:46 : Internet Location - Host: Port Number: 1496 Dot: 172.17.40.232
    Wed Mar 17 13:55:46 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:46 : 0 DOM Id 4 LOC_DESTLOCATION id 0xc partid 0x2092
    Wed Mar 17 13:55:46 : Wed Mar 17 13:55:46 : 0xc State: DESTROY Adv OwningSubPart: 0
    Wed Mar 17 13:55:46 : PartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092
    Wed Mar 17 13:55:46 : CM DestPartId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092
    Wed Mar 17 13:55:46 : Internet Location - Host: Port Number: 1496 Dot: 172.17.40.232
    Wed Mar 17 13:55:46 : Counters Send: 0 Receive: 0
    Wed Mar 17 13:55:46 : 0 DOM Id 4 LOCSET_DESTLOCSET id 0x2092 partid 0x0
    Wed Mar 17 13:55:46 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092 Version: 14
    Wed Mar 17 13:55:46 : PartitionsUsing: 1 FailedNumber: 13
    Wed Mar 17 13:55:46 :
    Wed Mar 17 13:55:46 : 0 DOM Id 4 PART_DESTREMINT id 0x2092 sub 0x0
    Wed Mar 17 13:55:46 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092 RUNNING - INBOUND
    Wed Mar 17 13:55:46 : Status: SpecialUsage:
    Wed Mar 17 13:55:46 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092
    Wed Mar 17 13:55:46 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:46 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:46 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:46 : UsingLocalPartIds:
    Wed Mar 17 13:55:46 : 0 DOM Id 4 PART_FAULTNOPROXY id 0x2092
    Wed Mar 17 13:55:46 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092 RUNNING - INBOUND
    Wed Mar 17 13:55:46 : Status: SpecialUsage:
    Wed Mar 17 13:55:46 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092
    Wed Mar 17 13:55:46 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:46 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:46 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:46 : UsingLocalPartIds:
    Wed Mar 17 13:55:46 : 0 DOM Id 4 PART_PARTLOST id 0x0 partid 0x2092
    Wed Mar 17 13:55:46 : 0 DOM Id 4 PART_FINCONNLOST id 0x2092
    Wed Mar 17 13:55:46 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092 FAULTED - INBOUND
    Wed Mar 17 13:55:46 : Status: SpecialUsage:
    Wed Mar 17 13:55:46 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092
    Wed Mar 17 13:55:46 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:46 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:46 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:46 : UsingLocalPartIds:
    Wed Mar 17 13:55:46 : 0 DOM Id 4 PART_SENDFAIL id 0x0 error 0x2092
    Wed Mar 17 13:55:46 : SYSTEM ERROR: Attempt to send to a partition
    (F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092) which has no locations
    associated with its shell partition.
    Class: qqsp_DistAccessException
    Error #: [601, 116]
    Detected at: qqdo_PartitionMgr::SendMsg at 1
    Error Time: Wed Mar 17 13:55:46
    Exception occurred (locally) on partition
    "T450MetrixServerComp_cl0_Part1", (partitionId =
    F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095, taskId =
    [F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2095.1813]) in application
    "T450MetrixServerComp_cl0", pid 2916 on node TEAAUS0102 in environment
    centrale.
    Wed Mar 17 13:55:46 : 0 DOM Id 4 PART_FINREMOVE id 0x2092 sub 0x0
    Wed Mar 17 13:55:46 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092 FAULTED - INBOUND
    Wed Mar 17 13:55:46 : Status: SpecialUsage:
    Wed Mar 17 13:55:46 : LocationSetPartitionId: F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2092
    Wed Mar 17 13:55:46 : Message(Send: 0 Receive: 3)
    Wed Mar 17 13:55:46 : Method(Send: 0 Receive: 1)
    Wed Mar 17 13:55:46 : Forward(Send: 0 Receive: 0)
    Wed Mar 17 13:55:46 : UsingLocalPartIds:
    Wed Mar 17 13:55:46 : 0 DOM Id 16 PART_REMPARTDEL id 0x0 partid 0x2093
    Wed Mar 17 13:55:46 : 0 DOM Id 16 PART_DESTREM id 0x2093 sub 0x0
    Wed Mar 17 13:55:46 : F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x2093 RUNNING - OUTBOUND
    Wed Mar 17 13:55:46 : Status: SpecialUsage:
    Wed Mar 17
  • 5. Re: Communication Errors
    865800 Newbie
    Currently Being Moderated
    Was this issue resolved? If yes, please guide how you achieved it as we face the same issue
  • 6. Re: Communication Errors
    866239 Newbie
    Currently Being Moderated
    I think the 'Attempt to send to a partition (F92A9860-B089-11D7-800D-8B9CD7FEAA77:0x209b) which has no locations associated with its shell partition.' is caused because one of ur replicates got shutdown, but Forte did not remove the entry in the naming serivce. Look to see if the number of ftexecs on the machine is the same as the number of paritions(including nones) on the e-console. How long has this bin happening ? Is it recent?
  • 7. Re: Communication Errors
    831397 Newbie
    Currently Being Moderated
    One reason could be
    Having some long running DB queries (check with your DBA) on UNIX.

    Add the log flags

    trc:cm:23
    cfg:sp:5

    and copy the error dump here.
  • 8. Re: Communication Errors
    865800 Newbie
    Currently Being Moderated
    Hi,

    Was this issue resolved? We are facing the same issue and are seeing the below error

    Task 36220: CM Keepalive terminating unresponsive connection for hose 3228 to location Internet Location - Host: Port Number: 1408 Dot: xxx.xx.xx.xxx

    Please help !!!

    Also, I saw some traces to be enabled in a post on this thread

    trc:lo:25 -- tracking exceptions
    trc:cm:*:4 -- tracking communications
    trc:cm:30:2 -- tracking server to connect to


    Are these Oracle traces or Forte Specific? where should they be added?
  • 9. Re: Communication Errors
    831397 Newbie
    Currently Being Moderated
    I gather you are new to that Forté thingy:

    Are you running Oracle and Forté on UNIX ?

    The Keepalive error may caused by irresponsive Forté partition due to Forté threading on UNIX. Check v$session_longops on Oracle.

    These traces are Forté specific.
    Add the flags to the shell script that starts the partition or alternatively there is a monitoring app called eConsole where you can see Foré nodes and Forté partitions running on those nodes. Use it to get to the partition that fails and add the log flags. Please RTFM (F = Forté) on how to do that.

    trc:lo:25:* is not to be used as this will raise many errors that are in fact not real errors.

    Given the little Forté knowledge you seem to have you'd better get your manager to hire a Forté guy.