CTI Server

Issue: 

TMAC UI do not show "station not registered" error message when Avaya station is not registered to Avaya SM.

Resolution:

1. Check following key values in TMACServer configuration,

<add key="AgentLogin_LoadStationStatus" value="1"/>

<add key="AgentLogin_CheckStationValidity" value="1"/>
<add key="AgentLogin_CheckStationValidityNumber" value="6200324511"/>

Note : Value for key  "AgentLogin_CheckStationValidityNumber" must be the VDN provided by avaya Admin for station registered check.

2. Also the station type must be value in Avaya must contain "SIP" eg: 9611SIPCC , any other type which do not contain "SIP", the station registration check will fail and TMAC UI allow login even when station is not registered

1. Add secondary AES Address and Port in next line of primary                                                              Eg:                                                                                                                                                           [Telephony Servers]
   apluctpidv002.id.uobnet.com=450
   apluctridv133.id.uobnet.com=450

2. Add alternate Link at the section [Alternate Tlinks]

Eg:

[Alternate Tlinks]                                                                                                                                                      Alternates(AVAYA#CM#CSTA-S#APLUCTPIDV034)=AVAYA#CM#CSTA-                    S#APLUCTRIDV133

Note left side is the primary TLINK and Right side is the secondary TLINK

Issue:

When Agent A transfer vivr chat to Agent B, transfer fail with a red error alert in TMAC UI.

Troubleshoot:

1. Check CTIServer log for CSTA error
2. Check TMACServer log for "TransferInitiated" keyword. Look for error code -10

Resolution:

1. Adding correct DMCC ranges in TMAC exception list (This has fixed resources busy issue)
2. Configured ASAI version as 12 (from 10) on AES webpage under AES TSAPI configuration- This has fixed consult & conference feature used by TMAC to consult and conf direct agent (by clicking on Finish button) – UOBI has AES R10.1.2

Issue: When agent tries to login, TMAC UI error pop up "CTI login failed"

Troubleshoot steps:

1. Check TMACServer log for keyword "SecurityLogin" and error code -1
2. Check TMACServer log for keyword "AgentLoginFailed" and error code -3
3. Check AESCTIWrapper log for "SecurityLoginDone" and  values for ErrorInfo: ResponseStatus and ErrorCode: , if these values are not matching the following TSAPI Error codes.
   1. #define ACSERR_APIVERDENIED -1 /* The API Version
      * requested is invalid
      * and not supported by
      * the API Client Library.
      */
      #define ACSERR_BADPARAMETER -2 /* One or more of the
      * parameters is invalid.
      */
      #define ACSERR_DUPSTREAM -3 /* This return indicates
      * that an ACS stream is
      * already established
      * with the requested server.
      */
      #define ACSERR_NODRIVER -4 /* This error return
      * value indicates that
      * no API Client Library
      * Driver was found or
      *installed on the system.
      */
      #define ACSERR_NOSERVER -5 /* The requested Server
      * is not present in the
      * network.
      */
      #define ACSERR_NORESOURCE -6 /* There are insufficient
      * resources to open an
      * ACS stream.
      */
      #define ACSERR_UBUFSMALL -7 /* The user buffer size
      * was smaller than the
      * size of the next
      * available event.
      */
      #define ACSERR_NOMESSAGE -8 /* There were no messages
      * available to return to
      * the application.
      */#define ACSERR_UNKNOWN -9 /* The ACS stream has
      * encountered an
      * unspecified error.
      */
      #define ACSERR_BADHDL -10 /* The ACS Handle is
      * invalid.
      */
      #define ACSERR_STREAM_FAILED -11 /* The ACS stream has
      * failed due to
      * network problems.
      * No further
      * operations are
      * possible on this stream.
      */
      #define ACSERR_NOBUFFERS -12 /* There were not
      * enough buffers
      * available to place
      * an outgoing message
      * on the send queue.
      * No message has been sent.
      */
      #define ACSERR_QUEUE_FULL -13 /* The send queue is
      * full. No message
      * has been sent.
      */
      #define ACSERR_SSL_INIT_FAILED -14 /* This return value
      * indicates that a stream
      * could not be opened
      * because initialization of
      * the OpenSSL library
      * failed.
      */
      #define ACSERR_SSL_CONNECT_FAILED -15 /* This return value
      * indicates that a stream
      * could not be opened
      * because the SSL connection
      * failed.
      */
      #define ACSERR_SSL_FQDN_MISMATCH -16 /* This return value
      * indicates that a stream
      * could not be opened
      * because during the SSL
      * handshake, the fully
      * qualified domain name
      * (FQDN) in the server
      * certificate did not match
      * the expected FQDN*/
      */
4. If error code doesn't match any of the above return values, note the number value in ResponseStatus, then run query in database table select * from AGT_Agents where pbxid= null or pbxid='0'
5. If any rows in database exists for pbxid=Null or pbxid='o' or pbxid='<any single digit integer value>' then manually update the PBXID field from OCM User Onboarding module with Avaya agent ID value.

Root Cause: PBXID being Null / 0 (zero) / Single digit is a product bug for CTIServer Version < 6.0.04.10

Issue description:

When only one agent is logged-in and two chats are queued to the agent, 1st chat is routed to agent and second chat will be in queue.

Analysis step:

1. Check TMAC for keyword "requeue" event for the second sessionid to confirm tmac has initiated requeue request to comm server.
2. In commserver sip log look for SIP invite for the given sessionid (last 4 digits). If sip invite exists, the sip signalling to CM has initiated
3. Look for TASPI events in CTITASPWRAPPER log for the given session id. If TSAPI session id doesn't exist, that confirms that avaya CM hasn't routed the call to agent.

Resolution:

Check MCH setting in chat skill was set to None or Many in Avaya.

It should be set to Many

Issue Description : -  Customer Information is not Displayed in the TMAC UI.

Components To Verify:- POM CTI wrapper logs, TMAC server logs.

Keyword to search:-  POM CTI Wrapper logs :- Customer UUI data received or not.

TMAC Server logs:- OnOutboundCallInitiated , Incoming call event, UUI DATA event, Outboundcall UUI data.

Troubleshooting :- Collect the UCID of the call with the having the issue. Check the Sequence of events.

Check in the TMAC server logs events as per the screenshot.

Summary/ Next Step:-

1. Whenever we get a preview contact, CTIServer will send two events to TMACServer followed by TMAC UI. OnOutboundCallInitiated (which internally we call it is as Incoming Call event) and OnOutboundCallUUIData event.
2. Once TMAC UI, received OnIncomingCall event, it will create tab and it will enable “Dial and Cancel” buttons in TMAC UI.
3. Once TMAC UI, received OnOutboundCallUUIData event, it will update/render the customer information in the UI.

Issue description: Unable to login to TMAC and error is Invalid extension.

CTI server logs

2020-04-29 20:28:45,003 [3] ERROR TetherfiTSAPIServer.TetherfiTSAPIServer - Exception in ResetSessionForStation:System.NullReferenceException: Object reference not set to an instance of an object.

at TetherfiTSAPIServer.TetherfiTSAPIServer.ResetSessionForStation(String sessionID)

2020-04-29 20:30:34,277 [3] DEBUG TetherfiTSAPIServer.TetherfiTSAPIServer - SecurityLogout:,2,AVAYA#CMSIM#CSTA#AESSIM:2020-04-29 20:30:34,277 [3] ERROR TetherfiTSAPIServer.TetherfiAgentSession - Exception in GetWrapperServerSessionForStation:System.ArgumentNullException: Value cannot be null.

Parameter name: key

at System.Collections.Concurrent.ConcurrentDictionary`2.ContainsKey(TKey key)   at TetherfiTSAPIServer.TetherfiAgentSession.GetWrapperServerSessionForStation(String sessionID)

TSAPI server logs:

2020-04-30 08:48:37,246 [3] DEBUG TetherfiTSAPIWrapper.TSAPIWrapper - maindevice1:GetQueueStatus failed for extension maindevice1, invokeID 84855, reason is acsHandle is nullptr:

Solution: ws port in CTI server and in TMAC server has to be same.

Issue Description:

While configuring an API link from POM CTI server config and application trying to access an external API but fails with the following message ,

"HttpWebRequest The underlying connection was closed: An unexpected error occurred on a send"

Troubleshooting Information:

Step 1: Try to simulate the issue in the lab environment.  The error was not able to be simulated in a lab environment.

Step 2: Try to establish the connection to API via postman - successful

Try to call the API link from the browser - successful

Step 3: Create a custom application to trigger the API call from other servers and observe the behavior.

Simulate the UAT behavior from SIT servers.

Copy the same custom application for API call into 4 SIT servers

Server1: API call fails with error

Server2: API call is successful

Server3: API call fails with error

Server4: API call is successful

Step 4: Do a continuous netstat on the POM WAS server(where the API is deployed)  while making a call to API.

The connection to the port does not establish, we see the request in "TIME_WAIT" mode.

Step 5:Do a trace on the server level with Wireshark application

From Server where API fails: no server hello seen

From Server where API is successful: Server hello seen

Possible Root Cause:

The client-server handshake is failing. https://www.thesslstore.com/blog/explaining-ssl-handshake/

SCHANNEL setting on Registry for ciphers:

Server Hardening:

Since there is a difference between 2 SIT servers, a possible guess could be due to server hardening.

So the server hardening script was run on [Server2: API call is successful]

After the server was hardened the API call failed on Server2

While comparing the cipher and SChannel keys from a hardened UAT server that can make a successful  API call to the same POM WAS server, the difference was seen with the Diffie-Hellman key [enabled=0].

Adding the key to the client servers where the API call failed resolved the issue.

The difference in the key exchange algorithm between the server and client seems to have an impact. The dependency also will be the cipher order used in the server(POM WAS).

https://www.venafi.com/blog/how-diffie-hellman-key-exchange-different-rsa