Communication Server

VIVR Text chat disconnects with "session closed" message after connecting to live agent via CM routing in 45sec.

Issue Description:

After self service flow is complete, chat is requeued to live agent via CM routing. Chat session abruptly ends after 45-50 seconds even when chat interaction is on-ongoing. VIVR page shows "Session end"  message

 

Troubleshooting steps:

  1. Check chat server log to identify OnDisconnect event is from customer side or agent side
  2. Once agent side disconnection is confirmed check TSAPI client log to see the TSAPI error code
  3.  Run trace on CM for the trunk used for Chatserver until the chat disconnection occur
  4. Check for CM trace log for disconnection, if the trace log say Temporary Failur 1192 error message it will confirm that the issue is with SIP call.

Root Cause:

When requeue happens from TPPVA >TMACServer > ChatServer (TComm) > Avaya CM , there will be an active SIP call from TComm to Agent Station. If the IP-IP Direct Media is not enabled for the used trunk in Avaya CM then call will get disconnected after 45seconds timeout

Resolution:

Follow below settings as per TComm Server deployment guide.

     

  • And go to next page by pressing “F7”, Fill the details as shown below:

Transport Method: tcp

Near-end Node Name: procr

Far-end Node Name: TextChatServer2 (node name)

Initiate IP-IP direct media : Y.

Near-end Listen Port & Far-end Listen Port: Default it will be 5060, you can use other ports provided you are able to telnet to the CM IP and that port successfully

press “F3” function key to submit the values.

TComm Server unable to connect to Workqueue with URI exception

Issue:

TComm server is unable to connect http address of Workqueue due to URI exception

"The provided URI scheme 'http' is invalid; expected 'https'"

Cause:

This is because Security mode in TComm config for WQ endpoint is set to Transport and WQ address given is http.

Resolution:

Use Security mode as None if Workqueue end point is http

Second / Concurrent chat is not routed to agent TAMC UI - Avaya CM

Issue description:

When only one agent is logged-in and two chats are queued to the agent, 1st chat is routed to agent and second chat will be in queue.

 

Analysis step:

  1. Check TMAC for keyword "requeue" event for the second sessionid to confirm tmac has initiated requeue request to comm server.
  2. In commserver sip log look for SIP invite for the given sessionid (last 4 digits). If sip invite exists, the sip signalling to CM has initiated
  3. Look for TASPI events in CTITASPWRAPPER log for the given session id. If TSAPI session id doesn't exist, that confirms that avaya CM hasn't routed the call to agent.

Resolution:

Check MCH setting in chat skill was set to None or Many in Avaya.

It should be set to Many

Customer Details passed by upstream is unable to decrypt once received on TMAC

Issue Description: In DBS customer details are passed to the Chat server when the chat is connected which will be in an encrypted format. These encrypted details will be decrypted and displayed on TMAC but in this case, it is not decrypted. Below Screenshot for reference.

Cause:

  1. The encryption-decryption key is not configured correctly in the chat server config- check for the below key(decrypt.key) and if the key is present in the folder

 

2. Check if decryption key values is enable or disabled

3. Check if the front end is using a different encryption method to encrypt the data

 

Resolution:

  1. Enable.decrypt value in rconfig should be true.

 

 

 

 

SIP Call Failure exception in textchat server logs while initiating chat from test app.

Issue:SIP Call Failure exception in textchat server logs while initiating chat from test app.

:

 

Component to check: TextChat Server. Testapp (chat proxy)

 

Log to check :    SIP Call Failure exception !:TextchatServer.Customer+SIPCallFailureException: StartFailed  in Text Chat logs.

 

Root Cause:  The issue was due to wrong pbx port number configured in rconfig file.

 

 

Resolution:

1.Make changes in rconfig file “pbx1.addr” (should be the CM IP) and “pbx1.port” (should be the port to connect to CM from textchat).

Text Chat server: “ Only one usage of each socket address (protocol/network address/port) is normally permitted "

Issue Description:

Text chat service is not starting and shows socket address usage error

Components to Verify

Text chatserver.exe file

Text chat server logs

Keywords to Search

Go to Text chat server folder. Open TextchatServer.exe file. ‘Log4NetConfigFile’ will have the path to Text chat server logs.

Search for the below keyword

socket address

Troubleshooting

In text chat server logs below log lines are printed

System.Net.Sockets.SocketException (0x80004005): Only one usage of each socket address (protocol/network address/port) is normally permitted at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)

Verify the value of socket in all the text chat service configuration deployed in the same server. Socket values should be unique. Refer the below keys where port no are given

<add type="tcp" port="4411" user="agent" is-secure="true"/>

<add type="tcp" port="3324" user="customer" />

<add type="tcp" port="12234" user="customer" />

<add type="smm" port="44561" user="customer" />

<add type="tcp" port="2258" user="customer" is-secure="true"/>

<add type="websocket" port="35477" user="monitor"/>

<add type="websocket" port="35478" user="customer"/>

Resolution

Using netstat command check if the port number is already used or not. If used, change the port number from the text chat server configuration .

Command : netstat -ano| findstr 5050 [where 5050 is the port that is checked]

If there are no entries after the command execution as above, it means the port is free and not used by any application. If any application is using the port, it would show up as "Listening"

TextChat server is not starting and showing error message "System.UnauthorizedAccessException: Access to the path 'Global\livechat_textchat_stat_event' is denied " in log

  •   Issue Description

Text chat server service is not started from the Service.

  • Components to Verify
          • Text chat server logs
          • Text chatserver.exe file
  • Keywords to Search

Go to Text chat server folder. Open TextchatServer.exe file. ‘Log4NetConfigFile’ will have the path to Text chat server logs.

Search for the below keywords

      • Global\livechat_textchat_stat_event
      • Access to the path
      • Config error:System.UnauthorizedAccessException
  • Troubleshooting
    • Verify if the below log lines are printed in the text chat server logs

                  TextchatServer.Server - Config error:System.UnauthorizedAccessException: Access to the path                                         'Global\livechat_textchat_stat_event' is denied.

    • Check value of the key ‘instance ‘in Textchatserver.config of all the textchat services deployed in the same server

Eg: <add key="instance" value="livechat"/>

  • Resolution

The duplicate instance name is added which is already there in running chat server .The instance name should be unique. Change the value of Instance name in the config.

Eg  <add key="instance" value="livechatDigi"/>

 

Chat not able to connect to agent with error "SIP Call Failure exception! : Textchatserver.customer+SIPCallFailureException: StartFailed"

a) Issue Description: Chat unable to connect to an agent even if the agent is available and a callback is registered.

b) Components to Verify:
ComponentWhat to CheckWhere to Check
TextChat server configMaxChatCallDurationInMinutes key valueGo to TextchatServer folder -> Textchatserver.exe.config -
c) Keywords to Search:
ComponentKeywordsLog Path
TextChat Server LogsSIP Call Failure exception! : Textchatserver.customer+SIPCallFailureException: StartFailedGo to TextchatServer folder -> Textchatserver.exe.config -> key ‘Log4NetConfigFile’ will have the path to TextchatServerLogs
libsiplogSession Interval Too SmallGo to TextchatServer folder\Logs\ libsip.log

Text chat sefver log:

Libsip Log:

Resolution:

In Textchatserver config, check the max chat duration key value.

In CM, we can run the below command and on the 2nd page we can see the below data:

display tac <number>

Here the number is the value of the trunk access code. list trunk-group command will give list of all trunks and from there we should be able to get the TAC number as well.

The parameter Preferred Minimum Session Refresh Interval(sec) should be less than that configured in Chat Server.

 

 

 

 

 

 

 

TComm Server queuing same customer session twice

a) Issue Description:

TComm Server shows logs "Routing to Agent via 'WorkQueue'" twice for the same session

b) Cause

CallAgentMessage config of TComm Server and AutoCallQueue config from TComm Proxy not in sync

c) Troubleshooting

Duplicate queuing for same session happens when AutoCallqueue on TComm Proxy config is set to true and enable.callagentMessage of TComm server rconfig is set to false. Ensure to keep both configs to true so that session queuing happens only once for one chat session request.

Below screenshot shows AutoCallQueue config in tcomm proxy config

Below screenshot shows enable.callagentMessage config in tcomm server rconfig

Call routing and queuing not happening in TComm server

a) Issue Description:

TComm Server shows logs stop at session start and doesnt show routing and callqueue after that

b) Cause

CallAgentMessage config of TComm Server and AutoCallQueue config from TComm Proxy not in sync

c) Troubleshooting

Call routing and queuing won't happen in TComm Server for new session request when AutoCallqueue on TComm Proxy config is set to false and enable.callagentMessage of TComm server rconfig is set to true. Ensure to keep both configs to true or false so that call routing and queuing happens for the new session request

Below screenshot shows AutoCallQueue config in tcomm proxy config

Below screenshot shows enable.callagentMessage config in tcomm server rconfig

Text chat service crashes immediately after start

Problem Statement:
Text chat service was stopping immediately after 5 to 10 secs for service start. Text chat service logs were not updating with any error after service start.

The event viewer had the below error.

Resolution:
Found that software called Cybereason ActiveProbe is blocking our application and stopping out service to connect to the far end,

Check the software's running in the server using the control panel as below.

Service was able to start after the software was disabled by the system administrator.

Unable to initiate chat using Test app with error "User Object is null" in Text chat server log

a) Issue Description:

Unable to initiate a chat successfully using the Test App with "user Object is null" error is seen Text chat server log.

Test app error:

 

b) Components to Verify:
ComponentWhat to CheckWhere to Check
Text Chat Test appIn API config, "message_handler_class"tomcat folder of test app/conf/config/api.config
CommLib.dllDll VersionGo to TextchatServer folder /CommLib.dll

c) Keywords to Search
ComponentKeywordsLog Path
TextChat Server Logsuser Object is nullGo to TextchatServer folder -> Textchatserver.exe.config -> key ‘Log4NetConfigFile’ will have the path to TextchatServerLogs
Tomcat Webapi logsRead write errorGo to Tomcat folder\ Logs\ Webapi.log

Resolution:

This issue is usually faced during the TextChat server upgrade and when there is a change in the Commlib.dll version. And based on the commlib.dll version "message_handler_class" value in api.config differs.

If CommLib.dll version <= 2.18.12.14  then message_handler_class in api.config should be BasicMessage.

If CommLib.dll version >= 2.19.01.11  then message_handler_class in api.config should be AdvancedMessage.

(Note: Usually while setup there is a possibility that it will be changed in the path Tomcat folder/Webapps/war folder/configs/api.config  instead it should be changed in tomcat folder/conf/config/api.config )

 

 

 

 

 

Comm Server doesn't proceed to submit incoming message from Line to Workqueue/TMAC

a) Issue Description

When a Line user sends message, message reaches Comm Server via SMM, but Comm Server doesn't proceed to send it to Workqueue/TMAC and seems to stop at Session Started event. No logs after the one shown below

b) Cause

This happens because of enable.callagentMsg config set to true in Comm server run time configs

c) Troubleshooting

With enable.callagentMsg set to true in Comm server rconfig, Comm server expects another message from requester after initial message to start a conversation. This is required when we use text chat interactions using Comm proxy wherein first message is usually considered as just a session start request and subsequent messages are actual messages to be responded to. Hence, Comm Server waits for a second message to come from same user and then sends to TMAC for a response. Since we are using SMM Proxy and Server for interactions from social media, every message is to be responded to, including the first, enable.callagentmsg config in rconfig needs to be set to false. Both the snapshots of rconfig key and port in comm server config and callagentmsg config in rconfig are given below

In above screenshot of comm server config, rconfig port is 65500 and appname is tchat. so, you can open rconfig in the same server with url http://localhost:65500/rconfig_tchat as shown below, find the enable.callagentmsg config, select and change value to false.

 

System.DllNotFoundException: Unable to load DLL 'libsip.dll'

a) Issue Description:

TComm Service throws error on start and "Unable to load libsip.dll" error is observed in TComm logs as below

2019-10-01 17:00:24,867 [5] ERROR TextchatServer.Program - Config Error(1):System.DllNotFoundException: Unable to load DLL 'libsip.dll'

b) Cause

TComm is not starting due to missing Microsoft Visual C++ 2010 x64 redistributable

c) Troubleshooting

Microsoft Visual C++ Redistributable 2010 x64 is one of the pre-requisites for TComm Server Installation. Make sure to download and install the redistributable from https://www.microsoft.com/en-in/download/details.aspx?id=14632 and then start the TComm Service.

Chat disconnected in Queue while Agent did a Blind Transfer to Queue

Issue:
Agent A transferred a chat to the Queue. While the chat was in Queue, it got disconnected even before Agent B could receive the chat.

Component to Check:  Communication Server

What to Check:

  • Find rconfig.port & rconfig.appname key in Communication Server config. Go to the browser and type
     http://localhost:<port>/<appname>
  • Enter the username and password
  • Check the value of "blindtransfer.timeout"

Log Keywords:

In Communication Server logs, 'Waiting with blind transfer timeout" keyword can be found as below.

If the value rconfig is set to 3000, then 3 seconds after Agent A transfers the chat to the Queue, Communication Server will disconnect the chat.

Resolution:

Increase the value of blindtransfer.timeout key via rconfig

or

stop the communication service, update the below key in rconfig.dat file and start the service.

<config name="blindtransfer.timeout" value="60000" type="uint" />

 

Textchat gets disconnected immediately after being received on TMAC

Issue Description:

As soon as the Textchat is received on TMAC, the chat gets disconnected

Components to Verify:
ComponentWhat to CheckWhere to Check
TMAC Server.1. Check TMAC - template.config for the key "TextChat_ServerConfig"
2. In TextChatServerConfig.json, check for "TextChatServerIP"," TextChatServerPort" & "ChatNumbers" configured.
TMAC - template.config
Communication Server1. Check TextChatServer.exe.config for the keys "cid.rangeStart", "cid.rangeEnd" & the listener port for agent as below

<add type="tcp" port="2233" user="agent"/>

The port and the range should be same as that configured in TMAC Server -> TextChatServerConfig.json

TextChatServer.exe.config
Keywords to Search:
ComponentKeywordsLog Path
TMAC Server LogsStart FailedGo to TMACServer folder -> AMACWebServerWin.exe.config -> key ‘Log4NetConfigFile’ will have the path to TMACServerLogs
Troubleshooting:
ComponentDetails
TMAC ServerCheck the output telnet <CommunicationServer> <agentPort> from Application Server where TMACServer is hosted

 

Staffed agents check by Communication Server gives -2 from TMAC Server

Issue Description:

Communication Server before initiating a call to Avaya CM Skill will send the VDN to TMAC Server to check the number of agents staffed for that particular skill. The response from TMAC  shows Staffed -2

Components to Verify:
ComponentWhat to CheckWhere to Check
TMAC Server.1. Check TMAC server logs if the request from Chat Server has reached TMAC
2. Check if TMAC Server is able to communicate with SMSAPI Service
TMAC server logs.
Communication Server1. Check if the correct TMAC Server end points are configuredrconfig - csodetails.url
Keywords to Search:
ComponentKeywordsLog Path
TMAC Server LogsGetQueueStatusGo to TMACServer folder -> AMACWebServerWin.exe.config -> key ‘Log4NetConfigFile’ will have the path to TMACServerLogs
Communication Server LogsStaffed =Go to CommunicationServer folder -> TextChatServer.exe.config -> key ‘Log4NetConfigFile’ will have the path to Communication Server Logs
Troubleshooting:
ComponentDetails
TMAC Server LogsCheck the output of GetQueueStatus
Communication Server LogsCheck the status code for Staffed:
-2 : VDN is invalid or cannot get the VDN Details from SMSApi
-3 : Invalid Skill
-4 : TSAPI Failed. Unable to get the Queue

 

Single Chat request from customer shows double on TMAC Wallboard - Work Queue Routing

Issue Description:

Customer initiates a chat, on TMAC wallboard, the Calls in Queue is seen as 2 instead of 1.

Components to Verify:
ComponentWhat to CheckWhere to Check
Communication ServerCheck Communication Server logs for keyword - "AddWorkItemWithCustomIdentifier" is written twice for the same Chat Session ID.Communication server logs.
Keywords to Search:
ComponentKeywordsLog Path
Communication ServerAddWorkItemWithCustomIdentifierGo to Communication Server folder -> TextChatServer.exe.config -> key ‘Log4NetConfigFile’ will have the path to Communication Server Logs
Troubleshooting:
ComponentDetails
Communication Server
  1. Open rconfig in the browser of application Server: http://localhost:<rconfig.port>/<rconfig.appname>
  2. <rconfig.port> & <rconfig.appname>keys are present in TextChatServer.exe.config
  3. Enable.CallAgentMsg key should be set to 'true'

 

Customer details are Encrypted on TMAC during Chat Interaction

Components to Verify:
ComponentWhat to CheckWhere to Check
Communication ServerIn rConfig - Check the value of 'enable.decrypt' keyhttp://localhost:<port>/<name>
The keys is defined in Communication TextChatServer.exe.config:<name> - file name
<port> - port on which the app is running
Communication ServerIn rConfig - Check the file name configured at 'decrypt.keyfile' keyhttp://localhost:<port>/<name>
The keys is defined in Communication TextChatServer.exe.config:<name> - file name
<port> - port on which the app is running
Next Steps:
  1. Change the 'enable.decrypt' value to 'true'
  2. Check the 'decrypt.keyfile' file and compare if private & public key are of the same set.

Text chat session timeout.

Issue Description:

Chat end reason is empty in OCM.

Components to Verify:
ComponentWhat to CheckWhere to Check
TMAC Server.Get the UCID/SIP ID and search for connection handle in the TSAPI events. Filter with the connection handle and go to TSAPI Disconnect Event.

Check for the disconnect cause in the disconnect event.

In normal scenarios disconnect cause will be  as -1

Apart from this we can classify it as an abnormal disconnect.

Also check if agent has exceed the chat duration limit.(check it the Limit is Exceed in the Session- Expries field)

TMAC server logs

Text chat server LibSip Logs

Keywords to Search:
ComponentKeywordsLog Path
TMAC Server LogsDisconnect:Cause:Go to TMACServer folder -> AMACWebServerWin.exe.config -> key ‘Log4NetConfigFile’ will have the path to TMACServerLogs
Troubleshooting:
ComponentDetails
TMAC Server Logs1) Check when the call got disconnected
Next Steps:
ComponentDetailsHow to perform operation
AESTo check with AES Team why the disconnect cause is different and what is the reasonNA