Application running properly in lab, but not onsite

If the application is not running in one instance say onsite but running in local lab, you may check the following:

  1. The DLLs might be blocked. You must always zip the build and transfer, because onsite server monitoring tools will block internet downloaded files. You can check the properties of all the DLLs and check if it is blocked.
  2. Check the error code. This will tell you where the issue is occurring. You can find this in network tab in console in case of web application or in browser when opening server endpoint. Or you can use other tools to find out this.
  3. You will get some idea in event viewer logs. Following log is seen in event viewer in one of the case. This tells that issue occurred in Microsoft Monitoring Agent in the server. We found that logs were not writing. Even though the log4net.dll and Logger.dll were not showing as blocked in properties, log4net.dll was blocked in one of the deployments. So, that version cannot be used. So we fixed it by using log4net.dll version

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: PerfMon64.dll, version: 8.0.10977.0, time stamp: 0x59cfb3d1
Exception code: 0xc0000409
Fault offset: 0x000000000013f68f
Faulting process id: 0x2abc
Faulting application start time: 0x01d7876805c29212
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.10918.0\PerfMon64.dll
Report Id: 45fe93c3-f35b-11eb-8123-005056886372
Faulting package full name:
Faulting package-relative application ID:





How to use filters in log4net logs?

By default, all logs are printed to logs without any filters. You can choose to write logs that match words or loggers or level ranges.

This must be configured in Log4Net.config file of your application. Following is the sample config file:

<?xml version="1.0"?>
<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>
<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date{ABSOLUTE} [%thread] %level %logger - %message%:Extra Info: %property{testProperty}%:%exception%newline"/>
<appender name="RollingFileAppender1" type="log4net.Appender.RollingFileAppender">
<file value="C:\Tetherfi\TetherfiHome\Logs\TMACWorkQueue\WQ.log"/>
<appendToFile value="true"/>
<rollingStyle value="Size"/>
<maxSizeRollBackups value="5"/>
<maximumFileSize value="2MB"/>
<staticLogFileName value="true"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%thread] %level %logger - %message%:%exception%newline"/>
<appender name="RollingFileAppender2" type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString" value="E:\GIT\WorkQueue\Server\Logs\WQClient.log"/>
<appendToFile value="true"/>
<rollingStyle value="Size"/>
<maxSizeRollBackups value="5"/>
<maximumFileSize value="5MB"/>
<staticLogFileName value="true"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%thread] %level %logger - %message%:%exception%newline"/>
<lockingModel type="log4net.Appender.FileAppender+MinimalLock"/>
<level value="ALL"/>
<appender-ref ref="RollingFileAppender1"/>
<logger name="TetherfiWorkAssignmentClient.WorkItemManager" additivity="false">
<level value="ALL"/>
<appender-ref ref="RollingFileAppender2"/>

"<filter>" tag blocks must be added the following xml block under "appender" tag. Following are the examples:

Logger Filter

<filter type="log4net.Filter.LoggerMatchFilter">
<loggerToMatch value="TetherfiWorkAssignment.Processor"/>
<acceptOnMatch value="false"/>

Each logger corresponds to a namespace and class of an application. In the sample XML provided, There is one logger "TetherfiWorkAssignmentClient.WorkItemManager" defined which will be written to a separate file. Rest all will be written to another file which is defined under "root" tag. In "root" tag, if you want to match logs of a logger and not write it, you can use the above block. This will reject all logs coming from logger "TetherfiWorkAssignment.Processor".

Search Filter

<filter type="log4net.Filter.StringMatchFilter">
<stringToMatch value="GenericTimerOnElapsed"/>
<acceptOnMatch value="false"/>

This filter allows you to filter by search word. This is a 'contains' operation. With this block, all logs having the word "GenericTimerOnElapsed" will not be written.

Level Range Filter

<filter type="log4net.Filter.LevelRangeFilter">
<param name="LevelMin" value="INFO"/>
<param name="LevelMax" value="ERROR"/>

This filter specifies to write logs of minimum type "INFO" and maximum type "ERROR". This means types below INFO such as DEBUG and types above ERROR such as FATAL will not written.

Deny All Filter

<filter type="log4net.Filter.DenyAllFilter"/>
This will reject all logs from writing. This is used in a switch-like scenario. If you want to write logs only from two loggers only, you can add two of "Logger Filter" block, and change value of "loggerToMatch"and "acceptOnMatch" to true. In addition to this, you would want to reject rest of the logs from other loggers. You can then include "Deny All Filter". This is the same for Search Filter. Remember, by default all logs are printed, which is why this is required to prevent logs which does not match any filters. For example:
<filter type="log4net.Filter.StringMatchFilter">
<stringToMatch value="MyFirstPhrase"/>
<acceptOnMatch value="true"/>
<filter type="log4net.Filter.StringMatchFilter">
<stringToMatch value="MySecondPhrase"/>
<acceptOnMatch value="true"/>
<filter type="log4net.Filter.DenyAllFilter"/>
This will write logs which contains words "MyFirstPhrase" and "MySecondPhrase" and reject rest of the logs.

How to add CORS filters for tomcat apps ?

What is CORS ?

Cross-Origin Resource Sharing (CORS) is an HTTP-header based mechanism that allows a server to indicate any other origins (domain, scheme, or port) than its own from which a browser should permit loading of resources. CORS also relies on a mechanism by which browsers make a “preflight” request to the server hosting the cross-origin resource, in order to check that the server will permit the actual request. In that preflight, the browser sends headers that indicate the HTTP method and headers that will be used in the actual request.

When browser throws CORS error?

CORS is a security mechanism built into modern web browsers. It basically blocks all the HTTP requests from your front end to any API that is not in the same “Origin” (domain, protocol, and port—which is the case most of the time).


Resolution :

Inspect the error in the web browser console for which the CORS error has been highlighted. Identify the Access-Control-Request-Headers in the browser network console, add CORS filters in web.xml of the respective tomcat app (Ex: C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\tetherfi-generic-webservice-interface\WEB-INF\web.xml).

Make sure the following tags

  1. "cors.allowed.headers" has all the HTTP access control request and custom headers (ex: Content-Type,X-Requested-With,accept,Origin,Access-Control-Request-Method,Access-Control-Request-Headers,X-RequestVerificationToken).
  2. "" has request origin (ex:
  3. has exposed headers (ex: Access-Control-Allow-Origin).

Below is the sample web.xml for reference,

<display-name>Tetherfi Generic Web Service Interface Application</display-name>



Note that, CORS may still exist if any headers are missing in "cors.allowed.headers" list.


'Could Not Load File or Assembly' in Logs


When the application fails to find and load an assembly (DLL), following exception is thrown:

System.IO.FileNotFoundException: Could not load file or assembly 'System.Configuration.ConfigurationManager, Version=, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.

Reasons and Solutions

  1. The assembly itself is not found.In this case, the assembly is not present in your application path. Copy the assembly from release and paste it here and try again.
  2. The specific version of that assembly is not found.Sometimes, the assembly version referred by the application and the present assembly might vary due to incorrect deployment or during release process.
    1. Navigate to the application path.
    2. Open the application config file and template config file.
    3. Head to the "runtime" section under "configuration".
    4. You will find the assembly binding redirects. These are used to redirect references of "oldVersion" to "newVersion".Following is the sample assembly redirect:<runtime>
      <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
      <bindingRedirect oldVersion="" newVersion="" />
      </runtime>Here, this says, that all references of assemblies from version to (oldVersion) should use (new Version)
    5. So, you should copy the version of Newtonsoft.Json.dll of version
    6. So, if you found an error for this DLL, probably you are using different version DLL.

How to find assembly version?

For TMAC Server, version onwards assembly versions are printed in logs on start up. Until this version, file versions were printed.

Assembly and file versions are different.

To find assembly version, run this command in your powershell window:


Substitute your application path and assembly name here.

Once you find your assembly version, you can verify what version assembly you have and do the needful.


Report To

If this issue was due to the incorrect release and not due to wrong deployment process, report this to the developer so that release can be corrected.

For TMAC components, report to

For WorkQueue components, report to

Copy and Paste between Remote and Local computer stopped working

Sometimes copying or pasting files, texts, etc from/to Windows servers through RDP connection stops working and can be fixed by following below steps


1. Open task manager in server and search for rdpclip.exe under the details tab and end the process


2.Type Windows+R and Enter rdpclip.exe and click OK.


CredSSP encryption oracle remediation - RDP Issue

When we are trying to connect VM using RDP if we received CredSSP encryption oracle remediation issue. Please follow the steps in the attached document to address this issue.CredSSP encryption oracle remediation issue.

How to enable libsip log

a) Description:

Where to look for lipsip logs

note: Lipsip log path will not be mentioned in log4net.

b) Components to Verify:
ComponentWhat to CheckWhere to Check
Any application which connects to CM

Eg: Dialer, Textchat server

Check whether logs folder is createdApplication folder path-> inside the app folder Logs folder should be created

c) Resolution:

Create logs folder inside the app folder if not created. libsip log file will generate inside the logs folder.


Windows authentication is not working when accessing the application even if the credentials are correct

a) Issue Description:

While accessing the application (TMAC/OCM/TMC..etc) in the browser, it is not accepting the Credentials provided even though it is correct.

b) Components to Verify:
ComponentWhat to CheckWhere to Check
Security permissionCheck whether the below access is given to the application folderGo to the application folder path for eg D:\Tetherfi\TMAC

1. Right-click on the TMAC folder and click properties
Got to the security tab and check if the below security groups are added.

-Network service
-Authenticated users



Add the Security Groups IIS_user, IUSR, Network service, Authenticated users in the security groups list. Follow the below steps to add.

  1. Right-click on the folder(eg TMAC) and go to properties.
  2. Click on Security tab ->Click on edit-> Click on Add
  3. Click on the Locations tab and select the server name and click on advanced.
  4. Click on find names and select the security groups and add and save.

Steps to disable log4net logs

Go to the log4net.config file under the application folder.

Change the header <log4net> to <log4net threshold="OFF"> This will stop writting the log.

Log4net.config for reference.



Component Versioning Strategy

Tetherfi standard for component versioning is a 4 doted decimal number of the format[.r] where

  • M - The major version. A single digit or two digit number. This changes only for major refactoring or major roll out of a product.
  • yy - Year of release. A two digit number. Specifies the year component of the release date.
  • mm - Month of release. A two digit number. Specifies the month component of release date.
  • dd - Day of release. A two digit number. Specifies the date of the month component of the release date.
  • r - Optional. Either 'a' or 'b' specifying alpha or beta releases. This component should only be added to alpha/beta releases.

e.g. Chat Server -, TMAC -, Media Server -


.NET Applications

All C# applications should set and maintain version in Project Properties -> Application -> Assembly Information


C++ Applications

All C++ applications should maintain a PrefixVersion.h file (where Prefix can be a application specific prefix) where the version is set. The application should log the version to log at startup.


JavaScript libraries

All JavaScript libraries should maintain version in their source code and must log it (preferably to server side logs) at startup.

WCF, Method not found with 404 resource not found


  • Perform windows search for Server Manager Interface.
    • On selection of Server Manager, the Server Manager dashboard window will be shown. Select Add Role or Feature to open the wizard. Then select Role-based or feature-based installation. Click Next and select the server which you want to work on.

Server Manager

    • Go to Features section, expand either .NET Framework 3.5 Features node or the .NET Framework 4.5 Features node depending on what is installed. If both are installed then we need to do below mentioned steps for both.
    • Expand WCF services section, then select HTTP Activation as shown in the screenshot below,

Features section

    • Keep clicking on next until you install it.


If the above solution doesn't work, try the below approaches.

  1. Check if you are missing config tag <security mode=”Transport”> inside the node <webHttpBinding> 
  2. Check if below tag is present, if not then add under <endpointBehaviors> tag <webHttp helpEnabled="true"  automaticFormatSelectionEnabled="true"/>


System.BadImageFormatException: Could not load file or assembly(from installutil.exe)

This issue usually is seen when the hosting system doesn't support the applications compiled version. It is based on the systems environment the program is going to run. Go to system information and look at the system type whether it is x64 or x32.

Based on system type get the files from release, incase a specific version of files you looking for is not found kindly request for the same.

A connection was successfully established with the server, but then an error occurred during the login process. No process is on the other end of the pipe

Issue description: This is a generic SQL error, occurs during authentication.


  • Ping SQL server from the application server, check if it is reachable.
  • Verify database connection string is correctly configured in the applications configuration file with a valid database username and password.
  • There is a possibility that sometimes SQL service stops automatically, Ensure SQL service is up and running.
    • Go to Windows search and search for Services. On selecting it, below window will show up.

windows services

    • Search for service name SQL Server, ensure status is in Running.  If not, then right-click on that service name and click on properties. The properties window will open as shown in the below screen. Make sure the status type is “Automatic” in the properties window. Click OK and then start the service by right-clicking on it again and then click on start.


  • If service is already up and running, then connect to SQL Server Management Studio using windows authentication. Then expand Security -> Logins and look for user sa, right-click and click on properties.

Security tab showing for user 'sa'


status of permission for user 'sa'

  • As highlighted above, if login is disabled change it to Enabled by clicking on the radio button and click OK. This must resolve this issue.

The Service on local computer started and then stopped

The Service on local computer started and then stopped, some services stop automatically if there are not in use by other services or programs

There are many reasons why a service would stop.  It is best to inspect the Event Viewer to determine the cause of the stoppage.  Possible reasons include:

  • onStart() throwing exceptions
  • onStart() is not starting the thread to work
  • Check the end points in the configuration. There can be issue with the base address or the client endpoints
  • Check for the opening and closing of the keys in configs

To go to Event Viewer search for it in Windows Search.

To check the error, click on Windows Logs -> Application and check for your service error (example pic shown below)

Rectify, solve and start the service again.

System.TimeoutException: The operation has timed out when connecting to servers

When client is communicating with servers, the operation has timed out issue can be seen in logs irrespective of component.

Description: The requesting channel which is client, is trying to connect to server but will not get response within the threshold time. Say sendTimeout is 1 min (see below), which means the response from server should be sent by 1 min. If it doesn't respond by that time then we can observe this TimeoutException.

Resolution: Increasing the sendTimeout value in the config to higher number (ex: 00:05:00 which is 5mins, HH: MM: SS) might solve the issue if the endpoint is reachable. Please refer the example config as shown below.

 <binding name="WSHttpBinding_IFaxer" closeTimeout="00:01:00"
openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
maxBufferPoolSize="20000000" maxReceivedMessageSize="20000000"

System.ServiceModel.CommunicationException: The maximum message size quota for incoming messages (65536) has been exceeded.

Issue Description:  This is general error which occurs when consuming service returns data size more than acceptable or default size.


Modify/Add or append below binding tag properties starting from “allowCookies” till the end of the binding tag in your app config file.

<binding name="somename" allowCookies="true"
<readerQuotas maxDepth="32"

‘System.UnauthorisedAccessException: Access to the path is denied’ error in logs

Issue Resolution 1:

This error is generated when permission for the application pool is not given in IIS.

Permission for the application pool can be given as shown below.

  1. Right-click on the IIS application and click on Edit Permissions as shown below
  2. In the Properties, Select the Security tab click on the ‘Advanced’ button
  3. In the Advanced Security Setting click on ‘Add’ button
  4. Click on the link ‘Select a principal’.Enter the name of the service account and click in ‘Check Names’ button click on ‘OK‘button.
  5. Select the permissions and click on ‘OK’
  6. Details of the newly added service account will be displayed as shown below


Issue Resolution 2:

This error may also generate when permission for the IIS_USER is not given in IIS.

Permission for the IIS_USR can be given as shown below.

  1. Goto IIS and right click on the application and Select ‘Edit Permissions’
  2. Goto ‘Security’ and Click on ‘Edit’
  3. Click on ‘Add’
  4. Click on ‘Advanced Settings’
  5. Click on ‘Find Now’ and search for ‘IIS_USER’ in ‘Search result’ table.
  6. Select ‘IIS_USER’ and Click ‘OK’
  7. ‘IIS_USER” will be displayed in ‘Select user or Groups’ popup. Click on ‘OK’
  8. Select ‘IIS_USER’ in ‘Permissions for UI’ popup and provide all the permissions. Click on ‘OK’
  9. Click on ‘OK’


Issue Resolution 3:

This error is generated when permission for the application pool is not given in IIS.

Permission for the application pool can be given as shown below.

  1. Click on ‘Application Pools’ in IIS and select the application pool of your application.
  2. Right-click on the application pool and select ‘Advanced Settings’
  3. Click on ‘Identity’ in ‘Advanced Settings’ popup
  4. Select ‘Custom account’ and click on ‘Set’
  5. Provide User name and password which is used to access an application
  6. Click on ‘OK’
  7. Right-click on application and select ‘Edit permissions’
  8. Go to ‘Security’, and click on ‘Edit’
  9. Select the application user from the list and provide all the permissions

A network-related or instance-specific error occurred while establishing a connection to SQL Server

Issue Description:

This issue will occur when SQL Server not configured properly.

Issue Resolution:

  • Make sure the Connection String configured properly in the application.

  • Check if SQL Server and SQL Server services are always running on the server hosting your SQL instance.

  • Select one of the services in Windows Services  and click the Properties menu item.
  • Set the service's start-up type to Automatic in the resulting pop-up and hit OK.

  • Right-click the service and select Start, if the service is not already started.

  • To check SQL instance configured for mixed (Windows and SQL) authentication

  • Click on SQL instance name in SQL Server Management Studio. Select the Properties menu item. Select the Security tab in the resulting pop-up.

  • Tick SQL Server and Windows Authentication mode and  restart SQL instance.

  • If your windows service and/or a web application hosted on the different server than the database, make sure your SQL instance set up to allow remote database connections.

  • Select properties by clicking on SQL instance name in SQL Server Management Studio and open the connections tab in the resulting pop-up.
  • Tick Allow remote connections to this server
  • Open SQL Server Configuration Manager and select the Protocols item under SQL Server Network Configuration. There will be a shortcut to SQL Server Configuration Manager in your Start menu.

  • Enable Named Pipes and TCP/IP.
  • Right-click TCP/IP, select Properties and make sure that connectivity enabled for the proper IPs in the IP Addresses tab. Then restart the SQL service.

  • TCP port 1433 is the most basic port used by SQL Server, though other ports are sometimes used as well.

The request was aborted : could not create SSL/TLS secure channel

The main reason for this type of error is
1. Invalid Certificate bind (expired/self signed)
Open certificate details and check certificate is valid and not self signed

2. Path given for certificate is wrong.
Check application configuration (app settings) section for certificate path.

3. Certificate is not binded properly in IIS
• Go to start, click on run, enter mmc, press enter
• Console window will open. Go to File and click on Add/Remove Snap-ins.
• From left pane select certificate. Click Add.
• On adding, Certificates snap-in windows will open. Select computer account. Select local computer.
• Click on finish. Click on ok twice.
• Expand certificates. Expand personal. Select certificate that you have bind in IIS.
• Right click and go to all tasks. Click manage private keys and click add.
• Go to advanced. Click on find now. From the search results, select the IIS_IUSRS
• Click on apply. Click Ok.

Steps for site bindings:
• Copy certificates given by Client team into the respective folder.
• Go to IIS, Go to Domain.
• Click on Server certificates. Click on complete certificate request.
• Enter certificate path and give certificate name. Click ok. Certificate will be listed in server certificates.
• Select default web site(under which your application is deployed).
• Click on bindings, select site bindings based on type and port(Example https 444).
• Click edit, select SSL certificate, click Ok.

Troubleshooting at application side :
Enable Trace logs. This helps to track every remote web request. Make below changes in either web.config or application.config file


<trace autoflush="true" />


<source name="System.Net">


<add name="System.Net"/>



<source name="System.Net.Sockets">


<add name="System.Net"/>



<source name="System.Net.Cache">


<add name="System.Net"/>












<add name="System.Net" value="Verbose" />

<add name="System.Net.Sockets" value="Verbose" />

<add name="System.Net.Cache" value="Verbose" />



Http Error Codes

List of error codes may seen in logs during troubleshooting
Successfully could connect and get the response.
Page not Found.
Check whether application is deployed in web server. It indicates that requested resource is not available.

Internal server error.
There must be error reason shown in the browser.
It could be due to application error or permission errors.

The request requires authentication.
Ex: It is thrown when access windows authentication enabled application without credentials

Access Forbidden.
It could be due to various reasons.
  - Client certificate is required
  - IP address rejected..etc

Bad Request.
Request sent to server is not correct. May be it is not in correct format.
scenario encountered often in one of the project : JWT token sent to server is not valid as it is decrypted with different public key.
Method not allowed
Ex: Server supports POST method but trying to access with GET method.

Error comes when server has a rate limiting

1XX Series is Progress [101 Used for switching protocols from HTTP to Web Sockets]

2XX Series is Success [202 for example is used these days in REST for accepting edit/add a record]
3XX Series is Redirection related [Say HTTP to HTTPS]
4XX Series is Protocol mismatches [Auth required but no auth sent etc.. Notable ones are 400,401,403,405,407,409,413,429]
5XX Series is Server error related [500,502,503,504 are the notable ones]

How to check whether system and application is reachable(ping and telnet)/Connectivity test

Run ping command to find whether system is reachable
ping [IP Address]or[Host Name]
if success,
Pinging with 32 bytes of data:
Reply from bytes=32 time<1ms TTL=128
Reply from bytes=32 time<1ms TTL=128
Reply from bytes=32 time<1ms TTL=128
Reply from bytes=32 time<1ms TTL=128
Ping statistics for
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
if not reachable,
Pinging with 32 bytes of data:
Reply from Destination host unreachable.
Reply from Destination host unreachable.
Reply from Destination host unreachable.
Reply from Destination host unreachable.
Ping statistics for
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Application listens on port. Run telnet to check whether application is reachable
telnet [IP Address] [Port]
if success, it opens blank page.
if not reachable,
C:\Users\User>telnet 8080
Connecting To not open connection to the host, on port 8080: Connect failed
In customer environments, sometimes ping is disabled. Do telnet to make sure application is reachable.
This is applicable to validate any connectivity test with DB, Web services or any other systems.

How to check .NET version

To find .NET Framework versions by viewing the registry (.NET Framework 4.5 and later)

  • On the Start menu, choose Run.
  • In the Open box, enter regedit.exe.

You must have administrative credentials to run regedit.exe.

  • In the Registry Editor, open the following subkey:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full

Note that the path to the Full subkey includes the subkey Net Framework rather than .NET Framework.


If the Full subkey is not present, then you do not have the .NET Framework 4.5 or later installed.

Check for a DWORD value named Release. The existence of the Release DWORD indicates that the .NET Framework 4.5 or newer has been installed on that computer.

The value of the Release DWORD indicates which version of the .NET Framework is installed.


The following table lists the keys of released versions of the .NET Framework only. It doesn't list the keys of preview or pre-release versions.

Refer below link for the .NET version and key verification.