Notes from the Field with Anywhere365 and Skype for Business
Over the last few months I have been delivering a project that uses Skype for Business and Anywhere 365 to deliver some complex routing scenarios for a customer. As this is my first real heavy weight deployment with Anywhere365 I wanted to share some battle notes.
I will state at this moment that I enjoy working with the product, you can do some really cool and cutting edge stuff with it. The notes are the bumps I found along the way and something you need to consider as you design and deploy. I do not mean for this post to be negative, but rather constructive.
The system requires Trusted Application Endpoints to route calls in and out of a UCC. Depending on your requirements this can range from 4 to 10’s.
- Each UCC requires at least 4 Trusted Application Endpoints (TAE) to start. 1 x Main Endpoint and 3 x Hunting Endpoints. The Main TAE is the one that you will be assigning a Line URI to and will be your main inbound and outbound endpoint from which you make and receive calls into the contact center. The hunting endpoints are used to hunt available agents and invite them into the call.
- All endpoints require a conferencing policy. This is because all calls are conferences, therefore the endpoints need permission to create a conference on the conferencing pool.
- If you want to also have an outbound contact center, you need an additional endpoint that is used for routing. This routing endpoint is also required for inbound agent interception (more on this later)
- Also if you want outbound calling, every endpoint requires a dial plan and voice policy. Calls will be made using these policiesand not the policies assigned to the agent’s Skype object. The endpoint is selected at random. The Caller ID is that of the main endpoint to the contact center or the Agent’s Skype number depending on the setting chosen. This means that hunting endpoints do not require LineURIs, but require EV to be enabled and dial plans, voice policies to be granted.
- Have multiple inbound lines? A modality endpoint is required for each one. The modality endpoint is like the main endpoint but instead can be used to either enter the same flow as the main endpoint, or be used for side insteps. A side instep is where you call a particular number to bypass the default call flow e.g. field service technician calls a service desk number to get through to a dedicated queue rather than having to listen to a bunch of IVR options.
- Want webchat? That’s another 4 x Chat Endpoints. 1 for the main chat endpoint, 3 for chat hunting endpoints. 3 chat hunts allow for up to 3 chat conversations per agent, if you dont add chat hunting endpoints, you can only have 1 chat per agent.
- If you use the WSP naming convention for these endpoints e.g.ucc_hunting00X@domain.com and you get to 100+ then use email@example.com keeping the double zero. This is because their failover script looks for 00 in the sip address to decide whether or not to hide them from the address list. However, if you use your own naming convention, this is also fine, just make WSP aware so they can alter their scripts accordingly.
- More of a Skype nuance this one, if you create multiple endpoints at once (and you will), be sure to check that you haven’t accidentally named two or more endpoints with the same SIP address. Due to the replication delay in Skype if you hit execute duplicates will be entered into the configuration and the unique sip address validation check is “skipped”. This means pain afterwards trying to figure out the duplicates and then deleting and re-creating them.
In order to use the Agent Extension Window, the Inflight Snapper or the Wallboard, you need to connect to a web service. The web service is hosted on all Anywhere365 servers, but is valid on the active server only. By default all scripts and deployment documentation provided by WSP do not clearly show what is required for HTTPS, only HTTP. If you have external agents then you’ll want this web service to be accessible over the internet. Therefore, HTTP is a bad idea, and HTTPS should be used. My recommendation is to use HTTPS both inside and outside of your organisation. Avoid using direct web connection to the Anywhere servers and use a reverse proxy instead, for both internal and external client ingress. This allows you greater flexibility for DR and does not require a URL update to the client software.
Use Split-DNS, or pin point to make your external FQDN of the web service accessible internally e.g. uccweb1.domain.com.
You can use a wildcard certificate, the application is IIS based so it doesn’t care so much for specific SAN entries. However, dedicated web SSL is always recommended. If using HTTPS the web.config file for each Anywhere365 web service will need to be updated to include a HTTPS binding, only HTTP is included by default. If you don’t do this, you will get a 500 Internal Server Error!
If you expose your web service over the internet to accommodate external agents, then there are additional IIS configurations you will want to consider to secure the Anywhere 365 web services. My recommendation is to configure two web sites in IIS. One for Internal and One for External. The internal site would require no authentication or NTLM, while the external website can be configured for Basic Auth to prompt the external agent to authenticate with the web site before being allowed access to the data via the Anywhere365 client side apps. Much like Skype’s web setup, we are taking additional steps to ensure secure access when external.
The interceptor is used to “intercept” incoming and outgoing calls of an agent. If I was the caller and I called the Agent’s Skype Line URI or a federated call over Skype, the conversation I had with that agent would not be recorded in the contact center. Therefore, I have circumnavigated process and jumped the queue. With the interceptor, Anywhere can prevent this from happening, but looking at each invite to an agent and if it contains audio in the SDP body, the invite will be intercepted and re routed to the contact center in which the agent is added.
There is a setting to ignore any incoming audio invites coming from internal domains, or perhaps you want to enable this for internal callers too.
The interceptor is needed for outbound calls too. If outbound calls require to be recorded (conversation and/or CDR) by the contact center then they need to be intercepted. The interception will ensure that the agent is placed into a conference first before placing the outbound call to the callee. The default experience to the callee is that of a normal call, but to the agent it looks like a conference, although, the agent will hear the dial tone played.
A side effect of having interception enabled means that you cannot use reason codes without the agent being a member of another UCC that is not enabled for interception (subscriber UCC). There is a script available for WSP to keep these two UCCs agent lists in sync so at least the admin effort is reduced for MACDs. I fed this back to WSP and they plan to address this in a future release, so hopefully by the time you are deploying, this is no longer an issue.
The interceptor is an MSPL script on each front end server. It has its own service that must be started. Sometimes it will fail to start if RTCSRV takes too long to start up, so make sure it is started or set to delayed start. As a result, if an agent requires interception, then they cannot be registered to an SBA. They must be registered to the FE pool that has the interceptor installed. Otherwise, the invite never makes it to the FE and the interceptor will not know about the conversation
Anywhere gets its configuration from SharePoint lists. Every 10 seconds the configuration is refreshed and cached on the Anywhere server. In delta syncs if you have edited an agent’s SIP address in the agent list, i.e. marital name change for instance, this is not picked up by the delta sync. It seems this is by design and the sync looks for line IDs greater than or missing in that of the last sync to bring down changes, this is why adds / removes work, but edits do not. This appears to only affect the agent list only. So therefore, you have two options. Option 1 – add the agents as a new agent and replace their skills assigned to their old SIP address, then remove their old identity. Or option 2 – restart the UCC with a clean cache to force a full sync. (There is a fix coming).
First Call Blues
After you have restarted or started the UCC for the first time, the first call can take a while to initiate. This is especially noticeable when forwarding between UCCs. Subsequent calls work fine. This is because there is a TLS handshake that needs to be established between the trusted application and the front end pool that it is registered to. This happens on first call, and subsequent calls use the same authentication token from the original handshake.
HA is Server HA, not Voice HA
When discussing High Availability it is important to remember that Anywhere does not perform HA in the same way as Enterprise Voice HA. If an Anywhere server goes down all in progress calls are terminated. No new calls can be received either until such time as the standby Anywhere server has started the UCC service and started the contact centers. This means that HA in Anywhere world is akin to HA in VMware when a host is lost for instance, and all virtual servers are started from a cold start on another host.
Active / Active Servers
It is possible to have two Anywhere servers running in a trusted application pool actively running uccs. However, they must not be running the same ucc application at the same time. If this happens, you will experience weird behaviours like agents not being placed into the correct call, callers hearing hold music as well as the agent speak, caller ID not working correctly and a whole raft of other weirdness.
The Anywhere application is a techie product. What I mean by that is it requires some significant hands on to setup all the advanced features like, automatic HA, DR, Extension Windows etc. When you deploy it, it’s not like a traditional MSI, input a few parameters and away you go, the whole product is deployed and now configurable in an Admin Control Panel, or PS Module. It requires config file manipulations and the deployment of supporting scripts as scheduled tasks to glue some of it together to make it work. Once it’s in it works and works reliably, my intention here is not to tar the product but to ensure it is understood. I do feel that the install process could be worked on to deploy all features and there is definite scope to use SharePoint lists to configure some of the server back end configuration elements in a pure System Admin Portal. This would make the product seem less custom developed app and more enterprise product.
DR is a Dream
I have to say, their DR process and script is probably the best I have seen. In order to get DR to work there are several scheduled scripts that run daily to ensure DR pain is minimal. Each day the config and applications are backed up to the DR standby server. Applications are synched with the DR pool so they exist in both production and DR pools. In a DR event, all that is then required is to execute the failover script to move the trusted application endpoints from the failed pool to the DR pool, update the CDR DB connection string and start the UCC Service on the DR pool. This means that you can perform a DR of Anywhere in less than 30 mins of initial failure.
CDR Database Resiliency and HA
Anywhere relies on the CDR database for historic records and last agent routing. It is recommended therefore to ensure that the CDR database is as close to the Anywhere servers as possible. Anywhere doesn’t have any baked in control or management of SQL like Skype has and therefore supports any native HA and resilient SQL strategy. To Anywhere it’s just an app database that they need to read and write CDR data to, it contains no configuration. So AlwaysOn, Clustering and Mirroring technology can be used for HA.
In a DR situation though, you’ll want to ensure that the CDR database is also replicated to the DR datacenter. My guidance is to follow the supported cross datacenter topologies for the SQL HA technology you are deploying. For Mirroring, it is best to mirror to a local peer with a local witness and configure log shipping to the DR database. In the same way as you would with pChat in Skype. The failover script allows you to update the CDR DB connection string so that Anywhere will be looking in the right place for data.
Wallboard and Snapper Improvements
The client side applications seem to be treated like independent applications rather than extensions of the contact center. Therefore, require localised configuration files to configure for each contact center. The snapper isn’t so bad, but could benefit from an autodiscovery service to auto configure the snapper to look at the agent’s contact center membership. Initial deployment is fine, but MACDs would require snapper reconfiguration by the agent. This is GUI based but nevertheless a step that could be improved.
The Wallboard however relies on the config file for SLA calculations. If a manager wants to change the default SLA calculations this is done in the wallboard config.xml file. It would seemingly be feasible to include these SLA type calculations in SharePoint and push these out to the wallboard
These two small improvements would make the client side applications feel more integrated and easier to manage.
Mark is an Independent Microsoft Teams Consultant with over 15 years experience in Microsoft Technology. Mark is the founder of Commsverse, a dedicated Microsoft Teams conference and former MVP. You can follow him on twitter @UnifiedVale