I wrote yesterday that I hoped that my post would “spark some ideas for your own upgrades and help you maintain momentum.” I’m wowed by the attention the post has received by both Ed Brill and others. We are fortunate as an organization to have an extensive talent pool given that we are a university. However, this work was done by a group of three very talented individuals who work within our technology organization itself. My point being that such a campaign can be developed with minimal resources, with in-house talent, and a short timeframe (we had a four week turnaround).
In addition to the postcard campaign, I’d like to share the digital signage that we’ve displayed on both campuses. Again, the same team prepared the video/sound. Kudos to “The Incredible Power of the Team.”
The alert goes out. “Uh oh, we’ve got another one – a server killer.” Okay, exactly what is a “server killer“? This is the term my team and I have affectionately coined for those processes that may crash your Domino server or at a minimum affect the overall performance of your server. What are these mystery “processes” and what are the symptoms of a “server killer”? Well, have you seen rapid and/or prolonged CPU spikes or a high number of threads on a particular task or the updall task running on names.nsf or on a particular database for hours? Have you seen agent manager spinning on an agent of unknown origin? Where do these processes come from? The most likely source is your user population (and not necessarily your Notes developers *wink*) . Here are some examples.
Example 1: User Agents. Unless the mail template has been customized in your environment, the default ACL for a user is set to Manager. As you know, manager access grants a user with no knowledge of Domino administration or application development – the proverbial keys to the kingdom – at least to his/her own database. The user can easily replace their template with any template available, for example the log.ntf or the pubname.ntf. Or as in this particular case the user can add an agent that they thought would help with say mail rules or make their of out of office agent unique. I’ve also seen users go out to the Lotus Sandbox and pull code from an ancient download (circa 1999) and attempt to put that into an agent. Okay so now what. Let’s say the said agent is set to run on your Domino server and read every name in the Domino Directory once a minute. In a heavily loaded server environment this could potentially become a server killer. So stop and consider…exactly what is preventing your users from each running agents against the Domino Directory?
Prevention: Think long and hard about taking the necessary steps to demote general users to Editor access only. With version 7 and beyond, the users may still perform daily user actions and modify the out of office agent with Editor access. Also be sure to take time to review your server Security Settings to confirm exactly who and/or what groups may run agents and verify agent types and permissions.
Example 2: Cell Phones. A user has gone to the cell phone store and the helpful sales assistant says “Purchase this add-on software for only $5 and you can synch your Notes mail with your phone.” The user buys the software, installs it on their workstation at home, and in the office. The software performs a synch every second against a 5 gigabyte database with 40,000 documents in the inbox from both workstations. You notice that the updater threads on the server have jumped through the roof and users across the domain are reporting that access from the Notes client is slow. How to do you catch such a server killer? First, review the log.nsf by user activity and scan for any large number of reads by a user. Of course, servers and application processes like the Blackberry Enterprise Server will show high reads, but if a user has more reads against his database than the servers combined – you’ve got someone running a server killer. The next step is to review the Database Properties – User Detail to scan for trends. For example, are the reads occuring on a scheduled basis, say every few seconds or did the number of reads against the database suddenly change and is now very high compared to the previous week’s average? This may be a clue that the user has a scheduled process running somewhere performing a synch. The final step is to contact the user. They will usually admit that they did not intentionally mean to crash the mail server, or worse yet cause the CEO’s server to run slow for five days.
Prevention: Have a management approved list of add-on software that is tested/supported in your environment. Otherwise users may feel they have carte blanche to install and run whatever software they want against your Domino servers. Some cell phone software also involves custom agents, so again be sure to demote users to Editor access to limit the scope of unsupported software.
Please note that while these examples may seem extreme, they can indeed affect server performance. If a user is running against a lightly loaded server, there may be less cause for concern. However, as the number of users on a given server increases be sure to monitor your overall server performance daily and avoid those dreaded “server killers.”
I’ve blogged on several occasions about our ongoing Notes 8.5 client upgrade. As many of the recent discussions have focused on marketing/advertising I thought I would share one small component of how we prepared, notified and hopefully helped our users become more engaged about the Notes 8.5 upgrade. The target group in this case was all our email users – and these individuals may or may not have been using the Notes client. So part of the message needed to not only address new features – but also include an overall reason for using Notes. Our in-house marketing group came up with the concept of “the incredible power of one.” I hope that seeing this may spark some ideas for your own upgrades and help you maintain momentum. So far users here continue to be pleased with the Notes 8.5 environment.
Images: © 2009 Virginia Commonwealth University. All rights reserved.