I wrote yesterday that I hoped that my post would “spark some ideas for your own upgrades and help you maintain momentum.” I’m wowed by the attention the post has received by both Ed Brill and others. We are fortunate as an organization to have an extensive talent pool given that we are a university. However, this work was done by a group of three very talented individuals who work within our technology organization itself. My point being that such a campaign can be developed with minimal resources, with in-house talent, and a short timeframe (we had a four week turnaround).
In addition to the postcard campaign, I’d like to share the digital signage that we’ve displayed on both campuses. Again, the same team prepared the video/sound. Kudos to “The Incredible Power of the Team.”
The alert goes out. “Uh oh, we’ve got another one – a server killer.” Okay, exactly what is a “server killer“? This is the term my team and I have affectionately coined for those processes that may crash your Domino server or at a minimum affect the overall performance of your server. What are these mystery “processes” and what are the symptoms of a “server killer”? Well, have you seen rapid and/or prolonged CPU spikes or a high number of threads on a particular task or the updall task running on names.nsf or on a particular database for hours? Have you seen agent manager spinning on an agent of unknown origin? Where do these processes come from? The most likely source is your user population (and not necessarily your Notes developers *wink*) . Here are some examples.
Example 1: User Agents. Unless the mail template has been customized in your environment, the default ACL for a user is set to Manager. As you know, manager access grants a user with no knowledge of Domino administration or application development – the proverbial keys to the kingdom – at least to his/her own database. The user can easily replace their template with any template available, for example the log.ntf or the pubname.ntf. Or as in this particular case the user can add an agent that they thought would help with say mail rules or make their of out of office agent unique. I’ve also seen users go out to the Lotus Sandbox and pull code from an ancient download (circa 1999) and attempt to put that into an agent. Okay so now what. Let’s say the said agent is set to run on your Domino server and read every name in the Domino Directory once a minute. In a heavily loaded server environment this could potentially become a server killer. So stop and consider…exactly what is preventing your users from each running agents against the Domino Directory?
Prevention: Think long and hard about taking the necessary steps to demote general users to Editor access only. With version 7 and beyond, the users may still perform daily user actions and modify the out of office agent with Editor access. Also be sure to take time to review your server Security Settings to confirm exactly who and/or what groups may run agents and verify agent types and permissions.
Example 2: Cell Phones. A user has gone to the cell phone store and the helpful sales assistant says “Purchase this add-on software for only $5 and you can synch your Notes mail with your phone.” The user buys the software, installs it on their workstation at home, and in the office. The software performs a synch every second against a 5 gigabyte database with 40,000 documents in the inbox from both workstations. You notice that the updater threads on the server have jumped through the roof and users across the domain are reporting that access from the Notes client is slow. How to do you catch such a server killer? First, review the log.nsf by user activity and scan for any large number of reads by a user. Of course, servers and application processes like the Blackberry Enterprise Server will show high reads, but if a user has more reads against his database than the servers combined – you’ve got someone running a server killer. The next step is to review the Database Properties – User Detail to scan for trends. For example, are the reads occuring on a scheduled basis, say every few seconds or did the number of reads against the database suddenly change and is now very high compared to the previous week’s average? This may be a clue that the user has a scheduled process running somewhere performing a synch. The final step is to contact the user. They will usually admit that they did not intentionally mean to crash the mail server, or worse yet cause the CEO’s server to run slow for five days.
Prevention: Have a management approved list of add-on software that is tested/supported in your environment. Otherwise users may feel they have carte blanche to install and run whatever software they want against your Domino servers. Some cell phone software also involves custom agents, so again be sure to demote users to Editor access to limit the scope of unsupported software.
Please note that while these examples may seem extreme, they can indeed affect server performance. If a user is running against a lightly loaded server, there may be less cause for concern. However, as the number of users on a given server increases be sure to monitor your overall server performance daily and avoid those dreaded “server killers.”
I’ve blogged on several occasions about our ongoing Notes 8.5 client upgrade. As many of the recent discussions have focused on marketing/advertising I thought I would share one small component of how we prepared, notified and hopefully helped our users become more engaged about the Notes 8.5 upgrade. The target group in this case was all our email users – and these individuals may or may not have been using the Notes client. So part of the message needed to not only address new features – but also include an overall reason for using Notes. Our in-house marketing group came up with the concept of “the incredible power of one.” I hope that seeing this may spark some ideas for your own upgrades and help you maintain momentum. So far users here continue to be pleased with the Notes 8.5 environment.
Images: © 2009 Virginia Commonwealth University. All rights reserved.
A few of our users have reported that since upgrading to Notes 8.5, they have seen a noticeable CPU spike on their workstations when working with certain documents containing graphics and/or these same documents were slower to open. We reviewed their workstation for the usual suspects, and also recreated perweb.nsf, deleted cache.ndk, etc. However, the same behavior continued to occur.
We found references to the following parameters with regards to the 8 client as a technote.
Once these three parms were added to the notes.ini and the Notes client was restarted, documents opened at “normal” display speeds and CPU spikes disappeared!
As our Notes 8.5 client upgrade progresses we’ve compiled the following items to check when users report “slowness” after a Windows-based Notes 8.5 upgrade.
As we’ve upgraded users to the mail8.5.ntf template, occasionally this error message will pop up in the console:
“Mail convert failed on <dbname>: Design Replacement Error: Attempt to use an invalid slot number.”
Running convert -u to specifically upgrade folders does not correct the issue. The database will only open after a fxup has been run.
A work around we’ve found is as follows:
1. Run fixup (or fixup -J if transaction log environment).
2. Open the database and REPLACE the design using the dwa7.ntf template.
3. Manually upgrade the folder design of all folders in the database (use $Inbox as the standard folder design).
4. Once the folder design upgrade has completed, REPLACE the design using the mail85.ntf template.
We’ve seen this behavior on those databases that are used primarily by IMAP clients (all flavors) and are speculating that something has gone amiss with the folder structure. We have also observed this in databases with large/complex folder structures and those that have been repeatedly upgraded from version to version. And as always we recommend taking a backup of the database prior to trying out these steps or call Lotus Support for assistance.
I read Stephan Wissel’s post yesterday and am so thankful I did! I’ve been trying for years to describe the self containment of Domino as an application development platform to co-workers and managers. I need to look no further – Stephan’s diagrams were right on! Bravo! I then wondered – what if you could take those same concepts a few several steps further and extrapolate exactly how much time or man hours are required to implement the non-Domino solution versus the Domino solution? Wouldn’t that provide a black and white comparison that could be used as a knife edge marketing tool? Have we been missing this all along? What about a competition say between a set of Sharepoint developers or standard web developers and a set of Domino developers to create the same web application….how many servers, how many man hours? Is there be a set of measures that could be used to nail this once and for all? Perhaps this would banish the concept that Domino is NOT only an email server? Maybe someone should give Denis Leary a call to act as the spokesperson!