Usability - Productivity - Business - The web - Singapore & Twins

The totally inofficial guide to Verse on Premises

Now that CNGD8ML is upon us, it is story time. Read about the why, who, what and what to watch out for.

To successfully deploy Verse, make sure to carefully read and implement the installation instructions. The availability of Verse makes Domino the most versatile eMail platform around, offering you the choice of: Notes Client, Outlook, POP2, IMAP4, iNotes, Verse, iOS, Android. Anywhay, here we go:

The back story

Verse on premises was a long (out)standing promise to the IBM customer base. Not everybody is ready to embrace the cloud, but interested in the new way to work. In SmartCloud Notes, the backend for Verse in the Cloud, all search is powered by Apache SOLR. If Verse got delivered as is, that would have required substantial hardware and skill investments for the on-premises customers.

So I made a bet with Michael Alexander, whom I worked with on TPTSNBN, that we could use standard Domino capabilities, not requiring Solr. Based on prototypes with vert.x and Java8 we gained confidence and got the go ahead to build the search component as OSGi plug-in (in Java6). So the search part (not the UI or other functionality) is on me.

The team(s)

There were two distinct teams working on the delivery of Verse on Premises (VoP): The core Verse team, that owns UI, functionality and features for both cloud and on premises and the search plugin team responsible to replace the Solr capabilities with native Domino calls.
The former is rather large, distributed between the US, Ireland and China. The later was led by the distinguished engineer David Byrd and just a few core coding members: David, Michael, Christopher, Raj and myself.
We were supported by a team of testers in Belarus and the Philippines. The test teams wrote hundreds of JUnit and Postman tests, just for the search API.

The Orangebox

Each project needs a good code name. The original Verse code name was Sequoia, which is reflected in the name of the plugins for core and UI functionality.

The search component, not being part of RealVerse™, needed a different name. In an initial high level diagram, outlining the architecture for management, the search component was drawn as an orange box. Since we "just" had to code "the orange box". The name stuck and led to our code name "Project OrangeBox" (PoB).
The inofficial Project Orange Box Logo
You can find Orangebox and POB in multiple places (including notes.ini variables and https calls the browser makes). So now you know where it is coming from.

The challenges

We set out to build POB with the following constraints:

  • No changes to the Verse UI code
    The UI makes heavy use of JAGR for aggregation and sends SOLR syntax queries over HTTPS. Short of the message bodies almost everything the Verse client requests is a SOLR search query, including opening the inbox.
    For your enlightenment: Open the developer tools in Chrome and watch what is transmitted over the wire.
    The UI code is completely identical to the code running in the cloud, no splitting of code streams was a design mandate - and no changes from the running production code
  • Java8 would not be ready in time
    So no Vert.x library, no RxJava (it uses threads heavily and that kills the Domino Java6 JVM performance), no lambda functions. Initially I used RxJava to process resultsets into the required JSON. However that didn't survive our load tests, mainly attributed to the JVM struggling with rapid thread transitions
  • Only public APIs
    Any of the calls made from the search plugin to Domino use only publicly documented Java APIs. Since we frequently load tested the application, I learned a big deal on behaviour under heavy load. One example: My initial approach was to use a list of item names I needed and then use document.getFirstItem in a loop to retrieve them. Turns out, under heavy load it is faster to use document.getItems and loop through those, picking the values I needed
  • Light on template changes
    In VoP 1.0 the only changes to the mail template are a few additional views. So you can keep your modified R9 template (eventually) for now - unless you messed with the search order in views and folders.
    However the Verse Client requires data for its list column, that is not in any current view (column). As a result we have to read each and every document that's displayed in the list. Nirmala our Chief of Performance made us work hard to get this fast.
    One of the nice side effects of this effort: Jim Cooper, C expert in the core Domino team, provided code updates (also included in the upcoming FP8), that dramatically improve read performance for any document - a big bonus for all Domino based applications

To solve the challenges I devised an architecture where any incoming query would run through code determining the execution strategy. If determined feasible, VoP would just read a view or folder (eventually with a category and a date limit).
When that wouldn't fit we hand it off to a fulltext query. Raj designed and coded the Query parser that, for the use cases of VoP, translates the SOLR query syntax into Notes FTSearch syntax. Quite a feast in code, kudos to Raj!

Operating it

Make sure, you read the instructions carefully. There are a number of preconditions to be met. Missing any of these and VoP will not work for that users. My suggestion to auto-fix some of them didn't make it into VoP 1.0

  • Your server needs to have https active (and please: no self-signed certificates)
  • There are a few extra design elements needed. Instead of providing a complete new template, you can continue to use your existing template and just add the new views. Only catch: if you have messes with the sort order, you need to reverse that changes. Want to be on the save side? Use the stock 9.0.1 mail template as starting point!
  • The mail files need to have fulltext index active (typically a given for iNotes shops, mostly off for Notes shops)
  • The special: The databases need to have folder references enabled. Read the instruction (16g) on that one carefully
  • The mail server needs designer or manager access to the mail files
  • In a cluster: the clustered mail databases must have the same file path (relative to the data directory) and name on all servers

Watch the log

The users will not get error indicators beyond: "Talk to your admin", but the log files (the one in workspace/logs) will capture details what is missing. It will also log any other errors that might arise. VoP was subjected to thousands of automation test as well as human testing. Nevertheless, it is software, you know where this is going. The PMR system is your friend (and ours too)!

Under the hood

Watching the HTTPS traffic is quite enlightening. I usually use the Chrome tools for it. Go and watch!

To be super clear: Nothing of the data going forth (HTTPS Form submissions) and back (JSON) is documented or guaranteed to stay and can change in any future version (Very much like iNotes, see my articles about that). So don't count on it.

For adventurous souls: the search plugin is using Apache WINK which gives some information away at its base URL /pob/api/search. Go and check it out. Some of the URLs you will find there:

  • /pob/api/search/inbox
    Returns an extended inbox in JSON format
  • /pob/api/search/suggestimportant
    Lists the people Verse believe belong into
  • /pob/api/search/needsaction
    Incoming mail that has been flagged as "Needs Action"
  • /pob/api/search/waitingform
    outgoing mail that has been flagged as "Waiting for"
  • If you happen to be a member of the LocalDomainAdmins or OrangeBoxAdmins group, there are a few more, which are secret and undocumented - and can't be subject to a PMR:
    • /pob/api/search/info
      Shows the version number and some cache information
    • /pob/api/search/monitor
      Real time statistics on search performance (also to be seen in the server stats!)
    • /pob/api/search/parser?query=something
      Takes a SOLR syntax query and shows its Domino FTIndex equivalent. Keep in mind: it isn't an universal translator, but tailored to Verse use cases (you don't want to learn SOLR syntax anyway)
    • /pob/api/search/design
      Various options like show/update design information or extract the current "Verse additional design elements" Template:
      • ?createtemplate=1 extracts the template with the additional Verse design elements. Fresh and crispy
      • ?showviews=1 renders the view design of the additional Verse views for inspection (to check if they are right)
      • ?replicaids=current submits a background job creating all Verse preconditions (FTIndex, Views, FolderRef) in the current database
      • ?replicaids=id,id,id same as above, but for the databases with the given replica id
      • ?serverwide=1 samee as above, but for all mail files (pulled from names.nsf) on that server. Do this only after a backup. You have been warned! Actually don't do this. It's not supported
      Use on your own risk
    As mentioned: Use at your own peril! (and not on a production system!)
As usual YMMV

Posted by on 30 December 2016 | Comments (10) | categories: IBM Notes


  1. posted by Lars Berntrop-Bos on Monday 02 January 2017 AD:
    Wow, great stuff! Thanks for all the hard work on getting this out the door, and many thanks for sharing all the isider info. That helps us implement this a lot easier!

    Cheers and kudos to the Project OrangeBox team!
  2. posted by Chris Whisonant on Wednesday 04 January 2017 AD:
    Thanks for the information. I especially like the information at the end about the additional options for Admins. Thanks also for all of the hard work on this project to you and the rest of the IBM team members.
  3. posted by Chris Whisonant on Wednesday 04 January 2017 AD:
    I've been trying to get the admin URLs to work but have run into some issues.

    If I go to /pob/api/search/info I get the following in the browser:
    {"error" : {
    "failure" : "Authorization required",
    "message": "You need to be admin, but you aren't"
    And on the server's error-log-0.xml I get the following message:

    [16CC:000C-1264] 01/03/2017 02:10:47 PM HTTP JVM: Database %1 has not been opened yet. For more detailed information, please consult error-log-0.xml located in C:/Program Files/IBM/Lotus/Domino/data/domino/workspace/logs

    I know that you're not here to provide support, but it also sounds like we may not be able to get support on the metrics part of this. Is this something that you've seen in your labs that you know how to correct. I created the OrangeBoxAdmins group with users in it and restarted the server. I'm now in both LocalDomainAdmins as well as OrangeBoxAdmins groups. Any thoughts are appreciated - thanks!

  4. posted by Paul Withers on Wednesday 04 January 2017 AD:
    Thanks for the efforts of you and the team (especially including Jim). The benefits and lessons learned for all developers are welcome. I hope there will be further enhancements in future versions, especially leveraging non-IBM code.
  5. posted by Carsten Lührmann on Wednesday 04 January 2017 AD:
    Great article, thanks for sharing some insight into IBMs development and the technology behind the scenes.

    „[...] Jim Cooper, C expert in the core Domino team, provided code updates (also included in the upcoming FP8), that dramatically improve read performance for any document [...]“
    Sounds great to me. However, maybe IBM should also deal with the Domino time creep issue (modifying documents at a rate greater than the time stamp resolution may lead to modification dates creeping into the future), this would make these performance gains even more useful.
  6. posted by Chris Whisonant on Wednesday 04 January 2017 AD:
    Stephan, thank you for the response.

    I decided to create a mail.box (we have mail1.box and mail2.box) and that has stopped us from getting the "Database %1 has not been opened yet." error message. However, from the browser we get the same error at https: //server.com/pob/api/search/info that I posted above about authorization. The only thing we get now in the error-log-0.xml on the server is the following.

    <CommonBaseEvent creationTime="2017-01-04T08:11:50.944-05:00" globalInstanceId="EL0ac30b2400015966cae0a2000001c9" msg="CLFAD0251W: Requesting user authentication. HTTP Code: 401" severity="30" version="1.0.1">
    <extendedDataElements name="CommonBaseEventLogRecord:level" type="noValue">
    <children name="CommonBaseEventLogRecord:name" type="string">
    <extendedDataElements name="CommonBaseEventLogRecord:sourceClassName" type="string">
    <extendedDataElements name="CommonBaseEventLogRecord:sourceMethodName" type="string">
    <sourceComponentId component="Expeditor 6.2" componentIdType="ProductName" instanceId="" location="AMAZONA-NLGRBH9" locationType="Hostname" subComponent="" threadId="9" componentType="http: //www.w3.org/2001/XMLSchema-instance"/>
    <situation categoryName="ReportSituation">
    <situationType xmlns:xsi="http: //www.w3.org/2001/XMLSchema-instance" xsi:type="ReportSituation" reasoningScope="INTERNAL" reportCategory="LOG"/>

    If I go to https: //server.com/pob/api/search/monitor, I get nothing in the logs but I get this in the browser. Same message but it's formatted better.

    "You need to be admin, but you aren't"
  7. posted by Stephan H. Wissel on Wednesday 04 January 2017 AD:
    Chris: You need to have a look into the log to see what was happening. The Java API that we use to determine group membership only works in the context of a database, so I use mail.box (Which usually works even when you have more than one mailbox). So what's in the log?
  8. posted by Chris Whisonant on Thursday 05 January 2017 AD:
    Just an update after Stephan and I chatted further about my issue. We decided to try deleting the existing mail1.box, mail2.box, and mail.box files and starting the server back configured to only use a single mail.box. After doing this, the search metrics started to work in the browser. Thanks, Stephan, again.
  9. posted by Tony Byrne on Tuesday 10 January 2017 AD:
    Hi Stephan,

    It looks good but my folders don't work, they give an error saying, 'Something went wrong. We are having trouble displaying your messages.' The console shows...

    HTTP JVM: A required argument has not been provided. For more detailed information, please consult error-log-0.xml located in /home/notes/domino/workspace/logs
    HTTP JVM: Can't open folder: . For more detailed information, please consult error-log-0.xml located in /home/notes/domino/workspace/logs

    Any help would be appreciated.

  10. posted by Stephan H. Wissel on Tuesday 10 January 2017 AD:
    the article mentions all you need:
    - read your log
    - the PMR system is your friend
  11. posted by Chris Whisonant on Wednesday 11 January 2017 AD:
    I do have another question about the technology. When you mentioned that the core changes will "dramatically improve read performance for any document - a big bonus for all Domino based applications" do you mean "Domino" as in only HTTP client reads or Notes client reads also?
  12. posted by Tihomir Pantovic on Thursday 19 January 2017 AD:
    Just put IF2 for 9.0.1FP7 and it looks like it broke something...

    Might be some regional settings stuff, I'll try to discover later today.

    "Something went wrong. We are having trouble displaying your messages." in UI and on console:
    19.01.2017 09:55:10 - Formula Error [/mail/vuser1.nsf/0/13B772902415FFB2C12580A8006612B0/?OpenDocument&sq=1&Form=l_JSVars&PresetFields=s_HandleAttachmentNames;1,s_HandleMime;1,s_HandleStoredForm;1,s_OpenUI;1,s_HideImages;0,s_HideRemoteImage;1,s_MIMEInIframe;0,s_CDInIframe;0,s_ForceUseInLineHtml;1,s_AppSupportsFormList;memo%7Cappointment%7Creply%7Cnotice%7Cpersonal%20stationery%7C,l_GetBodyEE;1,HTMLOptions;RowAtATimeTa and 30 more bytes] CN=User1 LastName1/O=Domain
  13. posted by Tihomir Pantovic on Friday 20 January 2017 AD:
    Submitted PMR, looks like FP7 bug with Non-US regional settings:

    { Link }
  14. posted by Andrew Magerman on Friday 20 January 2017 AD:

    I like the design decisions you made. Looks nice and clean, and won't bite you, hopefully. Last time I was this impressed by a Domino design decision was for DAOS, and before that, the new @Formula Engine that Damien Katz wrote.

    Happy that some of the improvements done for Verse serendipitously also improve the performance of the Domino Server.

  15. posted by Tihomir Pantovic on Saturday 21 January 2017 AD:
    Update #3 (last one, i promise Emoticon biggrin.gif )
    IF3 for iNotes solves this issue.
  16. posted by Daniel Reimann on Wednesday 08 February 2017 AD:
    Regarding Tony Byrne's issue above (Comment #9): I had the exact same problem although I definitely followed all instructions. This here solved it for me and I would have never come to that solution myself. Kudos to Yingle Jia!

    { Link }

    It's the setting "Make this site accessible to web search site crawlers" that has to be DISABLED for folders to work in Verse On Premises altogether.