I have spent many years working deep within the Notes Indexing Facility (NIF). I’m not done, because there is more to be done and harvested from the rich functionality well-bundled from day one by the earliest Domino developers. It can be a challenging area because so much variety of data flows through the code and onto the glass. If one is not careful, you will make one user story a delight while another becomes quite less so. 

One of the most problematic operations in NIF is its handling of updates. Someone could say “Well, what is the problem? Other database products have no issue updating their indexes, why is Domino any different?” The answer lies in the ease with which views and folders are created in Designer. It is a very common story that views constantly grow in number and complexity. 

So, I often reply to the question of the difference this way: “If you went to your MySQL or Oracle or DB2 or MongoDB administrator and asked for 200-1000 indexes on your database, what would they say?” Of course, they would laugh you out of the room, as they should. Yet, that number of indexes is quite normal on a Domino database and it is why they are updated lazily, on a scheduled or as triggered or forced. There are a variety of update/refresh rules and I will not go into them here. 

Some of the costliest issues with NIF are due to excessive queueing for updates. Most of the APIs and documentation call it “refreshing” when a user opens a view and wants to see the latest updates, which is completely reasonable. The trouble what when a large number of users (well, threads) do that at once, they each compete for the right to update the view in a “fair read/write” queue – that is, those who only want to read get no preference over writers (we don’t starve updates). The result can be a breach in Service Level Agreement (SLA) timings and opening a view simply takes too long. 

Following is a diagram I put together for SPR JCUS8MXLA2. Included in the mix is the “Update Task” representing scheduled, dedicated view updating: 

You may or may not know that in the v9.01 feature packs, Domino development rather quietly produced a feature to attack this problem. It is called “inline view updating” or just “inline.” You can toggle it on via

load updall <database name> [-T <viewname>] -inline on/off

What this does is to apply view updates at time of document updating. All refresh requests at view open time can therefore be treated as nops, as they are. Document updating becomes a heavier unit of work, but the view open and read performance trade-off is worth it.

A second effort was launched in Domino V10, which detects and then automatically creates a dedicated update thread to high usage views. It is enabled via NIF_VIEW_USAGE_ENABLED=1 in notes.ini.

I wanted to measure the effect of the first effort, -inline, so this summer at HCL Chelmsford (well, at home due to COVID-19) I’ve had the privilege of working with an intern in core Domino, Joseph Calles. I set Joseph on a quest to provide real timings with -inline on and off. He wrote some scripts that allowed him to vary the read vs update load against a 200K document database with three views and collected stats using “show trans” at the Domino console. Joseph capped the document update and view operations at 10/second, but of course your numbers will scale according to volume, bulk of data and hardware utilized.

No one is claiming these numbers are real-world, the data and processing is automated. But they are consistent relative to the gathered set. And they show some trends people should know about as they consider using -inline. The variance in test load was simple: for each run, the number of all operations was incremented by one per second.

The OPEN_COLLECTION transaction is what bogs down trying to update a view when inline is off. The benefit is very clear to see in the first graph, and it should be noted that without inline, time to open a view increases much faster than linearly. That .. is the log jam I mentioned.

In the second graph, open view and read view entries are combined to provide a single duration. This is close to what a user would experience opening a view and getting his/her first window of entries rendered. Again, the spike with larger numbers of transactions without -inline is clear.

Now, very few SLAs will be violated with 800 millisecond response. Again, the point is the curve. Showing maximum time to open a view shows the erratic, worst-case user experience, which would indeed violate most SLAs (note, times are in seconds in this graph):

Truth in advertising — document update times do increase with -inline on, but you would expect them to. And they don’t increase exponentially but linearly. But don’t confuse this with a decrease in throughput — it is not affected, only time to complete:

This final graph treats all operations — update, finds, view opens and view entry reads equally and averages their combined duration. So, overall duration impact of -inline processing if you will:

But it’s important to remember the original use case — multiple users trying to open and read a view but first “refreshing” (that is, updating). With -inline on, the refresh operation is unnecessary since any view being read is updated during a NSFNoteUpdate operation. So those view open operations are much lighter weight.

To say it again, this behavior is “normal” for transaction-centric database systems, especially the relational variety where data lives in neat rectangles (ok, they have progressed a bit). But in Domino, a document-storing and -processing NoSQL engine that supports so many different indexes constructing index data so many different ways, a lazy update model is appropriate lots of the time. And the -inline feature addresses those cases where it is not appropriate.

So, if you have contentious views — and please, not all views are hotly contended for — I recommend you strongly consider turning on inline view updating, using

load updall <database name> -inline on -T <name of contentious view>

And reap good benefit.

I would beware of turning it on for all views in a database as the document update cost could then surge. You know your most contentious views; tackle those first.

Comment wrap
Further Reading
article-img
Digital Solutions  /  August 11, 2020
Low and Behold: Why Low Code Development Matters. Insights into a modern way to build applications
By: HCL Domino Team
Low code is a software development platform that gives non-coders and coders alike the chance to develop applications through visual interfaces instead of traditional hand-coded programming.   In plain English, it’s the power to create processes without the technical experience or hand coding background application development once needed.  (It’s like suddenly being able to sing, dance, and generally dominate like Beyoncé without having to undergo 25+ years of endless practice and constant performing.)    Comparing application development to pop stardom may not be the most common way to highlight the potential benefits of low code. But by not requiring a background in coding expertise, low code allows a wider audience to create applications, making for a faster and potentially more innovative environment that companies can benefit from. (It’s like turning lemonade into even better lemonade. Ok that’s the final reference to Queen B, promise). But we’re just getting started diving into the upside of low code. Develop Business Software at Incredible Speeds Low code development should be fast.    Unlike the laborious line-by-line hand coding process that is traditionally associated with software programming, low code can go from concept to reality in a flash.  Low code allows developers to execute apps up to 60-70% faster than handwritten code, bypassing the time-consuming steps of repeatedly writing, testing and debugging code until it works correctly.  Low-code development can give designers drag-and-drop features through a visual user interface, making it easy to build intuitively while cutting down on complicated obstacles that only expert coders can spot and fix.  By making the process more visual and less repetitive, low code brings a newfound velocity to a typically slow undertaking.   Increase Enterprise Productivity  When you increase the speed of app building, and you save designers and developers time, you open new space up for productivity throughout the entire enterprise. But it’s not only the efficiencies that companies benefit from.  Those closest to a problem can fix it without a drawn-out drama that drains weeks of time and energy from the IT department. Solving...
article-img
Digital Solutions  /  August 7, 2020
Domino Volt New Release: Your Questions Answered!
By: Martin Lechleider, Director, Product Management, HCL Domino Volt
Last week, we hosted our new Domino Volt July release webinar covering the latest features and enhancements. We had several demos on how to directly access your Domino data, workflow enhancements, service catalogs, PDF integrations and more. For IBM z and IBM i customers, we covered the latest integration with Z and I Emulator (ZIE) to turn green screen apps into REST endpoints that Domino Volt can use to build new workflows and apps.  We received a lot of great questions. You can find those questions — and the answers — below. Or catch all the excitement in this replay here:   Want more Domino Volt?  Try v1.0.1 in the updated Domino Volt sandbox. Register for a free sandbox account. For those who already have a sandbox account, new sample apps have been uploaded here.  We recently launched a Domino Volt roundtable series, where we introduce different important topics and host open discussions with our customers and partners. Join us for the following sessions: Domino Volt App Integration Strategies: Learn about integration techniques and options within Domino Volt. August 12, 2 pm ET. Register now.     Domino Volt Deployment Topologies: Learn about different deployment options to fit your needs. August 27, 2 pm ET. Register now. DOMINO VOLT FEATURES  Q: What additional resources are needed on a Domino server to handle Domino Volt?  A: Domino Volt requires Domino V11.0.1 or greater. That’s is all!  Q: If an app is built in Domino Volt, can it be modified in Domino Designer?  A: You could add additional views, agents, or other elements as long as you do not change or delete things that Domino Volt created.  Q: Where do you upload the PDF for fill-in?  A: In settings, there is a File section where you can load PDFs as well as images, CSS, JS, etc. as needed in the app  Q: Can a signature from a Domino Volt form be printed in PDF? Or even pictures from your smartphone?  A: The PDF fill capability in Domino Volt does not support adding an image or picture to a PDF.  Q: Can you deploy your...
article-img
Digital Solutions  /  July 29, 2020
Licensing Update: HCL Complete Collaboration (CCB) Guest Licensing
By: Uffe Sorensen, Global Director of DS Strategy, HCL Software
Since July 2019, it has been HCL Digital Solutions’ mission to help our customers and business partners by introducing contemporary licensing and making license management easier for all our products. Today, we are taking the next step towards consolidating all of our Domino licensing on a "per user" license model – the HCL Domino Complete Collaboration offering a.k.a “CCB.” Effectively immediately, customers with HCL Domino Servers deployed under CCB Authorized User entitlements can be accessed by a Licensee’s entitled CCB Authorized Users and Guest Users. A “Guest User” can be an anonymous web user (no authentication), or an authenticated web user with a predefined maximum level of Domino application access (ACL) as “Depositor.” In addition, HCL Domino servers under CCB may participate in mail routing (SMTP), directory lookup and authentication (LDAP) for non-HCL Domino programs and permit access to free/busy time calendar information. HCL made these changes to the CCB license based on customer requests to facilitate: • A read-only external or internal web site for Guest Users • Data collection using surveys generated by Domino Volt 1.0.1 and its anonymous user support available today. • Removing the need for a separate Utility Server license for Guest Users with complex PVU reporting requirements. • Clarifying licensing for mail and calendar interoperability in multi-vendor scenarios involving our partner’s solutions. In August, HCL plans to update the formal License Information found here. In the meantime, you may refer to this blog or you can request a formal Product Notice for compliance. If you have any questions about this announcement or have any licensing questions, please contact your HCL product specialist or Business Partner. Uffe Sorensen & the Domino Product Team Disclaimer – HCL’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at HCL’s sole discretion. Information regarding...
a/icon/common/search Created with Sketch.