We know that there can be sticky challenges in sharing Domino data — the fidelity of Notes Rich Text has long been an issue when sharing data outside of the platformThe standard starting point is the Notes Rich Text field renderer/editor and Notes Rich Text Item storage format. They become hard constraints that any solution needs to fit within, a box outside of which we should not think. 

At HCL Labs, we take a more radical approach. When new members were recently brought on board, they were instructed to not just think outside the box but to set the box on fire! This may result in approaches that don’t suit everyone. But the mandate we were given for Project Rosetta, an internal proof of concept, was to find an innovative solution for fidelity of formatted content for use outside of Domino with only one musthave: the data must be stored in Domino, in some format. Jason Roy GaryDigital Solutions CTO, covered the outcome in the DNUG launch event and the recent OpenNTF webinar. You can catch the 10-minute presentation between 40:27-50:44 here or read on for more details.

Terms of Reference 

One of the early actions was to be very specific about terminology. For a Domino audience, the phrase “rich text” is inextricably coupled with the Notes editor and a specific storage format. As a result, we have made a conscious effort to refer to handling “formatted text”. I would strongly encourage anyone else discussing this to do the same, to avoid false assumptions. 

Two radical options remained on the table throughout: that Notes may need a different editor/renderer for this content and that existing content may require a one-off conversion, if that content wants to leverage the benefits. There were also two key expectations we acknowledged: This was not intended to replace Notes Rich Text and not all existing Notes Rich Text should be converted. If the content is only used in Notes Client or thirdparty solutions manage the problem already, the status quo is acceptable. 

Starting Point 

For our starting point, we asked this question: what are the standard editors beyond Domino, and what formats do they use? This seems a simple question, but what became apparent is that the Notes Rich Text editor is used for three specific purposes, each of which have specific paradigms and interoperable formats when considered independently of Domino. 

“Document processing” 

Occasionally in Domino I’ve seen Rich Text Editors and Items used for managing complex, strictly formatted content like policies and procedures. Sometimes the applications around that content has been designed to mimic complex document processing functionality like change tracking as well. Beyond Domino, this content is typically managed in a specific document processing tool, like Microsoft Word and even collaborative editors like HCL Connections Docs, Google Docs or Collabora.

After many years of resistance, the vendors have all moved to a standard format for interoperability, OOXML. The storage format includes some metadata not openly editable in the document processing editors, things like author, tags etc. But that metadata is fixed and limited in scope. The editors do not allow manipulation of the custom metadata. The editors also only permit editing one file at a time. 

Despite attempts to “store once, share everywhere,” complications of security and access often result in the content being copied around. The document is sent as a single discreet package with all images, embedded files etc included. This commonly results in a file that is too large for HTTP or even email sending, requiring the user to compress the file or copy to some secure file sharing solution, whether that be products like Connections or temporary transfer protocols like SFTP. 

This is a very specific paradigm, matching a Form with virtually no additional metadata and a single Rich Text Item. We acknowledged it as a valid use case, but a narrow one. It would be interesting to investigate whether we can store the content in a format that would allow in-place rendering but also allow editing in one of those external editors, e.g. Microsoft Word. But it was not appropriate to the current scope.

Email 

Email has always been a core part of Notes and Domino. At the time Domino was launched, email was not in wide usage. Indeed, MIME was not defined by IETF until 1992. However, MIME has become the de facto standard for interoperability outside of Domino. If you receive an email from outside Domino, it will be stored as MIME. Only emails from Domino domains will be stored as Notes Rich Text. 

Again, there is specific but limited metadata. For MIME emails, the Item type that addresses are stored in is not Text, it’s RFC822 Text. That’s a different data type internally within Domino, but one that the Notes Client is (presumably) programmed to interpret in a specific way. That is interesting and informative. 

The storage structure is also very particular for this use case. The email is only ever referenced from source by the sender. Everyone else receives their own copy of the email. There has never been an attempt to store a “single version of the truth” which all recipients reference. Consequently, what is circulated is a single package containing text, images and files – as with document processing. Size of associated content may be prohibitive, which is why many emails these days reference external images and may link to files on central services like Connections Files or Box. 

This is also a very specific paradigm, not matching the typical usage in applications. Would interoperability of email be easier if Domino stored this content as MIME instead of Notes Rich Text, and if the Notes Client had an alternative editor / renderer to use MIME rather than Notes Rich Text? That’s beyond the scope of our current investigation. But again, this type of content was left out-of-scope. 

Formatted Content in Fields in Forms 

The third scenario is formatted content in arbitrary fields in forms. The editors are not document processing tools, nor mail clients. There were various editors used, but the output fell into two categories – HTML and markdown. 

For editing markdown, there are two types of editors. The first type are editors in IDEs (VS Code) or standalone (Joplin) and these are designed to edit markdown files only. As with the document processing tools, these are out of scope. The second type are markdown editors that can be embedded into an application, like the editor on OpenNTF’s website or Stephan Wissel’s comment area on his blog. These more closely correlate to the kind of editor for a Domino application. They also map closely to editors that provide content as HTML, such as TinyMCE or framework-specific editors like the Vaadin rich text editor. 

There are some other significant differences to document processors and email editors. Firstly, the content is not intended to be copied around, only referenced. Secondly, images and file attachments are typically stored separately, which allows them to be cached by whatever client is displaying the content. Indeed, blog platforms like WordPress require these assets to be stored separately and the editors enforce this. Even in Domino, Declan Lynch’s Blogsphere template takes the approach of storing assets separately. Thirdly, there may be multiple formatted text editors on a form. And fourthly, the metadata adjacent to the formatted content – i.e. other fields – is random and rarely do two forms contain the same sets of metadata. 

This is the scenario Project Rosetta targeted. As can be seen from this analysis, there are key differences to how Notes Rich Text editors and Items function. But the use case we targeted was exclusively for content that is intended to be shared beyond Domino. 

Architecture Choices 

Our proof of concept had a simple goal: investigate feasibility of allowing content to be managed as HTML or markdown, ideally converted between the two, and stored in Domino. 

Typically, content is entered in one or the other. Markdown is a nice flexible approach for quick editing with basic semantic formatting, and it opens up some interesting opportunities for consumption. There are also some options available in markdown that are not easily available in HTML like note blocks. But I’m conscious that expecting users from Marketing or HR, for example, to enter content as markdown is not realistic. Converting between markdown and HTML and vice versa would solve this problem and is analogous to the low-code / pro-code round-tripping approach that has been discussed for Domino’s lowcode vision. 

We built the API layer on top of Project Keep. The challenge we had was, if we were to send this content alongside other fields, how do we determine the data type that should be stored? In Keep and in Domino HTTP, attachments are already uploaded and retrieved as separate REST calls to accessing field data. So, we took the same approach here. The flow, at this point, would be to create the document with one call, then upload attachments or images as a separate call, then uploading or retrieving formatted content with Content-Type as “text/html” or “text/markdown.” Could that change? Of course. 

In terms of technology, Keep is Java and in Java the standard library for HTML manipulation is JSoup, the standard library for markdown conversion is flexmark. 

The benefit of keeping this manipulation between the two output types on the server is that clients just need to speak HTML or markdown. Different flavours of markdown may prove a challenge in the future, but that’s for the future. For app developers, you bring your own editor, we provide consistent conversion. Handling it on the server also allows a phase for cleaning the HTML. I am particularly keen to ensure we retain this as it opens up a number of potential innovative opportunities which I’m not ready to discuss at this time. 

Admittedly, it may be trying to solve a problem no one has. But R&D is about thinking about a problem differently and coming up with a solution not suggested before. It’s about being Icarus, daring to fly higher but willing to fall. 

I think we achieved what we set out to do, with a demo that went from mobile, to web, to an idea of what Nomad could handle with appropriate editors on top of the raw HTML / markdown, back to the web and back to mobile. It’s certainly not complete and work on Notes Client / Nomad would be required. And, as I said, it doesn’t cover all scenarios. But it demonstrates getting formatted content into and out of Domino without the peculiarities of Notes Rich Text, using something more universal. That opens up a new way for migrating formatted content from non-Domino databases into an NSF, so appealing to non-traditional audiences. And as Jason showed at the end of the session, thinking beyond Notes Rich Text for formatted content is a requirement for non-traditional clients and transfer formats, like EWS. 

Comment wrap
Further Reading
article-img
Digital Solutions | April 15, 2021
The Countdown to the Newest HCL Domino and Sametime Begins! Join Us for Our Global Launch.
I am thrilled to announce that registration is now live for what will surely be a true, blockbuster event on June 7. This exclusive virtual event celebrates our latest versions of HCL Domino v12 and HCL Sametime. The first 1200 registrants got a free gift (we blew past that number in a couple of days!). I personally love the amazing video that helps capture the excitement as we could down the days to these “breakaway” releases. (An agenda will be released soon. Follow us on Twitter to get the latest.)  The launch event will include thought-leading sessions, demos, customer speakers, and more. And, it kicks off an exciting collection of Domino content delivered in a series we are calling the “Domino Dozen” (once you’ve registered for the launch, you will have access to this program). Each day, for 12 days following the launch, we will drop a new, essential Domino-related treasure. These pieces will include webinars, panel discussions, and technical deep dives, early access to special programs, thought leadership pieces, and more.   Also, join us for the closing keynote and happy hour on June 23 — we are calling it “Nerdi Gras” as a hat tip to the days of Lotus past. We will be releasing a party kit soon, with downloadable favors, so you can join in the fun. For those of us who’ve been on this Domino journey with us for so many years — and for those of you who we will be welcoming as new customers — we can’t wait to celebrate. Join us.
article-img
Digital Solutions | April 2, 2021
HCL Domino v12 Beta 3 Has Arrived
What a blast! In our webinar hosted this week, we announced Beta 3 is now available for download on Flexnet for all Domino customers. Watch the replay here.  This will be the last beta drop before the v12 release so don’t miss your chance to participate and help us shape the future of Domino! In this third iteration, we are unleashing another set of great features. See the full list here or read highlights below. We’ve also included a Q&A received during our webinar and a special shoutout to all our customers and partners who covered our beta in blogs and tweets below! Notes Client Highlights New 64bit Notes Basic Client For the first time ever, we are releasing a 64-bit Notes Basic client for beta testing. Many of our customers have been voting for this and we’re pleased to deliver. (See this idea from Cormac.) Our main intent for this release is to engage customers, business partners, and third-party vendors to try out 64-bit Windows Basic version and test existing extension integrations. Our plan is to provide several iterations of this 64-bit client in beta and once we feel that we have addressed most of the known and reported issues, we will release the Standard 64-bit client. More Improvements for the Workspace In Beta 2, we introduced a new Notes client user interface for the workspace. Thanks to all your feedback, we have addressed several improvement requests such as reducing excessive spacing between icons and adding the ability to change the font color of workspace database icon (which is also policy controlled). Now you can also partially or completely collapse the workspace navigator and easily identify multiple replicas with the added a visual indicator. Read more here:  Updates to the new Notes V12 workspace design  Collapsible workspace sidebar  Apple M1 Support Good news for customers planning to use Apple’s new M1 hardware with HCL Notes. Beta 3 now also supports those type of CPUs. Domino Server Password Synchronization A new secure method to synchronize password changes from Active Directory to Domino allows customers to reduce the number of passwords that users have to remember. Password changes from Active...
article-img
Digital Solutions | March 16, 2021
HCL Domino v12: The 4 New Security Features You’ve Been Waiting for
While no platform is immune to the possibility of hacking, the question I would pose is: Has your Domino infrastructure ever been hacked?  Didn’t think so. It’s probably boring to say that the most straight forward answer is HCL Domino is rock solid on security.   When set up correctly and optimised, HCL Domino is the most secure platform of its type.  It’s true though.  Reliable and secure is a good thing. A very good thing.  The HCL Domino v12 beta is out now.  If you haven't already tried it, it's free for all existing licensed Domino customers.  It's waiting there in flexnet for you to download and try it out!  It's the first time a beta of this type is in existence and it has multiple interactions (we're currently on beta 2; beta 3 is scheduled for the end of March. Register here to join us for the beta 3 webinar. What I really love about it is the almost instantaneous feedback from the beta forum, from those in charge of development.  Domino v12 is scheduled for full release in Q2 of this year.  (June 2021 timeframe is given at the moment). Read an overview of what’s coming here. Here's is a list of all the NEW NATIVE security features coming in Domino v12 and there's a whole host of them: Automating certificate management  Time-based one-time password (TOTP) authentication  Enforce internet password lockout based on IP address  TLS 1.0 is disabled by default  Support for PEM-formatted TLS host keys and certificates  Two new curves supported for TLS 1.2 ciphers that use ECDHE for forward secrecy New template signing ID uses 2048-bit keys NRPC port encryption supports forward secrecy using X25519 Import internet certificates that contain unsupported critical extensions Suppress key rollover alerts during ID vault synchronization New Query Vault command options Support for SameSite cookie  Also note native support for DKIM is planned in the 12.0.x timeline. (Again natively, you can achieve DKIM with third party mail gateways). We could argue about which are the best and more important ones here, but I'm going to concentrate on the 4 new security features in Domino v12 that...
Close