We know that there can be sticky challenges in sharing Domino data — the fidelity of Notes Rich Text has long been an issue when sharing data outside of the platformThe standard starting point is the Notes Rich Text field renderer/editor and Notes Rich Text Item storage format. They become hard constraints that any solution needs to fit within, a box outside of which we should not think. 

At HCL Labs, we take a more radical approach. When new members were recently brought on board, they were instructed to not just think outside the box but to set the box on fire! This may result in approaches that don’t suit everyone. But the mandate we were given for Project Rosetta, an internal proof of concept, was to find an innovative solution for fidelity of formatted content for use outside of Domino with only one musthave: the data must be stored in Domino, in some format. Jason Roy GaryDigital Solutions CTO, covered the outcome in the DNUG launch event and the recent OpenNTF webinar. You can catch the 10-minute presentation between 40:27-50:44 here or read on for more details.

Terms of Reference 

One of the early actions was to be very specific about terminology. For a Domino audience, the phrase “rich text” is inextricably coupled with the Notes editor and a specific storage format. As a result, we have made a conscious effort to refer to handling “formatted text”. I would strongly encourage anyone else discussing this to do the same, to avoid false assumptions. 

Two radical options remained on the table throughout: that Notes may need a different editor/renderer for this content and that existing content may require a one-off conversion, if that content wants to leverage the benefits. There were also two key expectations we acknowledged: This was not intended to replace Notes Rich Text and not all existing Notes Rich Text should be converted. If the content is only used in Notes Client or thirdparty solutions manage the problem already, the status quo is acceptable. 

Starting Point 

For our starting point, we asked this question: what are the standard editors beyond Domino, and what formats do they use? This seems a simple question, but what became apparent is that the Notes Rich Text editor is used for three specific purposes, each of which have specific paradigms and interoperable formats when considered independently of Domino. 

“Document processing” 

Occasionally in Domino I’ve seen Rich Text Editors and Items used for managing complex, strictly formatted content like policies and procedures. Sometimes the applications around that content has been designed to mimic complex document processing functionality like change tracking as well. Beyond Domino, this content is typically managed in a specific document processing tool, like Microsoft Word and even collaborative editors like HCL Connections Docs, Google Docs or Collabora.

After many years of resistance, the vendors have all moved to a standard format for interoperability, OOXML. The storage format includes some metadata not openly editable in the document processing editors, things like author, tags etc. But that metadata is fixed and limited in scope. The editors do not allow manipulation of the custom metadata. The editors also only permit editing one file at a time. 

Despite attempts to “store once, share everywhere,” complications of security and access often result in the content being copied around. The document is sent as a single discreet package with all images, embedded files etc included. This commonly results in a file that is too large for HTTP or even email sending, requiring the user to compress the file or copy to some secure file sharing solution, whether that be products like Connections or temporary transfer protocols like SFTP. 

This is a very specific paradigm, matching a Form with virtually no additional metadata and a single Rich Text Item. We acknowledged it as a valid use case, but a narrow one. It would be interesting to investigate whether we can store the content in a format that would allow in-place rendering but also allow editing in one of those external editors, e.g. Microsoft Word. But it was not appropriate to the current scope.

Email 

Email has always been a core part of Notes and Domino. At the time Domino was launched, email was not in wide usage. Indeed, MIME was not defined by IETF until 1992. However, MIME has become the de facto standard for interoperability outside of Domino. If you receive an email from outside Domino, it will be stored as MIME. Only emails from Domino domains will be stored as Notes Rich Text. 

Again, there is specific but limited metadata. For MIME emails, the Item type that addresses are stored in is not Text, it’s RFC822 Text. That’s a different data type internally within Domino, but one that the Notes Client is (presumably) programmed to interpret in a specific way. That is interesting and informative. 

The storage structure is also very particular for this use case. The email is only ever referenced from source by the sender. Everyone else receives their own copy of the email. There has never been an attempt to store a “single version of the truth” which all recipients reference. Consequently, what is circulated is a single package containing text, images and files – as with document processing. Size of associated content may be prohibitive, which is why many emails these days reference external images and may link to files on central services like Connections Files or Box. 

This is also a very specific paradigm, not matching the typical usage in applications. Would interoperability of email be easier if Domino stored this content as MIME instead of Notes Rich Text, and if the Notes Client had an alternative editor / renderer to use MIME rather than Notes Rich Text? That’s beyond the scope of our current investigation. But again, this type of content was left out-of-scope. 

Formatted Content in Fields in Forms 

The third scenario is formatted content in arbitrary fields in forms. The editors are not document processing tools, nor mail clients. There were various editors used, but the output fell into two categories – HTML and markdown. 

For editing markdown, there are two types of editors. The first type are editors in IDEs (VS Code) or standalone (Joplin) and these are designed to edit markdown files only. As with the document processing tools, these are out of scope. The second type are markdown editors that can be embedded into an application, like the editor on OpenNTF’s website or Stephan Wissel’s comment area on his blog. These more closely correlate to the kind of editor for a Domino application. They also map closely to editors that provide content as HTML, such as TinyMCE or framework-specific editors like the Vaadin rich text editor. 

There are some other significant differences to document processors and email editors. Firstly, the content is not intended to be copied around, only referenced. Secondly, images and file attachments are typically stored separately, which allows them to be cached by whatever client is displaying the content. Indeed, blog platforms like WordPress require these assets to be stored separately and the editors enforce this. Even in Domino, Declan Lynch’s Blogsphere template takes the approach of storing assets separately. Thirdly, there may be multiple formatted text editors on a form. And fourthly, the metadata adjacent to the formatted content – i.e. other fields – is random and rarely do two forms contain the same sets of metadata. 

This is the scenario Project Rosetta targeted. As can be seen from this analysis, there are key differences to how Notes Rich Text editors and Items function. But the use case we targeted was exclusively for content that is intended to be shared beyond Domino. 

Architecture Choices 

Our proof of concept had a simple goal: investigate feasibility of allowing content to be managed as HTML or markdown, ideally converted between the two, and stored in Domino. 

Typically, content is entered in one or the other. Markdown is a nice flexible approach for quick editing with basic semantic formatting, and it opens up some interesting opportunities for consumption. There are also some options available in markdown that are not easily available in HTML like note blocks. But I’m conscious that expecting users from Marketing or HR, for example, to enter content as markdown is not realistic. Converting between markdown and HTML and vice versa would solve this problem and is analogous to the low-code / pro-code round-tripping approach that has been discussed for Domino’s lowcode vision. 

We built the API layer on top of Project Keep. The challenge we had was, if we were to send this content alongside other fields, how do we determine the data type that should be stored? In Keep and in Domino HTTP, attachments are already uploaded and retrieved as separate REST calls to accessing field data. So, we took the same approach here. The flow, at this point, would be to create the document with one call, then upload attachments or images as a separate call, then uploading or retrieving formatted content with Content-Type as “text/html” or “text/markdown.” Could that change? Of course. 

In terms of technology, Keep is Java and in Java the standard library for HTML manipulation is JSoup, the standard library for markdown conversion is flexmark. 

The benefit of keeping this manipulation between the two output types on the server is that clients just need to speak HTML or markdown. Different flavours of markdown may prove a challenge in the future, but that’s for the future. For app developers, you bring your own editor, we provide consistent conversion. Handling it on the server also allows a phase for cleaning the HTML. I am particularly keen to ensure we retain this as it opens up a number of potential innovative opportunities which I’m not ready to discuss at this time. 

Admittedly, it may be trying to solve a problem no one has. But R&D is about thinking about a problem differently and coming up with a solution not suggested before. It’s about being Icarus, daring to fly higher but willing to fall. 

I think we achieved what we set out to do, with a demo that went from mobile, to web, to an idea of what Nomad could handle with appropriate editors on top of the raw HTML / markdown, back to the web and back to mobile. It’s certainly not complete and work on Notes Client / Nomad would be required. And, as I said, it doesn’t cover all scenarios. But it demonstrates getting formatted content into and out of Domino without the peculiarities of Notes Rich Text, using something more universal. That opens up a new way for migrating formatted content from non-Domino databases into an NSF, so appealing to non-traditional audiences. And as Jason showed at the end of the session, thinking beyond Notes Rich Text for formatted content is a requirement for non-traditional clients and transfer formats, like EWS. 

Comment wrap
Further Reading
Digital Solutions | September 15, 2020
Project Eleven: Get the Most Out of Domino with Easy Direct Upgrades
HCL is fully committed to enabling you and your organization to maximize the value you get from the Domino and Sametime platforms today and in the future.   In the latest release of Domino v11.0.1, we’ve delivered innovative new capabilities such as low-code app dev and integration with Microsoft Active Directory. We’ve also launched a brand new Sametime meetings solution over the last two years. The response from customers and business partners has been unanimously positive!    Today, we’re excited to announce, “Project Eleven,” a new program designed to help every Domino and Sametime customer running on releases prior to v11 have a smooth upgrade experience. As part of the HCL Digital Solutions Academy, Project Eleven’s goal is to enable you to:   Go straight to Domino v11.0.1: HCL now supports direct upgrades from any of Domino version prior to v11. Whether you’re on v8, v9, or older, you can go straight to v11.0.1 with no additional steps necessary. Experience the latest version of Sametime risk-free: Secure meetings and data privacy are the most important features in today’s environment. Sametime meetings runs on Cloud Native technologies allowing you to auto scale as needed. Experience our brand new, highly secure, scalable video meetings and enterprise chat risk-free. Augment your skills today: Attend our online training sessions and get best-in-class support and training from a technical advocate in your local language. We’re rolling out more than 40 webinars and workshops and new articles, cookbooks, and help guides. Speak to a technical expert: Schedule an appointment with one of our highly skilled technical advisors to understand your options and create an upgrade plan that’s right for you.    Check out the new Project Eleven webpage to find all the resources you need to make your upgrade worry-free.                                             
Digital Solutions | September 8, 2020
Introducing HCL Domino Early Access Program
We’re proud to announce the HCL Domino Early Access Program, a new program that gives our customers and business partners a chance to test new product features early in the development cycle. This program provides a preview of features and functionality that may be shipped with a future version of HCL Domino and gives us the ability to incorporate your valuable feedback. At this stage, we are providing early access to features that contribute to HCL Domino v12.  The Early Access Program aims to get feedback on specific features, with a more formal traditional beta program planned at a later stage to receive feedback on the overall release quality and language packaging.  The Early Access Program is feature-driven and iterative, meaning new iterations and code drops will be made available as soon as it is declared ready for testing by our development team. New code drops can be provided in just a few weeks after the previous one, on short notice, without sticking to a fixed release cycle.   Based on your feedback we may proceed, improve or adjust the feature, or in some cases withdraw the feature from the code stream. The following graphic illustrates this concept: Each code drop is provided in the form of a Docker image which enables us to ship new iterations faster and to minimize the time to upgrade from one code drop to the next. You can expect us to deliver code drops at a fast pace as we want participants to have exclusive access to our latest product feature set. Please note that only the newest code drop will be available for you to download. How to Participate HCL Ambassadors who have registered Domino v11.0.1 Beta Preview will automatically be added to Domino Early Access Program. Customers and partners with a current Notes/Domino entitlement will also have access to the Early Access Program as of today. There is no need to register separately to participate. For overall...
Digital Solutions | September 3, 2020
New Release: HCL SafeLinx 1.1, – Secure VPN Services for Cloud or On-Premise
At HCL, we understand that securing your network is a topmost priority. HCL SafeLinx provides remote users with secure web-based and client-based access to enterprise applications and today, we’re excited to announce that the latest release of HCL Safelinx 1.1. For Domino customers using HCL Nomad, SafeLinx v1.1 enables you to securely access your Domino apps on a phone, tablet, or desktop browser without the need of an additional VPN. Introduction of HCL SafeLinx SafeLinx uses standards-based protocols to enable secure access from mobile computing devices outside the firewall to business applications and data on your organization's internal network. SafeLinx supports an extensive list of standard Internet Protocol (IP) and non-IP wireless bearer networks, server hardware and operating systems, and mobile security protocols. Remote users can connect to your internal network from a range of desktop, notebook, and handheld mobile devices and operating systems. Top Product Features Client and clientless access Network choices: Supports a worldwide selection of wireless and wireline network technologies, enabling a worldwide scope of connectivity. Cellular, LAN, satellite, dial-up, radio (IP, non-IP) Authentication: Multiple strong authentication options (Radius, Secure ID, LDAP, Active Directory, Certificate) Encryption DES, Triple DES, RC5, AES and FIPS 140-2-certified Network Optimization and Scalable and Reliable Customer Benefits Supports all major platforms: Windows, Linux, MacOS, iOS, and Android Best enterprise VPN with reverse proxy capabilities Mobility clients and Distributed Administration - Gatekeeper Dedicated mobile app for Android and iOS Two-way secure authentication Supports seamless, non-disruptive cross-network roaming Quick and easy deployment of servers and agents What's New in HCL SafeLinx 1.1? This new release includes improved functionalities of VPN and reverse proxy and support for TLS 1.3. View documentation on SafeLinx 1.1 here. For all Notes Domino Complete Collaboration cCustomers, SafeLinx v1.1 is now available as a free entitlement and will be listed...
a/icon/common/search Created with Sketch.