We know that there can be sticky challenges in sharing Domino data — the fidelity of Notes Rich Text has long been an issue when sharing data outside of the platformThe standard starting point is the Notes Rich Text field renderer/editor and Notes Rich Text Item storage format. They become hard constraints that any solution needs to fit within, a box outside of which we should not think. 

At HCL Labs, we take a more radical approach. When new members were recently brought on board, they were instructed to not just think outside the box but to set the box on fire! This may result in approaches that don’t suit everyone. But the mandate we were given for Project Rosetta, an internal proof of concept, was to find an innovative solution for fidelity of formatted content for use outside of Domino with only one musthave: the data must be stored in Domino, in some format. Jason Roy GaryDigital Solutions CTO, covered the outcome in the DNUG launch event and the recent OpenNTF webinar. You can catch the 10-minute presentation between 40:27-50:44 here or read on for more details.

Terms of Reference 

One of the early actions was to be very specific about terminology. For a Domino audience, the phrase “rich text” is inextricably coupled with the Notes editor and a specific storage format. As a result, we have made a conscious effort to refer to handling “formatted text”. I would strongly encourage anyone else discussing this to do the same, to avoid false assumptions. 

Two radical options remained on the table throughout: that Notes may need a different editor/renderer for this content and that existing content may require a one-off conversion, if that content wants to leverage the benefits. There were also two key expectations we acknowledged: This was not intended to replace Notes Rich Text and not all existing Notes Rich Text should be converted. If the content is only used in Notes Client or thirdparty solutions manage the problem already, the status quo is acceptable. 

Starting Point 

For our starting point, we asked this question: what are the standard editors beyond Domino, and what formats do they use? This seems a simple question, but what became apparent is that the Notes Rich Text editor is used for three specific purposes, each of which have specific paradigms and interoperable formats when considered independently of Domino. 

“Document processing” 

Occasionally in Domino I’ve seen Rich Text Editors and Items used for managing complex, strictly formatted content like policies and procedures. Sometimes the applications around that content has been designed to mimic complex document processing functionality like change tracking as well. Beyond Domino, this content is typically managed in a specific document processing tool, like Microsoft Word and even collaborative editors like HCL Connections Docs, Google Docs or Collabora.

After many years of resistance, the vendors have all moved to a standard format for interoperability, OOXML. The storage format includes some metadata not openly editable in the document processing editors, things like author, tags etc. But that metadata is fixed and limited in scope. The editors do not allow manipulation of the custom metadata. The editors also only permit editing one file at a time. 

Despite attempts to “store once, share everywhere,” complications of security and access often result in the content being copied around. The document is sent as a single discreet package with all images, embedded files etc included. This commonly results in a file that is too large for HTTP or even email sending, requiring the user to compress the file or copy to some secure file sharing solution, whether that be products like Connections or temporary transfer protocols like SFTP. 

This is a very specific paradigm, matching a Form with virtually no additional metadata and a single Rich Text Item. We acknowledged it as a valid use case, but a narrow one. It would be interesting to investigate whether we can store the content in a format that would allow in-place rendering but also allow editing in one of those external editors, e.g. Microsoft Word. But it was not appropriate to the current scope.

Email 

Email has always been a core part of Notes and Domino. At the time Domino was launched, email was not in wide usage. Indeed, MIME was not defined by IETF until 1992. However, MIME has become the de facto standard for interoperability outside of Domino. If you receive an email from outside Domino, it will be stored as MIME. Only emails from Domino domains will be stored as Notes Rich Text. 

Again, there is specific but limited metadata. For MIME emails, the Item type that addresses are stored in is not Text, it’s RFC822 Text. That’s a different data type internally within Domino, but one that the Notes Client is (presumably) programmed to interpret in a specific way. That is interesting and informative. 

The storage structure is also very particular for this use case. The email is only ever referenced from source by the sender. Everyone else receives their own copy of the email. There has never been an attempt to store a “single version of the truth” which all recipients reference. Consequently, what is circulated is a single package containing text, images and files – as with document processing. Size of associated content may be prohibitive, which is why many emails these days reference external images and may link to files on central services like Connections Files or Box. 

This is also a very specific paradigm, not matching the typical usage in applications. Would interoperability of email be easier if Domino stored this content as MIME instead of Notes Rich Text, and if the Notes Client had an alternative editor / renderer to use MIME rather than Notes Rich Text? That’s beyond the scope of our current investigation. But again, this type of content was left out-of-scope. 

Formatted Content in Fields in Forms 

The third scenario is formatted content in arbitrary fields in forms. The editors are not document processing tools, nor mail clients. There were various editors used, but the output fell into two categories – HTML and markdown. 

For editing markdown, there are two types of editors. The first type are editors in IDEs (VS Code) or standalone (Joplin) and these are designed to edit markdown files only. As with the document processing tools, these are out of scope. The second type are markdown editors that can be embedded into an application, like the editor on OpenNTF’s website or Stephan Wissel’s comment area on his blog. These more closely correlate to the kind of editor for a Domino application. They also map closely to editors that provide content as HTML, such as TinyMCE or framework-specific editors like the Vaadin rich text editor. 

There are some other significant differences to document processors and email editors. Firstly, the content is not intended to be copied around, only referenced. Secondly, images and file attachments are typically stored separately, which allows them to be cached by whatever client is displaying the content. Indeed, blog platforms like WordPress require these assets to be stored separately and the editors enforce this. Even in Domino, Declan Lynch’s Blogsphere template takes the approach of storing assets separately. Thirdly, there may be multiple formatted text editors on a form. And fourthly, the metadata adjacent to the formatted content – i.e. other fields – is random and rarely do two forms contain the same sets of metadata. 

This is the scenario Project Rosetta targeted. As can be seen from this analysis, there are key differences to how Notes Rich Text editors and Items function. But the use case we targeted was exclusively for content that is intended to be shared beyond Domino. 

Architecture Choices 

Our proof of concept had a simple goal: investigate feasibility of allowing content to be managed as HTML or markdown, ideally converted between the two, and stored in Domino. 

Typically, content is entered in one or the other. Markdown is a nice flexible approach for quick editing with basic semantic formatting, and it opens up some interesting opportunities for consumption. There are also some options available in markdown that are not easily available in HTML like note blocks. But I’m conscious that expecting users from Marketing or HR, for example, to enter content as markdown is not realistic. Converting between markdown and HTML and vice versa would solve this problem and is analogous to the low-code / pro-code round-tripping approach that has been discussed for Domino’s lowcode vision. 

We built the API layer on top of Project Keep. The challenge we had was, if we were to send this content alongside other fields, how do we determine the data type that should be stored? In Keep and in Domino HTTP, attachments are already uploaded and retrieved as separate REST calls to accessing field data. So, we took the same approach here. The flow, at this point, would be to create the document with one call, then upload attachments or images as a separate call, then uploading or retrieving formatted content with Content-Type as “text/html” or “text/markdown.” Could that change? Of course. 

In terms of technology, Keep is Java and in Java the standard library for HTML manipulation is JSoup, the standard library for markdown conversion is flexmark. 

The benefit of keeping this manipulation between the two output types on the server is that clients just need to speak HTML or markdown. Different flavours of markdown may prove a challenge in the future, but that’s for the future. For app developers, you bring your own editor, we provide consistent conversion. Handling it on the server also allows a phase for cleaning the HTML. I am particularly keen to ensure we retain this as it opens up a number of potential innovative opportunities which I’m not ready to discuss at this time. 

Admittedly, it may be trying to solve a problem no one has. But R&D is about thinking about a problem differently and coming up with a solution not suggested before. It’s about being Icarus, daring to fly higher but willing to fall. 

I think we achieved what we set out to do, with a demo that went from mobile, to web, to an idea of what Nomad could handle with appropriate editors on top of the raw HTML / markdown, back to the web and back to mobile. It’s certainly not complete and work on Notes Client / Nomad would be required. And, as I said, it doesn’t cover all scenarios. But it demonstrates getting formatted content into and out of Domino without the peculiarities of Notes Rich Text, using something more universal. That opens up a new way for migrating formatted content from non-Domino databases into an NSF, so appealing to non-traditional audiences. And as Jason showed at the end of the session, thinking beyond Notes Rich Text for formatted content is a requirement for non-traditional clients and transfer formats, like EWS. 

Comment wrap
Further Reading
Digital Solutions | January 20, 2021
Start Your Engines! HCL Domino v12 Beta is Here! Are You Ready?
If you joined us at Digital Week 2020, you’ve probably heard about the preview of Domino v12. (It was the most popular session of the event!) Now that the release of v12 is around the corner, we would like to provide all current customers an exclusive preview. Today, we’re officially launching the first public beta of Domino v12!   We are inviting you to join this beta program and take part in shaping the future of our product, helping us deliver the best-ever product experience! All existing Domino customers are automatically entitled and will be able to download the required software packages from Flexnet today. (See below for Q&A.)  The goal of the Domino v12 Beta Program is for our community of beta participants to conduct an honest, constructive, and thoughtful review and testing of the Domino v12 beta software, which includes HCL Domino V12 , HCL Notes V12 , HCL Domino Designer & Admin V12 and HCL Traveler V12    In the first phase of the beta program, we are delivering the following components:    Domino on Docker (English)   Domino for Windows, Linux, AIX (English)   Traveler for Windows, Linux, AIX (Multilingual)   Notes Standard for Windows (English)   Notes Standard for Mac (English) Designer and Admin Client (English)     At a later stage of the beta program, we will be providing Domino and Traveler for IBMi, as well as additional language support.  Beta participants, please let us know how you think about the product by submitting your feedback in our beta forum. For general input and new ideas or feature enhancement requests, please use the Domino ideas forum here.   v12 Beta 1 Highlights  Domino  Reduce your total cost of ownership by using DAOS tier 2 objects across servers   A wide range of security improvements including: - Two Factor Authentication (TOTP) - Automated SSL/TLS certificate management with built-in support for Let’s Encrypt - Implementing IP based blocking of failed authentication attempts based on this idea - And many more Automate your Domino server installation and initial configuration Domino directory enhancements: A variety of improvements around the Domino directory design (pubnames.ntf) to improve usability for administrators   Domino Designer  XPages now Bootstrap 4.4.1 - Thanks to Howard for this idea.    Improvements to build responsive and...
Digital Solutions | January 12, 2021
HCL Domino Volt: Zero to Hero in 30 Days
You’ve all heard the HCL Domino Volt tagline, “Build enterprise apps lightning fast.” But what do we really mean when we say “fast”? We recently hosted a webinar to show you what you can build in 30 days or less. You can catch the replay here or read the recap below. You’ll also find responses to our live Q&A during the webinar below.   If you don’t already know, HCL Domino Volt is a low-code capability for business users and citizen developers to easily build powerful, secure, and enterprise-grade workflow-based applications. From business-process apps to customer-facing mobile apps, you can create solutions for any industry, across different use cases. What Can You Do in 30 Days?   So, let’s imagine that you have a solution or app in mind but have little to no knowledge of HCL Domino Volt. How do you go about using it to build an app? Assuming you have a day job and can only squeeze in 1-2 hours a day, this is what a timeline could look like for you: Day 1-3 (3 days): Learn how to use HCL Domino Volt. Sign up for a free sandbox account and get access to a list of training resources.   Day 4-9 (5 days): Pick a use case and define the requirements. What are you trying to solve and build? Does your app need workflows and approval processes? How will those be mapped out? Your requirements might also require refinements and iterations along the way.   Day 10-17 (7 days): Build the app! Refer to our documentation and wiki pages along the way use our forum to participate in our community and ask questions.   Day 18-25 (7 days): Share, test, and refine the app. Building your app is an iterative process. This is where you’ll build the app, share and gather feedback, and work towards a finalized version.     Day 26-29 (3 days): Style the app. You can either use the current themes provided or add custom themes that align with your organization’s branding. Or use CSS or HTML to...
Digital Solutions | December 8, 2020
HCL Domino Volt: The December Release is Here!
While the holiday season is upon us, we’re not slowing down! Including the launch earlier this April, we’ve had three releases of HCL Domino Volt — with many more to come in 2021. Starting today, customers can access v1.0.2 on the HCL License Portal and updated documentation here.   To learn more about building apps with HCL Domino Volt and what new features are in the latest release, please join our webinar next week, “Zero to Hero in 30 days,” where we’ll demonstrate apps built across different use cases, industries, and skill levels. You will learn what you can realistically deploy in 30 days.   Highlights of this new release include:  Improved user experience Standardize the look of your apps More ways to use HCL Domino to get things done   Improved usability: With a new properties panel and no modal dialogs, we’ve eliminated extra steps and hidden settings. These changes have made the tool more intuitive and requires fewer clicks to get things done.   Standardize the look of your apps: Drive brand consistency and standardize the look of your apps​. You can now add custom themes that align with your aesthetics and your organization’s branding. Users will be able select the themes you add when designing their apps.  Leverage Domino to get things done: The following features provide more ways for you to get things done by leveraging Domino’s capabilities:  ‘Contains’ search operator (uses full text indexing): Quickly find information you need by searching for data by text string. It works with data view and services.   Sort by app-specific fields ​(uses DQL): Easily analyze data you’ve collected by sorting your fields in the data view.  Name-picker item on the palette (uses Domino directory): Easily find and select people and groups in the Domino directory and assign them to roles or to receive notifications. Workflows and other improvements  Map internet email to role and notifications: You now have the flexibility to use any internal email address when assigning roles and notifications, not just Notes addresses.  Email attachments:  Now you can include files that have been attached during the form fill process in email notifications.  Application history: You...
a/icon/common/search Created with Sketch.