We know that there can be sticky challenges in sharing Domino data — the fidelity of Notes Rich Text has long been an issue when sharing data outside of the platformThe standard starting point is the Notes Rich Text field renderer/editor and Notes Rich Text Item storage format. They become hard constraints that any solution needs to fit within, a box outside of which we should not think. 

At HCL Labs, we take a more radical approach. When new members were recently brought on board, they were instructed to not just think outside the box but to set the box on fire! This may result in approaches that don’t suit everyone. But the mandate we were given for Project Rosetta, an internal proof of concept, was to find an innovative solution for fidelity of formatted content for use outside of Domino with only one musthave: the data must be stored in Domino, in some format. Jason Roy GaryDigital Solutions CTO, covered the outcome in the DNUG launch event and the recent OpenNTF webinar. You can catch the 10-minute presentation between 40:27-50:44 here or read on for more details.

Terms of Reference 

One of the early actions was to be very specific about terminology. For a Domino audience, the phrase “rich text” is inextricably coupled with the Notes editor and a specific storage format. As a result, we have made a conscious effort to refer to handling “formatted text”. I would strongly encourage anyone else discussing this to do the same, to avoid false assumptions. 

Two radical options remained on the table throughout: that Notes may need a different editor/renderer for this content and that existing content may require a one-off conversion, if that content wants to leverage the benefits. There were also two key expectations we acknowledged: This was not intended to replace Notes Rich Text and not all existing Notes Rich Text should be converted. If the content is only used in Notes Client or thirdparty solutions manage the problem already, the status quo is acceptable. 

Starting Point 

For our starting point, we asked this question: what are the standard editors beyond Domino, and what formats do they use? This seems a simple question, but what became apparent is that the Notes Rich Text editor is used for three specific purposes, each of which have specific paradigms and interoperable formats when considered independently of Domino. 

“Document processing” 

Occasionally in Domino I’ve seen Rich Text Editors and Items used for managing complex, strictly formatted content like policies and procedures. Sometimes the applications around that content has been designed to mimic complex document processing functionality like change tracking as well. Beyond Domino, this content is typically managed in a specific document processing tool, like Microsoft Word and even collaborative editors like HCL Connections Docs, Google Docs or Collabora.

After many years of resistance, the vendors have all moved to a standard format for interoperability, OOXML. The storage format includes some metadata not openly editable in the document processing editors, things like author, tags etc. But that metadata is fixed and limited in scope. The editors do not allow manipulation of the custom metadata. The editors also only permit editing one file at a time. 

Despite attempts to “store once, share everywhere,” complications of security and access often result in the content being copied around. The document is sent as a single discreet package with all images, embedded files etc included. This commonly results in a file that is too large for HTTP or even email sending, requiring the user to compress the file or copy to some secure file sharing solution, whether that be products like Connections or temporary transfer protocols like SFTP. 

This is a very specific paradigm, matching a Form with virtually no additional metadata and a single Rich Text Item. We acknowledged it as a valid use case, but a narrow one. It would be interesting to investigate whether we can store the content in a format that would allow in-place rendering but also allow editing in one of those external editors, e.g. Microsoft Word. But it was not appropriate to the current scope.

Email 

Email has always been a core part of Notes and Domino. At the time Domino was launched, email was not in wide usage. Indeed, MIME was not defined by IETF until 1992. However, MIME has become the de facto standard for interoperability outside of Domino. If you receive an email from outside Domino, it will be stored as MIME. Only emails from Domino domains will be stored as Notes Rich Text. 

Again, there is specific but limited metadata. For MIME emails, the Item type that addresses are stored in is not Text, it’s RFC822 Text. That’s a different data type internally within Domino, but one that the Notes Client is (presumably) programmed to interpret in a specific way. That is interesting and informative. 

The storage structure is also very particular for this use case. The email is only ever referenced from source by the sender. Everyone else receives their own copy of the email. There has never been an attempt to store a “single version of the truth” which all recipients reference. Consequently, what is circulated is a single package containing text, images and files – as with document processing. Size of associated content may be prohibitive, which is why many emails these days reference external images and may link to files on central services like Connections Files or Box. 

This is also a very specific paradigm, not matching the typical usage in applications. Would interoperability of email be easier if Domino stored this content as MIME instead of Notes Rich Text, and if the Notes Client had an alternative editor / renderer to use MIME rather than Notes Rich Text? That’s beyond the scope of our current investigation. But again, this type of content was left out-of-scope. 

Formatted Content in Fields in Forms 

The third scenario is formatted content in arbitrary fields in forms. The editors are not document processing tools, nor mail clients. There were various editors used, but the output fell into two categories – HTML and markdown. 

For editing markdown, there are two types of editors. The first type are editors in IDEs (VS Code) or standalone (Joplin) and these are designed to edit markdown files only. As with the document processing tools, these are out of scope. The second type are markdown editors that can be embedded into an application, like the editor on OpenNTF’s website or Stephan Wissel’s comment area on his blog. These more closely correlate to the kind of editor for a Domino application. They also map closely to editors that provide content as HTML, such as TinyMCE or framework-specific editors like the Vaadin rich text editor. 

There are some other significant differences to document processors and email editors. Firstly, the content is not intended to be copied around, only referenced. Secondly, images and file attachments are typically stored separately, which allows them to be cached by whatever client is displaying the content. Indeed, blog platforms like WordPress require these assets to be stored separately and the editors enforce this. Even in Domino, Declan Lynch’s Blogsphere template takes the approach of storing assets separately. Thirdly, there may be multiple formatted text editors on a form. And fourthly, the metadata adjacent to the formatted content – i.e. other fields – is random and rarely do two forms contain the same sets of metadata. 

This is the scenario Project Rosetta targeted. As can be seen from this analysis, there are key differences to how Notes Rich Text editors and Items function. But the use case we targeted was exclusively for content that is intended to be shared beyond Domino. 

Architecture Choices 

Our proof of concept had a simple goal: investigate feasibility of allowing content to be managed as HTML or markdown, ideally converted between the two, and stored in Domino. 

Typically, content is entered in one or the other. Markdown is a nice flexible approach for quick editing with basic semantic formatting, and it opens up some interesting opportunities for consumption. There are also some options available in markdown that are not easily available in HTML like note blocks. But I’m conscious that expecting users from Marketing or HR, for example, to enter content as markdown is not realistic. Converting between markdown and HTML and vice versa would solve this problem and is analogous to the low-code / pro-code round-tripping approach that has been discussed for Domino’s lowcode vision. 

We built the API layer on top of Project Keep. The challenge we had was, if we were to send this content alongside other fields, how do we determine the data type that should be stored? In Keep and in Domino HTTP, attachments are already uploaded and retrieved as separate REST calls to accessing field data. So, we took the same approach here. The flow, at this point, would be to create the document with one call, then upload attachments or images as a separate call, then uploading or retrieving formatted content with Content-Type as “text/html” or “text/markdown.” Could that change? Of course. 

In terms of technology, Keep is Java and in Java the standard library for HTML manipulation is JSoup, the standard library for markdown conversion is flexmark. 

The benefit of keeping this manipulation between the two output types on the server is that clients just need to speak HTML or markdown. Different flavours of markdown may prove a challenge in the future, but that’s for the future. For app developers, you bring your own editor, we provide consistent conversion. Handling it on the server also allows a phase for cleaning the HTML. I am particularly keen to ensure we retain this as it opens up a number of potential innovative opportunities which I’m not ready to discuss at this time. 

Admittedly, it may be trying to solve a problem no one has. But R&D is about thinking about a problem differently and coming up with a solution not suggested before. It’s about being Icarus, daring to fly higher but willing to fall. 

I think we achieved what we set out to do, with a demo that went from mobile, to web, to an idea of what Nomad could handle with appropriate editors on top of the raw HTML / markdown, back to the web and back to mobile. It’s certainly not complete and work on Notes Client / Nomad would be required. And, as I said, it doesn’t cover all scenarios. But it demonstrates getting formatted content into and out of Domino without the peculiarities of Notes Rich Text, using something more universal. That opens up a new way for migrating formatted content from non-Domino databases into an NSF, so appealing to non-traditional audiences. And as Jason showed at the end of the session, thinking beyond Notes Rich Text for formatted content is a requirement for non-traditional clients and transfer formats, like EWS. 

Comment wrap
Further Reading
article-img
Digital Solutions | August 23, 2021
Introducing Domino v12.0.1 and HTMO 3.0.3 Beta Program!
After the successful launch of HCL Domino v12.0 in June 2021, we are approaching the next milestone: We’re happy to announce two new beta programs at once, HCL Domino 12.0.1 and HTMO 3.0.3 have both launched a Beta Program today! You are invited to join the beta program to test our latest release and provide feedback to development teams.   We’re calling on our community to share open, effective, and thoughtful review and testing of our software product. Your input is essential to how we develop our products, and we will continue to listen to you every step along our roadmap.  What you can expect from this first beta of Domino v12.0.1: HCL Notes Client Domino Online Meeting Integration The Notes Client 12.0.1 is providing native integration of online meeting platforms like Zoom, Webex, GoToMeeting, Teams and of course Sametime Meetings. So, from within the Notes calendar users can now more easily schedule online meetings on their preferred platform without having to install a plug-in. Improved Workspace Design with High Resolution Icons The new Notes client is finally allowing to use high resolution icons for your databases, which makes a big difference to the look and feel of your workspace! No need to stick to small and old style 32x32 pixel database icons anymore. We have already updated our standard templates with new icons, will you update yours? Advanced Properties Box You have asked for it, we have delivered: an all-new and resizable document property box: Thanks to Panagenda, the new "advanced properties" box allows to search for field names or values, it also provides the requested ability to copy to clipboard or export to a CSV file: What's new in Notes v12.0.1? https://help.hcltechsw.com/notes/beta/12.0.1/whats_new_01.html HCL Domino Designer We have already provided a 64Bit Notes client (in beta as of now), and based on your request are now providing a 64Bit Domino Designer client. Also, Domino Designer v12.0.1 provides the ability to include high quality icons for rendering and enables developers to optimize the database full...
article-img
Digital Solutions | August 18, 2021
New Tech Updates for HCL Domino, Notes, Traveler, Verse, and Connections
While you may have been on a summer vacation, we at HCL have been busy improving our products! Here is a small overview of what happened in the last couple of weeks and what you can expect soon from your most favorite collaboration platform.  Group 3 Language Translations for Notes and Domino 11.0.1  Good news for many of our customers: HCL Notes 11.0.1 is now available in more languages.   You asked for it (DOMINO-I-831 and NTS-I-842) - In addition to the 16 languages the Notes client was already providing, we are delivering the following language translations:  Danish  Finnish  Norwegian Catalan Hebrew Hungarian Slovenian Thai Turkish  Install kits for the HCL Notes Standard and Basic Client V11.0.1 in those languages can be found at our License and Download Portal under the Notes/Domino version 11.0.1.   Of course, we are also providing the Multilingual User Interface (MUI) kit for those languages, as well as the Install Shield Tuner files to customize your installation.  Group 3 language translations for the Notes Client v12 are being worked on. Level up for our Domino Rest API  In June, we started the beta program for our new Domino Rest API. Now in August, it’s time for the next iteration. Shifting gears to Beta 2 will introduce several improvements customers were asking for:  ODATA support was added for reading/writing from/to Domino, which is especially of interest for e.g., Salesforce customers who want to store data in Domino A new Domino server add-in task is provided to start/stop/reload the Domino Rest APIs and show its status from the Domino server console An installer is now allowing to install the Domino Rest APIs on a Domino Linux server. Previously only a docker container was provided  For more details, about the new features please refer to the changelog of this project.  Please note: The Domino Rest APIs are still in beta — the first eGA release is scheduled to be shipping with Domino v12.0.1 later this year. Customers interested in joining the beta program can find the required files available for download at our License and Download Portal.  HCL Traveler 12.0 updated with Fix Pack 1  Fix Pack 1 for...
article-img
Digital Solutions | June 11, 2021
Announcing the Domino REST API Beta Program  
Today, we’re announcing the start of the beta program for our new Domino REST API, formerly known as “Project Keep.” This is the latest addition to our ongoing Domino Early Access Program. The Domino REST API will introduce hundreds of new APIs that aperture to information on Domino — further extending access to your Domino data. It’s a modern take on the REST API for opening Domino access to a world of standards — and API-first developers. What is it?   Domino REST API is a new feature that runs alongside the server and allows you to expose your Domino data in the form of standardized Open-API-based methods securely and easily. Using a browser-based admin UI, application owners can define which data will be made available for view or update on a REST API. It extends the Domino principles of reader/author document access definitions into the world of Internet protocols.  It also includes Swagger UI which allows the visualization and interaction with APIs without having any of the actual implementation logic in place. The APIs are automatically generated from an OpenAPI (formerly Swagger) specification with visual documentation, making it easier to later implement the back-end code.  What is special about the new Domino REST APIs?  Secure by default, with fine granular controls per form, field and user basis  Implements latest open standards  HTTP/2-ready, for server-to-server or client-to-server communication API-first design with full interactive documentation  Low barrier to entry as it runs on a Domino server and/or your Notes client  Admin UI and Postman samples included  State-of-the-art JWT access token integrated with your existing IdP infrastructure What can be accessed via the Domino REST APIs? You can access content like views, documents, and fields, as well as database design, agents, and ACL settings. And, of course, featuring DQL queries to quickly access the data you are looking for.  Built-in declarative security ensures the API will only allow access to fields the caller is authorized to see or update. This can effectively prevent computed fields to be overwritten and limits participants in a workflow to update their fields only.   How to participate   The Domino REST API is now available as a...
Close