Web and Desktop Integration for MaterialX

These are some on-going thoughts and investigation into integrating libraries, pipeline tools, and interactive editing within the context targeting both "Web" and "Desktop" usage.

Context

Note that these terms are used very loosely with the following suppositions:

For "Web":

• Asset sizes are restricted based on deployment device.
• The preferred format for assets can differ from desktop such as the ability to stream content.
• Dependent resource access is restricted (e.g by security) so data must be packaged.
• The target is client-side user facing interactions (not command line)
• The aim is not to support an online "pipeline"

For "Desktop":

• Local and remote storage is freely accessible
• Asset sizes can be arbitrary.
• There can be a mix of interactive and command based tooling often as part large tooling "pipeline".
• Data does not need to be packaged into a single unit. "Side-car" files are acceptable with arbitrary path references (though in general some predefined organization would be adhered to for a given pipeline)

An over-arching goal is to have lessless interoperability between Web and Desktop tools and libraries with the minimal number of external dependencies.

Implementation Considerations

For simplicity we assume a Javascript/Typescript implementation is more suitable for web and Python for desktop.

• These have been chosen due to avoid dependencies on (pre)compilation as well the ability to easily create public library packages (e.g. on npm or PyPi respectively).
• Iterating for prototyping is naturally easier using an interpretive language.

It is possible to have a single source of logic such as a C++ implementation and then encoding this into the desired target languages. There are however valid concerns about efficiency (size and speed) of deployment for web is often cited as a major concern or even a "blocking" issue. For example, using something like emscripten to produce WASM can produce slower and bloated runtime libraries if not optimized properly.

However, for functionality that is part of a "standard" it still feels that the correct approach for maintainability and consistency is to a single source of truth for implementation logic with deployment to target languages as needed.

Other valid considerations include:

• Avoiding duplicating the API's which are native to the target. For example, it does not make sense to duplicate file support for web deployment.
• Taking into consideration the inherent asynchronous behaviour for web deployments and logic which forces synchronization.

Documents

File Format

MaterialX can provide a light-weight and standardized format for a material / shader description within a "document" (file / string / io stream). File referencing of other documents is supported when using the native XML format. It also allows for explicit external resource references -- namely image files. No such reference system exists for geometry as the the material is decoupled from the geometry it will be bound to (unlike say OpenUSD).

As an alternative format, it has been put forth that JSON is a more suitable format to provide inherit aspects such as security and suitability for streaming. Though not formalized, bi-directional lossless interop for this has been written as a library (*) . Due to it's simplicity it can be written easily in Python or Javascript. If this were to become part of the standard functionality then migration to a single source of logic would be desired.

File Referencing

(*) Of note is that JSON inherently does not support referencing other JSON files. In general discussion around XML support for MaterialX, it reference certain type of content has been discouraged. (namely definition libraries). In that sense this constraint is consistent for usage for both formats. This also aligns with integration into OpenUSD where MaterialX document may be specified as a reference with composition management left to OpenUSD itself.

File Servers

Though it is possible to pre-package assets as part of a web deployment this is static and can be restricted by resource overhead.

For shared assets, server-side hosting support is thus desirable.
This generally takes the form of "material libraries" where
packaged assets can be retrieved on demand.

Example tooling for the AMD GPU Open and PhysicallyBased material library servers using Python and Javascript was examined and documented here. Below is data pulled from a server and displayed in a graph editor:

An corresponding interactive page for PhysicallyBased is available here.

Of note is that not all material libraries will provide Material content as MaterialX. In this case a remapping step is required, especially if the parameterization is assumed to map to a specific shading model.

See the section about zip files for more discussion on asset packaging.

Asset Packaging

Image URIs

Specifically for image resources references are actually denoted as "file name" references (URIs). There are no restrictions in the specification as to where resources should be placed and how they are referenced, though there are some recommendations in the supplemental documentation. Thus as is often seen for desktop DCCs both absolute and relative pathing can cause issues when the root document is referenced from or moved a different location.

There is a "resolving" mechanism which can be used for file paths in MaterialX which can allow for token replacement as well as absolute path resolving geared at desktop usage. In general this tooling is not well suited for deployment to Web.

For example a format such as glTF is not suited for consuming absolute image paths since these assets are not accessible / discoverable. Packaging image references allows for removal of this dependence if fully embedded. (Having a side-car data files (.bin) still entails a secondary reference).

As MaterialX supports "URIs", it is possible to make image references more robust by:

1. Referring to a server address (http),
2. Embedding data directly as a data/imagestring with the appropriate mimetype.

It is also possible to use blob storage allocated during a sessions or as web page local storage. This has it's pros and cons.

• One advantage is that users can load in images and bind them to a shader during an interactive session without pre-packaging them statically (e.g via webpack).
• These can also be loaded into web renderers that support this such as ThreeJS. For page display these are natively supported as <img> DOM references.
• One downside is that this is transient data. It is possible to converted back into image data and written out as a packaged asset when saving the content.

Below is an example of usage of each URI type with the top viewer using a ThreeJS renderer which takes as input the URI
and converts the data into a runtime texture. The property editor on the web pages uses <img> element's for display.

blob

http

data

Image Formats

Naturally not all image formats may be supported for web and desktop integrations. This is more a content creation and management concern but worth mentioning to avoid having content which may not show up across all desired integrations. For example, webp web files may not be supported by a desktop API or exr files may not be supported by a web API.

Packaging Approaches

Zip Files

MaterialX itself does not provide any "packaging" mechanism to handle resource dependencies. Some material library sites such as the AMD GPU Open material library package MaterialX documents and dependent images as zip files. It provides the ability to make server side requests to download zip files for individual materials with specific image sizes (*)

There are both desktop and web APIs which have streaming support to unpack such data. Usage on desktop is mostly trivial if file paths are relative and the package unzips to a relative path locations.

For Web, different consumption approaches are possible:

1. Unpack the images and directly store into the renderer's cache. This is done in the sample Web MaterialXViewer when zip files are dropped.
   Below is an example of a AMD GPU material zip dropped into the viewer.
   
2. Unpack the images and store as embedded data or into local cache and referenced as a data blob.

Either or both can be used depending on usage. For example the first can be more efficient for direct viewing. The latter can be more suitable for content inspection.
An example page for unzip preview can be found here. Below is snapshot of the page:

(*) Aside:

• It is noteworthy that different zip packages are available for the same material but with different image sizes. One or more can be chosen depending on the what is most suitable for deployment.
• Another approach would be to have a single material file with tokenized references which can be used to reference different server side images. This would make it more friendly for on-demand loading.

Translation and Interop (glTF, USD)

A natural alternative is to translate from MaterialX into another format. The most notable being glTF and usd formats, with possible packaged asset deployment being: glb and usdz.

Currently the tooling for this interop is targeted at desktop usage due to the complexities of packaging required data which extends beyond just materials.

The main disadvantage for editing workflows is that these are last mile formats which can be lossy and thus not well suited to lossless bi-directional data exchange. ( See this informative article about last mile formats. )

Additionally, there is the requirement to bring in additional library dependencies. For example, the default OpenUSD distribution is quite large for Web deployment and it's modules cannot be decoupled for a smaller footprint. There is also no official Web releases at time fo writing, though a independent compact library (tinyusdz is currently available which has desktop and web support.

Hybrid Integration

It is possible to take a hybrid approach especially for desktop UI tools.
One example is to use APIs which allow for web pages to be hosted such as Qt.

Tools and features can be partitioned by which target is best suited for deployment. As an example, a ThreeJS viewer can be used to render into a html page which managed by a QWebEnginePage, with local storage being used to pass assets back and forth, and Javascript code / libraries used to execute any tooling.

This is the approach currently taken to to exchange MaterialX converted to glTF to a web page viewer for QuiltiX: documented here

The partition of functionality is:

• A Python implementation that supports MaterialX and OpenUSD as well as any required and MaterialX to glTF distillation / baking process.
• A Javascript implementation for web visualization.

Below is a snapshot of the integration, where the web page is hosted
on the UI widget (panel) on the left.