Integration of Photogrammetry, Reflectance Transformation Imaging (RTI), and Multiband Imaging (MBI) for Visualization, Documentation, and Analysis of Archaeological and Related Materials

Introduction

This paper describes a practical workflow that enables the integration of photogrammetry-based 3D modeling, Reflectance Transformation Imaging (RTI), and Multiband Imaging (MBI)  into a single representation that can, in turn, be rendered visually using open-source software. To illustrate the workflow, we apply it to a fragment of an Egyptian painted wood sarcophagus depicting the winged goddess Isis that is now in the Institute of Fine Arts (New York University) Study Collection (Figure 1A).¹ Our intent is to show that this workflow can contribute to the visualization, documentation, and analysis of a broad range of visual and material culture, though our focus in developing the workflow has been art historical and archaeological practice and research. The object used as an example here is from a museum collection, though its state of preservation, with broken edges and only partially extant original surfaces, is analogous to that of objects found in the field during archaeological fieldwork. The flexibility of the process we present is most readily seen in an animation that is embedded in the body of this article (Figure 1B and Figure 2) and that highlights particular aspects of the wooden sarcophagus fragment. Watching that animation will give readers a sense of what the workflow can accomplish. However, we believe that the steps we describe are suitable for use with a wide variety of material culture and artworks and that they are adaptable to many research agendas. One indication of the potential for application in varied contexts is that other projects are working to overlay various types of complementary imagery on photogrammetric models.² It is likely, therefore, that this is an area of practice that will develop quickly. The workflow described here contributes to this collective progress by integrating multiple forms of imagery on a single 3D model and by using Blender as the tool that brings all the imagery into a single environment for both detailed rendering as single images and animated display.⬈#p1

In the discussion that follows, attention is given to the specific software used in each step. This includes the commercial package Agisoft Metashape for photogrammetry processing; the freely-available software packages RTIBuilder and RTIViewer for RTI processing; the freely-available CHARISMA³ software tools developed as add-ins for nip2 (the open-source graphical interface of VIPS) for MBI processing; and the open-source animation suite Blender, which is used for final rendering and animation.⁴ Despite this listing of software packages, what follows is not intended as a detailed tutorial but rather a presentation of the major steps that allow the integration we achieve.⁵ A strength of this workflow is that it allows the rendering of the particular and idiosyncratic aspects of any given object. This means that the exact application of the process described will depend on the specific goals of the researchers undertaking the effort. The Egyptian painted wood sarcophagus fragment used as our test case here has a very varied surface. The chisel marks that show the preparation of the wood surface, a partially extant plaster layer intended to receive paint directly, and evidence of different application techniques, along with varied pigments, are all aspects of this object that are present in its current state of preservation. Many of these surface details are readily revealed by RTI and MBI, which is why we chose to combine the output of those particular techniques with a photogrammetric 3D model. As seen in the animation and figures, the combination of all three techniques supports very close consideration of this individual piece. We stress, however, that the workflow is flexible and likely to have application in many other circumstances because it can potentially integrate a photogrammetric model with all types of imagery that can be captured by analogous methods to those described below.⬈#p2

Materials and Methods

Photogrammetry, RTI, and MBI are well-established imaging techniques widely used by cultural heritage professionals.⁶ Previous work has recognized the desirability of integrating the results of photogrammetry and of RTI more closely;⁷ and there have been efforts to recover 3D models directly from RTI data, which is a different process from what we pursue here.⁸ While no photographic technique produces a perfect or undistorted image, which means that absolutely perfect alignment between any set of independently produced images is likely impossible, we believe the techniques used here complement each other and produce a useful representation of the object under investigation. Specifically, the combination we have achieved results in visually rich and informative, high-resolution renderings that emphasize physical shape, surface variability, and spectral properties. The combination of photogrammetry, RTI, and MBI facilitates detailed study and visualization of an artifact’s surface that highlights otherwise difficult-to-perceive features. It is also a feature of our work that the digital representation it produces allows these aspects of the rendered object to be communicated and investigated without requiring direct physical or potentially destructive examination. Accordingly, this workflow shows promise for applications where prolonged access to and handling of material is ill-advised, restricted, or impractical.⬈#p3

It is fundamental to this project that we are adapting existing software and techniques. The following list provides brief definitions of the three techniques to be combined, along with the key points that allow for the integration of each one with the others.⬈#p4

• 3D Modeling by Photogrammetry: Source images of a single object from many overlapping camera positions are compiled to build a 3D model that can be manipulated in a virtual environment. As a result of the photogrammetric process, the relationship of each camera position to all other camera positions is firmly established. For this study, photogrammetry data was processed following the suggestions put forward by Cultural Heritage Imaging (CHI).⁹
• Reflectance Transformation Imaging (RTI): Source images of a static object from a stationary camera, with changing light source positions, are compiled to build an interactive file that documents the object’s surface interaction with light and allows active relighting by the user.¹⁰ The work described in this article takes advantage of the fact that an image taken from the stationery camera can be used in the set of photogrammetric images. RTI data was captured and processed following the workflows published by CHI.¹¹
• Multiband Imaging (MBI): Source images of an object are taken using a digital single-lens reflex (DSLR) camera that has been modified to capture in the ultraviolet, visible, and infrared regions, along with an apochromatic lens, bandpass filters, and light of controlled frequencies to document luminescence and reflectance properties within broad bands of the electromagnetic spectrum. As with RTI, images taken with this camera can be integrated into the set of photogrammetric images. MBI data was processed following the stipulations published in the CHARISMA manual.¹²

Additionally, in this article “UV” is used in the context of “UV texture,” a term meaning a 2D image that is mapped onto a 3D mesh. Here, “U” and “V” refer to the axes of the plane. We distinguish this usage of UV from the same abbreviation meaning “ultraviolet.”⬈#p5

We also note that the development of the workflow described here has been collaborative. Two of the authors are conservators who have worked on archaeological and museum materials and share an interest in the application of digital technologies to the field of objects conservation.¹³ The other is a Roman archaeologist who has worked to integrate 3D modeling into his fieldwork and into the presentation of Greco-Roman material culture.¹⁴ The dialog between disciplines has led us to adopt the phrase “visualization, documentation, and analysis” to indicate our intent that the method we describe be relevant to many forms of work. The animation that we present as one potential output illustrates the utility of our workflow for enriched communication of multiple aspects of an object. That animation is inherently a visualization that can serve many roles, among which is the detailed documentation of the manufacture of the painted Egyptian panel. This in turn can support discussion of ancient Egyptian manufacturing techniques and craftsmanship. Such discussion is a form of analysis and interpretation. To the extent these terms represent stages, they overlap so we believe that digital tools can usefully elide the distinction between them. Communication of results, which is very much our goal, underlies all three terms.⬈#p6

Method of Integration

To introduce the workflow for integrating photogrammetry, RTI, and MBI, we start with an analogy: the photogrammetric 3D mesh serves as virtual scaffolding for the display of other types of imaging data, which we term “auxiliary imagery.” In slightly more technical language, the workflow can be succinctly described by emphasizing that individual source images for the 3D model’s photorealistic UV texture can be replaced during the texture build in Agisoft Metashape. Replacing source images with auxiliary imagery allows for the creation of a new UV texture that achieves repeatable visual alignment between that auxiliary imagery and the 3D mesh. In this instance, the auxiliary imagery that we employ originates from RTI and MBI, thereby allowing the output of those techniques to appear on the 3D model in the correct alignment.⬈#p7

This section gives a brief illustrated overview of the main steps that result in a robust alignment between the 3D models and the auxiliary imagery so that these other images can be displayed on the model. The basis for our alignment of RTI and MBI with 3D models relies on establishing common camera positions between photogrammetry and the auxiliary imagery. The description assumes that the following stages of work have been completed: (1) source images of the object from many overlapping camera positions have been captured and processed to build a 3D model; and (2) at least one of the camera positions used in building that model has also been used for collection of auxiliary imagery (here RTI and MBI).⬈#p8

Figure 3 shows two screenshots of a 3D model of the test object as it appears in Metashape. In both Figure 3A and 3B, most camera positions are shown in blue. Figure 3A highlights in pink the position of the camera that was subsequently used to capture RTI source images. Figure 3B highlights the position of the camera that was then used to capture MBI images. Since these highlighted camera positions are the aspects of the workflow that enable integration, we call them “connection shots.”¹⁵ Successful connection shots should have the same camera position as the auxiliary imagery and the same even illumination as the photogrammetry set. When processing the photo set in software such as Metashape, it is fundamental for future integration that connection shots, along with all the other photos from which the 3D model is built, are aligned with the resulting 3D model in Metashape. This is an intentional side effect of the photogrammetric process: it results in a set of photographs aligned with a model. Once a set of photographs is so aligned, any other images captured by the same camera in any of the same positions are then also automatically aligned with the model, by which we mean that the color information from any image can be projected onto the 3D model in a predictable and meaningful manner.⬈#p9