Virtual reconstruction of historical textiles using graph neural networks

Archaeological artifacts are often found in a fragmentary state and archaeologists need to identify matching fragments and reassemble a puzzle to reconstruct the original and analyze its motifs. The process resembles jigsaw puzzle solving; puzzle solving is challenging for archaeological textiles, which are especially prone to fragmentation, degradation, fading and deformations [1]. A vivid example of such an open puzzle problem, is the Oseberg Tapestry from the Viking Age [6]. Previous works [2,3] used classical texture classifiers, as well as pre-trained neural networks to extract features from fragments and feed the feature vectors to clustering algorithms to identify similar fragments, but the performance was not deemed satisfactory (see [1] for full overview). In this work, we take an alternative approach and use Graph Neural Networks (GNN) for puzzle solving process. The eventual intended application is the case of the Oseberg Tapestry, which tells interesting stories from the Viking Age, but has many highly fragmented and missing pieces with irregular shapes [6]. However, due to its complexity and lack of complete reference, our study uses well-preserved, but stylistically similar textile. Using artificially fragmented textile allows us to evaluate the accuracy of puzzle solving, which is harder for the Oseberg Tapestry, due to lack of the ground truth. By artificially fragmenting the textile, we simulate an unrestricted pictorial puzzle [4], a type of puzzle where pieces are arbitrarily shaped, forming a complex planar adjacency graph that demands sophisticated reconstruction techniques. As mentioned in Harel and Ben-Shahar [4], there have been a few solutions for this type of puzzle, due to its complex nature. In recent years, one notable work comparable to our problem is a work on fragment assembly using graph neural network-based (GNN) reconstruction of papyrus fragments [5]. By virtually segmenting the textile imagery data (RGB and depth maps) into non-adjacent, different sizes of square patches, we construct a dataset needed for our GNN reconstruction model. The core of our reconstruction process involves extracting features from these tapestry patches using a pre-trained neural network. The extracted feature vectors form the nodes of fully connected graphs, encapsulating the relationships between tapestry segments. Using graph neural network’s pooling techniques, we can dynamically exchange information between nodes and update node embeddings containing visual and geometric data. Lastly, using the updated graph, we predict the edge connections information used to reconstruct the fragmented tapestry accurately. This methodology aims to predict the puzzle’s assembly, laying a foundation for future work in the virtual reconstruction of fragmented historical textiles.