In the context of this work, 4D volumetric video represents 4D mesh sequences M t s , 2D textures T t s and 3D skeletal motion p t s computed from multiple view video capture. A 4D volumetric video dataset consists of N _S sequences s = [1 … N _S ] and each sequence consists of N T s frames at a time instance t = [ 1 … N T s ] .

State-of-the-art volumetric performance capture of people with loose clothing and hair (Collet et al., 2015) results in high resolution reconstructed shape and texture appearance. Raw volumetric video typically results in an unstructured mesh sequence where both the mesh shape and connectivity changes from frame-to-frame (Prada et al., 2016). Several approaches have been introduced for temporal alignment over short subsequences to compress the storage requirements (Collet et al., 2015) or global alignment across complete sequences (Huang et al., 2011; Cagniart et al., 2010; Regateiro et al., 2018).

In this work we employ the skeleton-driven volumetric surface alignment framework (Regateiro et al., 2018) to pre-process captured 4D volumetric video of people to obtain a temporally coherent mesh structure across multiple sequences. This framework receives as input synchronised multiple view video from calibrated cameras and returns 3D skeletal joints and temporally consistent 3D meshes with the same mesh connectivity at every frame. The texture appearance is retrieved by re-mapping the original multiple view camera images onto the temporally consistent 3D meshes providing a dynamic texture map with consistent coordinates for all captured frames. The input to the deep network presented in the following sections consists of centred 4D temporally consistent mesh sequences with the corresponding 2D texture maps and 3D skeletal joint locations.

Variational networks have become a popular approach to learn a compact latent space representation which can integrate with deep neural networks. In this section, we employ a variational encoder-decoder to learn a compact latent space mapping between 3D skeletal pose and the corresponding 4D shape, illustrated in Figure 2. The generative network architecture maximises the probability distribution of the 3D skeletal joint positions p = { { p t s } t = 1 N T s } s = 1 N S , encoded in the latent space z = { { z t s } t = 1 N T s } s = 1 N S , and learns the generative mapping of the decoder to the corresponding 4D mesh M ̃ t s . While we define input p as 3D skeletal joint positions, it can be replaced with other pose representations consisting of 3D landmarks, e.g. facial keypoints.

Generative networks learn dependencies from the input data and capture them in a low-dimensional latent vector z t s , creating compact representations z t s ∈ R d , where d is the latent space dimension (128 dimensions throughout this work). The probability density function P(p) for the skeletal pose is given by: P ( p ) = ∫ P ( p | z ) P ( z ) d z (1)

The distribution P(p|z) denotes the maximum likelihood estimation of dependencies of p over the latent vector z, and P (z) is the prior probability distribution of a latent vector z. To ensure a compact representation P(p|z) is modelled as a Gaussian distribution with mean μ(z) and diagonal co-variance σ(z) multiplied by the identity I , which implicitly assumes independence between the dimensions of z. P ( p | z ) = N ( p | μ ( z ) , σ ( z ) 2 * I ) (2)

The Pose2Shape network architecture is composed of an encoder, which receives 3D skeletal joint positions as input, and a decoder, see supplementary material network details, that generates high resolution 3D meshes. The encoder is trained to map the posterior distribution of data samples p to the latent space z, meanwhile forcing the latent variables z to comply with the prior distribution of P(z). However, both the posterior distribution P(z|p) and P(p) are unknown. Therefore, variational networks give the solution that the posterior distribution is a variational distribution Q ( z t s | M ̃ t s ) . In order to make Q ( z t s | M ̃ t s ) consistent with the distribution P(z), we use the Kullback-Leiber (KL) divergence (Kingma and Welling, 2013): K L ( Q ( z t s | M ̃ t s ) ‖ P ( z t s ) ) (3)

The decoder is trained to regress from any latent vector z t s in the learnt space z to a 4D mesh representation M ̃ t s . Eq. (4) defines the loss function minimised by the network to achieve a compact latent space representation and generative network output. L = ( Q ( P ( p t s | z t s ) | M ̃ t s ) − M t s ) + ω K L ( Q ( z t s | M ̃ t s ) ‖ P ( z t s ) ) (4)

This is an optimal approximation of the true samples M t s , where ω weighs the importance of the KL divergence, and M t s is the ground truth 4D mesh for the 3D skeletal pose p t s of sequence s at time t.

In this section, we propose the use of a Pose2Appearance network for the synthesis of high-resolution dynamic mesh texture maps from the encoded skeletal pose latent space representation. A similar approach described as the progressive growing of GANs was first introduced by Karras et al. (2017) to improve image synthesis quality and training stability of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014).

A GAN consists of two networks, a generator and a discriminator. The generator produces images from a latent code, and the distribution of these images should be indistinguishable from the training distribution. The discriminator evaluates the quality of the images produced by the generator, forcing the generator to learn how to produce high-quality images so that the discriminator cannot tell the difference. A progressive generator generally consists of a network where the training begins with a low-resolution image and progressively increases the resolution until it reaches a target resolution. This incremental multi-resolution approach allows the training first to discover the large-scale structure of the distribution of the images and then shifts the attention to finer-scale details, whereas in traditional GAN architectures, all scales are learned simultaneously.

In this section, we adapt the generator from the progressive growing of GANs (Karras et al., 2017) to learn how to synthesise high-resolution texture appearance from the latent probability distribution learned from 3D skeletal motion, Section 3.2. The proposed Pose2Appearance for high-resolution texture map synthesis from the latent space vector is illustrated in Figure 3.

The Pose2Appearance initially starts with a small feed-forward network, see supplementary for details, which consists of four fully-connected layers, where the input consists of learnt latent vector z t s of dimension 128, which corresponds to the dimensions of the latent space learned from the Pose2Shape network, and the output dimension of the fourth layer is 512, to match the input size requirements of the first convolutional layer, as illustrated in Figure 3. The convolutional layers consist of nine blocks, where each block represents a different resolution, and its output is a high-resolution texture T ̃ t s .

We also experimented with a VAE network for appearance synthesis. This experiment was found to result in significant blur and loss of detail. The VAE assumes the same input and output, hence not being a suitable architecture for the problem. For this reason, a more sophisticated network approach is required, Section 5.3 for comparison with state-of-the-art methods.

The latent space of the learnt motion allows the pre-trained generators for shape Q ( z t s | M ̃ t s ) and texture A ( z t s | T ̃ t s ) to interpolate between the captured 4D volumetric video shape and appearance sequences. Because the variational encoder-decoder produces a compact latent space it is possible to generate novel content by sampling from the learned space or interpolation of sampled latent vectors. Sampling of the latent space allows reproduction of the original 4D volumetric video sequences with a low reconstruction error. The sampling can be performed in two ways: random walk in the latent space that fits in the Gaussian distribution learned; or through 3D joint positions given as input to the networks. In this work, sampling is performed through 3D joint position as input. Interpolation in the learnt latent space allows transitions between observed sequences to create plausible novel motions. Interpolating the latent space is only possible because of the compact space representation produced by the generative network. This is performed, firstly, by sampling latent vectors using 3D joint position as input to the network. Once the latent vector is computed for two motion frames then interpolation is performed according to Eq. 5. z i = ( 1 − α ) z t 1 s 1 + α z t 2 s 2 (5) where z _i is the interpolated latent vector, α defines a normalised weighting [0‥1] between latent vectors z t 1 s 1 and z t 2 s 2 . Intermediate 4D shape and texture frames are synthesised to qualitatively evaluate how well the network is representing the 4D shape and appearance, Section 5.2.

The framework receives as input, skeletal motion data from 4D volumetric video estimated using a Skeleton Driven Surface Registration (SDSR) framework (Regateiro et al., 2018) and latent vectors z t s of each motion sequence M t s learned in Section 3 for 4D shape and appearance learnt from a skeletal pose.

In the context of this section, a sequence of motion frames { { F t s } t = 1 N T s } a = 1 N S refers to collections of frames which contain representative latent vectors, and skeletal structures given by the SDSR framework as follows, F t s = { z t s , S t s } , where S t s is a skeletal structure from a motion sequence, which contains N T s number of frames t = [ 1 … N T s ] , representative of the original motion dataset. Lastly, it is necessary to utilise the pre-trained mesh generator Q ( z t s | M ̃ t s ) and the appearance generator A ( z t s | T ̃ t s ) from Section 3 to interpret each latent vectors z t s stored as a motion frame in latent motion sequence F t s .

The generative networks synthesise M ̃ t s meshes and T ̃ t s texture maps for every z t s ∈ F t s , which represents a temporally consistent 4D mesh and appearance, i.e. the topology, vertex connectivity and texture coordinates are constant across all frames and sequences. The construction of a motion graph is independent of the learnt model, allowing the framework to generalise its application to other types of models. A motion graph is interpreted as a directed weighted graph structure built from captured 4D volumetric video sequences, where graph nodes represent frames that contain latent vectors which hold information about shape, motion and appearance, and edges link nodes together to represent motion pathways between frames.

The data is required to be pre-processed; this offline process starts with training the generative networks described in Section 3 for a skeleton motion sequences of a human character. Once training is complete the generators Q ( z t s | M ̃ t s ) and A ( z t s | T ̃ t s ) are used to recover the 3D meshes and 2D textures represented by each latent vector z t s , to allow the pre-processing step to be automated. The first step in the pre-processing stage is to connect frames within the same sequences automatically, and if possible create loops for cyclic motions, consequently a sequence can infinitely repeat itself. Loops are generated via searching on a similarity matrix S I M T ( F t u i , F t v j ) for all pairs of frames in the same sequence to automatically choose the minimum cost, Section 4.3 and Section 4.4. Transitions within the same sequence should produce the most natural motion; hence the shape and motion cost should be small.

The next step is to fully connect the graph by adding all possible transition combinations between sequences to allow better path estimations to be found for all frames. This step will generate a fully connected graph with appropriated edge weights using shape, motion and dynamic time warping metrics, as detailed in the following sections. Lastly, the graph is optimised using Dijkstra’s algorithm to minimise the number of transition in the final motion graph, as detailed in Section 4.5.

Similarity is computed for every pair of frames in the input 4D volumetric video sequences S I M T ( F t u i , F t v j ) , where F t u i is a frame t _u from the ith sequence F t u i = { M t u i , T t u i } , comprising meshes M t u i and textures T t u i , where i = [1 … N _S ]. For a given latent vector z t s the decoder Q ( z t s | M ̃ t s ) reconstructs temporally consistent geometry, and the appearance generator A ( z t s | T ̃ t s ) reconstructs the 2D texture appearance of generated frame. The shape, motion and appearance similarity is computed for every pair of source F t u i and target F t v j frames, having t u ∈ [ 1 , N T s ] and t _v ∈ [1, N _T ] frames for all sequences i, j ∈ [1, N _S ]. S I M T ( F t u i , F t v j ) = θ S I M M ( M t u i , M t v j ) + ( 1 − θ ) S I M A ( T t u i , T t v j ) (6) Where θ weights the relative importance of shape and appearance similarity, giving a complete similarity matrix S I M T ( F t u i , F t v j ) for all frames generated by the learnt 4D volumetric video representation. To measure shape similarity we use the Euclidean distances and velocities between mesh vertices as illustrated in Eq. 7. S I M M ( M t u i , M t v j ) = 1 N V ( ‖ x t u i − x t v j ‖ + ‖ v t u i − v t v j ‖ ) (7) Where vertex velocity v t u i = ( x t u i − x t u − 1 i ) , and N _V is the number of vertices. The appearance similarity uses the average absolute difference of the 2D texture appearance between two frames as illustrated in Eq. 8. S I M A ( T t u i , T t v j ) = 1 N X ‖ T t u i − T t v j ‖ (8) Where N _X is the number of pixels. The similarities are normalised to the range (0,1) as follows: S I M Q ( F t u i , F t v j ) = S I M Q ( F t u i , F t v j ) − m i n ( S I M Q ( F t u i , F t v j ) ) m a x ( S I M Q ( F t u i , F t v j ) ) − m i n ( S I M Q ( F t u i , F t v j ) ) (9) Where SIMQ (⋅) is either SIM _M (⋅) or SIM _A (⋅) similarity metrics for shape and appearance. The pre-computed similarity matrix S I M T ( F t u i , F t v j ) for all frames allows to evaluate in real-time the similarity cost between any source and target meshes.

An edge in a motion graph represents a transition between two frames, where for clarity frames will be described as nodes. For every edge, we associate a weight to represent the similarity of shape transitions between nodes quantitatively. Realistic transitions should require little change in shape and appearance corresponding to a small similarity score. Hence the metric used takes into account the optimal surface interpolation cost between any pair of nodes (Boukhayma and Boyer, 2017). The cost of transitioning is the sum of intermediate poses between source node u and destination node v weighted by the similarity score for each intermediate frame.

In order to smoothly blend source node u from a 3D mesh sequence to destination node v from another sequence, it is necessary to consider a blend window of length b. This window represents a successive number of nodes b _u , on the source sequence it begins at node u and ends at node u + b _u − 1, in the destination sequence a window b _v ending at node v and starting at node v − b _v + 1. Once, the window frame is initialised between source and destination sequence, it is necessary to extrapolate the nodes that gradually blend both sequences, generating smooth realistic transitions. To extract the optimal nodes from source and destination sequences we use a variant of dynamic time warping (DTW) (Muller, 2007; Witkin and Popovic, 1995 Wang and Bodenheimer, 2008; Casas et al., 2013) to estimate the best temporal warps w _u and w _v respectively with respect to the similarity metric defined in Eq. (6). DTW was first introduced by Sakoe and Chiba (1990) for signal time alignment, it was used in conjunction with dynamic programming techniques for the recognition of isolated words and it had been widely used since then mainly for recognition tasks. The transition duration varies within a third of a second and 2 s (Wang and Bodenheimer, 2008), hence we allow the length b _u and b _v to vary between boundaries b _min and b _max . The optimal transitions with minimal total similarity cost D(u, v) through the path generated from the DTW algorithm. D ( u , v ) = min b u , b v , w u , w v , D l , D l ∑ t ∈ [ 0 , D l ] S I M T ( F t u i , F t v j ) (10) where S I M T ( F t u i , F t v j ) is the shape similarity cost defined in Section 4.3, and D _l is the length of the path found by the DTW algorithm considered as the transition duration, see supplementary material for illustration. The optimisation above finds the following optimal parameters (b _u , b _v , w _u , w _v , D _l , D _l ), which are considered later for motions synthesis. Similar to Section 4.3, we define the edge weight between nodes to be the surface deformation cost D (u, v) and its interpolated duration cost D _l (u, v). Eq. (11) summarises the definition for the edge cost between nodes u and v. D ′ ( u , v ) = min [ α ( v − u ) , D ( u , v ) + α D l ( u , v ) ] (11)

For the case nodes u and v are from the same sequence the surface deformation should be minimal. To control the tolerance between surface deformation and transition duration we add weight α.

This process will create a fully connected digraph where edges are weighted for the shape similarity and transition cost between nodes, in the following Section 4.5 we will discuss how to prune and optimise the connectivity of the complete digraph.

The last stage in the framework aims to find a globally optimal solution to minimise the number of transitions between nodes. Plausible transitions can be achieved by selecting the minimum cost transition from the similarity matrix between sequences, to generate a motion graph. A fully connected digraph was generated from Section 4.4, which connects every pair of nodes for all existing motion sequences. Therefore selecting the minimum cost transition for every node would maintain dense connectivity in the graph.

We have implemented a globally optimal strategy that extracts and maintains only the best paths between every pair of nodes (Huang et al., 2009) Casas et al., 2011; Casas et al., 2013; Boukhayma and Boyer, 2015; Boukhayma and Boyer, 2017). This strategy corresponds to extracting the essential sub-graph from the complete digraph induced from the input sequences (Bordino et al., 2008). This method ensures the existence of at least one transition between any two nodes in the graph, which potentially yields a better use of the original data with less dead ends. Given the fully connected digraph, we use the Dijkstra algorithm on every pair of nodes to extract the shortest paths between source and target nodes. Once this process is completed, we remove all edges that do not belong to the new generated paths, giving a connected digraph that contains only the necessary least cost transitions. The resulting structure is also referred to as the union of shortest-path trees rooted at every graph node. This solution will guarantee the minimal difference when transitioning from frames of different sequences.

This section demonstrates generation of 4D volumetric video using the Deep4D motion graphs. To generate a continuous stream of animation between motion sequences it is necessary to calculate the least costly transition path between a source frame F t u i and a target frame F t v j from different motion sequences. As discussed previously, the least costly transition should be a transition within the same motion sequence, consequently if the animation remains unchanged by the user the framework will play the same motion in a loop. If the user requests the character change to a new motion state, the animation framework computes the minimum transition cost D ( F t u i , F t v j ) from the current motion frame F t u i to the selected motion sequence F t v j , and returns the following parameters (b _u , b _v , w _u , w _v , D _l ), Section 4.4. These parameters allow interpolation of the intermediate frames between frame u and v with a transition length of D _l , creating a seamless transition in real-time between different motion sequences. The approach presented in Section 4.4 finds the corresponding pair of frames by computing the shortest path on the warps (w _u , w _v ). The following sub-sections discuss how to synthesise 4D volumetric video and how intermediate frames are generated using generative networks.

The variational encoder-decoder uses Eq. (4) as a metric to predict plausible shape reconstructions from skeletal pose. The Pose2Appearance network uses the mean squared error (MSE) as loss function between generated images and ground truth as a metric to predict plausible high resolution textures. The comparison was performed between the training data, to ensure minimum error when sampling the original sequences, and validation data to guarantee a plausible result when generating unseen mesh.

We compare generated 3D meshes with ground truth geometry acquired from multiple view stereo reconstruction (Starck and Hilton, 2007). 3D mesh evaluation is performed using Hausdorff distance defined as d _H (A, B) = max{ sup _a∈A d(a, B), sup _b∈B d(b, A)}, where d(a, B) and d(b, A) is the distance from a point a to a set B and from a point b to a set A, which has been shown to be a good measurement between 3D meshes. The comparison contains training and validation data for all sequences, Table 1. The appearance is evaluated using three metrics that are commonly used to assess image quality: mean squared distance (MSE); multi-scaled structural similarity (MS-SSIM); peak signal to noise ratio (PSNR), Table 1 for results.

We compare our network generated results to rendered images of the original textured model and synthesised 4D volumetric content, Figure 10 and supplementary material for more results. Our network is able to capture dynamic shape detail and high frequency appearance details such as wrinkles and hair movement, Figure 10. The network is also capable of interpolating the existing data to generate novel geometry and appearance within the learned space. To test the interpolation performance of the network, the mesh and appearance of two encoded frames were selected and intermediate frames synthesised. Figures 7,8 shows a more challenging example for two randomly selected frames with large differences in shape and appearance, note that the method is able to produce a natural transition between frames.

The proposed generative network maps 3D skeletal pose to 4D volumetric video sequences consisting of shape and appearance. To evaluate this capability we use existing public skeletal motion capture sequences (CMU Graphics Lab, 2001) to synthesise novel 4D animations. To drive the generative network, we use the 3D skeletal joint positions p t s to obtain the encoded latent vectors z t s , sampling from the learnt distribution P (p|z). Figure 9 shows three characters driven using a novel motion capture sequence This demonstrates the potential to generate novel plausible 4D shape and appearance sequences from MoCap input for similar motions.

In this section, we evaluate the performance of the progressive appearance generator against the state-of-the-art method proposed by Lombardi et al. (2018) for facial image synthesis. The variant network architecture was chosen to allow for appearance synthesis only, as we intend to evaluate the texture synthesis quality, see supplementary for network illustration. Therefore we have removed the mesh and view-point conditioning from the original network architecture. We trained this network on 2D textures from the Thomas (Boukhayma and Boyer, 2015) and Martin (Klaudiny and Hilton, 2012 datasets, where the training took approximately 10 days for 10⁴ training cycles, with a mini-batch size of 64. This network minimises the MSE error and the KL-divergence simultaneously, similar to the proposed approach.

Figure 11 illustrates qualitative evaluation for this experiment. We have chosen one random sample from the training dataset to evaluate the quality of the texture synthesis given a seen example. Figure 11A presents heat-map images to compare the synthesised result against the ground-truth for the proposed and Lombardi networks. It is visible that the proposed network outperforms the Lombardi et al., 2018 approach, this is more visible on the close-up Figure 11B, where the details on the t-shirt have been lost when using the Lombardi et al., 2018 network. The proposed network is capable of preserving the printed image on the t-shirt along with wrinkles present in the original image. The lack of detail and the presence of blurred results from state-of-the-art Lombardi et al., 2018 network has led to the network presented in Section 3.3. The proposed approach is a more sophisticated network, capable of preserving fine details and complex structures, and achieves faster training given limited computational hardware.

This section includes a comparison of the proposed Pose2Shape network against linear blend skinning (LBS) techniques demonstrating the benefits of using the proposed network. LBS is a widely used approach in real-time character animation for deforming a surface mesh according to an underlying bone structure, where every bone contains a transformation matrix that affects a group of vertices. This relation is given by a weighting attribute that weights the contribution of a bone transformation on a vertex. LBS is computationally efficient and commonly used in animation frameworks, allowing real-time character animation by manipulation of surface geometry using a low-dimensional skeletal structure. Although, it does not allow propagation of non-linear surface deformation, and it can cause artefacts on the mesh surface. To understand if the proposed Pose2Shape model is capable of learning non-linear attributes from the input data instead of only learning a linear mapping, we compare the results against LBS. For this comparison, we present two experiments; the first experiment evaluates the interpolation performance against LBS. The second experiment compares the synthesis of a mesh sequence against using LBS to animate the same motion sequence, please see supplementary material for second experiment. To compare the meshes, we use the Hausdorff distance metrics, discussed in Section 5.1.

Figure 12 illustrates the results for the first experiment using the Thomas (Boukhayma and Boyer, 2015) dataset. The top row represents the original sequence of walking motion, the source and target frames surround by green and orange boxes, respectively, represent the frames used for interpolation. The middle row shows the results of interpolating the latent vectors representative of the source and target frames. Latent vectors were generated by encoding the respective skeletons of the source and target frames. As a consequence, we can synthesise intermediate poses following Eq. (5). The bottom row shows the LBS results for source and target frames. LBS is achieved using the animation capabilities of the SDSR framework (Regateiro et al., 2018), which allows mesh manipulation through skeletal animation. Therefore, given the original skeletal motion frames, we map the source frame onto the target frame whilst generating the intermediate frames, as illustrated in the bottom row.

This experiments demonstrates the ability to generate a more accurate reconstruction of the original mesh compared to LBS. To support these figures, Table 1, shows quantitative evaluation for all the datasets between LBS and the proposed results.

Table 2 demonstrates the proposed approach is capable of compressing 4D volumetric video through a deep learnt representation. The latent space representation achieves up to two orders of magnitude reduction in the size of the captured 4D volumetric video depending on sequence length. The decoders have an approximate size 105 MB with the texture encoder size constant, 94 MB, due to the fixed texture image resolution and the mesh encoder dependent on the mesh resolution, 10–18 MB.

Presented results were generated using a desktop PC with an Intel Core i7-6700K CPU, 64 GB of RAM and an Nvidia Geforce GTX 1080 GPU. Our training time is approximately 4 days on a single GPU. The non-optimised animation framework performance achieves ≈10 frames per second (fps) at full resolution. The performance bottleneck is in the Pose2Appearance network from Section 3.3 as a result of the high number of convolutional layer and training parameters. The generative networks from Section 3.2 is capable of achieving ≈35 frames per second (fps). The characteristic of the Pose2Appearance network allows for multi-scale texture resolution improving rendering performance and memory usage, see supplementary material for illustration. The generator is capable of reconstructing multiple resolutions of the appearance, increasing rendering performance and decreasing memory usage, allowing the possibility to use on platforms with memory constraints.

Primary limitation is quality of the 4D volumetric video sequences for training. The synthesis will reproduce artefacts present in the input data such as shape error or appearance misalignment. Currently this is limited by the publicly available 4D video sequences but will improve as 4D volumetric video improves. The current implementation is not optimised for texture rendering due copy operations between CPU and GPU memory, with optimisation this could achieve >30 fps for shape and appearance synthesis. Motion capture data synthesis may create undesired artefacts on the appearance and shape if the skeletal motion is outside the space of observed 4D motions as this requires extrapolation in the latent space. Currently the network is only able to represent one character a time, an interesting extension for future work would be to encode multiple characters in a single space, or a single person wearing multiple types of clothing.