We are inspired by recent methods that have revisited geometric features, such as the bisector surface for scene–object indexing (Zhao et al., 2014) and affordance detection (Ruiz and Mayol-Cuevas, 2020). Initiating from a spatial representation makes sense if it helps reduce data training needs and simplify explanations—as long as it can outperform data-intensive approaches. Our affordance descriptor expands on the Interaction Bisector Surface (IBS) (Zhao et al., 2014), an approximation of the well-known Bisector Surface (BS) (Peternell, 2000). Given two surfaces S 1 , S 2 ∈ R 3 , the BS is the set of sphere centers that touch both surfaces at one point each. Due to its stability and geometrical characteristics, the IBS has been used in context retrieval, interaction classification, and functionality analysis (Zhao et al., 2014; Hu et al., 2015; Hu et al., 2016; Zhao et al., 2016; Zhao et al., 2017; Ruiz and Mayol-Cuevas, 2020). Our approach expands on these ideas and is geometrically intuitive and straightforward. It explicitly captures areas that are important to be in scene-contact and those that are not. Importantly, we show how this approach can be generalized from just one or a small number of samples to a large unseen number of scenes.

Our one-shot training process represents interactions by 3-tuples (M _h , M _e , and p _train ), where M _h is a posed human-body mesh, M _e is an environment mesh, and p _train is the reference point on M _e that supports the interaction. Let P _h and P _e be the sets of samples on M _h and M _e , respectively, their IBS I is defined as I = p ∣ min p h ′ ∈ P h ‖ p − p ′ ‖ = min p e ′ ∈ P e ‖ p − p ′ ‖ (1)

We use the Voronoi diagram D generated with P _h and P _e to produce I . By construction, every ridge in D is equidistant to the couple of points that defined it. Then, I is composed of ridges in D generated because of points from both P _h and P _e . An IBS can reach infinity, but we limit I by clipping it with the bounding sphere of M _h with tolerance ibs _rf .

The number and distribution of samples in P _h and P _e are crucial for a well-constructed discrete IBS. A low rate of sampled points degenerates on an IBS that pierces the boundaries of M _h or M _e . A higher density is critical in those zones where the proximity is high. To populate P _h and P _e , we first use a Poisson-disc sampling strategy (Yuksel, 2015) to generate ibs _ini evenly distributed samples on each mesh surface. Then, we perform a counter-part sampling that increases the number of samples in P _e by including the closest points on M _e to elements in P _h , and similarly, we incorporate in P _h the closest point on M _h to samples in P _e . We perform the counter-part sampling strategy ibs _cs times to generate a new I . However, we observed that for intricate human–scene poses, convergence to an IBS without mesh piercing is challenging. If the IBS is penetrating the scene, we perform a collision-point sampling strategy. This adds as sampling points, a sub-sample of points where collisions happen and their counter-part points (body or environment). We then simply recompute the IBS and repeat the counter-part sampling and collision-point sampling strategies until we find a candidate I that does not collide with M _h or M _e . This is a straightforward process that can be implemented efficiently.

To capture the regions of interaction proximity on our enhanced IBS as mentioned above, we use the notion of provenance vectors (Ruiz and Mayol-Cuevas, 2020). The provenance vectors of an interaction start from any point on I and finish on M _e . Formally, V p = a , v ⃗ ∣ a ∈ I , v ⃗ = arg min e ∈ M e ‖ e − a ‖ − a (2) where a is the stating point of the delta vector v ⃗ to the nearest point on M _e .

Provenance vectors inform about the direction and distance of the interaction; the smaller the | v ⃗ | , the more important it is in the description. Let V p ′ ⊂ V p be the subset of provenance vectors that finish on any point in P _e , and we perform a weighted randomized selection sampling of elements from V p ′ with the allocation of weights as follows: w i = 1 − | v ⃗ i | − | v ⃗ min | | v ⃗ max | − | v ⃗ min | , i = 1 , 2 , … , | P e | (3) where | v ⃗ max | and | v ⃗ min | are the norms of the biggest and smallest vectors in V p ′ , respectively. The selected provenance vectors V train integrate to our affordance descriptor with an adjustment to normalize their positions, with the defined reference point p _train as follows: V train = a i ′ , v ⃗ i ∣ a i ′ = a i − p train , i = 1 , 2 , … , n u m p v (4) where num _pv is the number of samples from V p ′ to integrate. The provenance vectors alone, however, are insufficient to work successfully on highly articulated objects, such as human poses. They are unable to capture the whole nature of the interaction. We expand this concept by taking a more comprehensive description that considers both areas of the IBS, those that are proximal to surfaces and those that are not.

We include a set of vectors into our descriptor to define the clearance space necessary for performing the given interaction. Given S _h , an evenly sampled set of num _cv points on M _h , the clearance vectors that integrate to our descriptor C train on the interaction are defined as follows: C train = s j ′ , c ⃗ j ∣ s j ′ = s j − p train , s j ∈ S h , c ⃗ j = ψ s j , n ̂ j , I (5) ψ s j ′ , n ̂ j , I = d max ⋅ n ̂ j if   φ s j , n ̂ j , I > d max φ s j , n ̂ j , I ⋅ n ̂ j otherwise (6) where p _train is the defined reference point, n ̂ i is the unit surface normal vector on sample s _j , d _max is the maximum norm of any c ⃗ j , and φ ( s j , n ̂ j , I ) is the distance traveled by a ray with origin s _j and direction n ̂ i until collision with I .

Formally, our affordance descriptor, AROS, is defined as f : M h , M e , p train → V train , C train , n ̂ train (7) where n ̂ train is the unit normal vector on M _e at p _train . We calculate n ̂ train for speeding up the detection process.

Let A = ( V train , C train , n ̂ train ) be an affordance descriptor; we define its rigid transformation with τ ∈ R 3 being a translation vector and ϕ being the rotation around z defined by R _ϕ .

Given a point p _test on an environment mesh M _test and its unit surface normal vector n ̂ test , we determine that such a location supports a trained interaction A if we can find that (1) has a small angle difference between n ̂ test and n ̂ train , (2) once translated to p _test and oriented with ϕ _test , there is a correct alignment of V ϕ τ A , and (3) a gated number of the C ϕ τ A is in collision with M _test .

A significant angle difference between n ̂ test and n ̂ train permits to short-cut the test and reject p _test with reference to A . We establish ρ n ⃗ as the decision threshold for the angle difference. ρ n ⃗ is adjustable based on the level of mesh noise.

If we observe a normal match between p ̂ train and p _test vectors, we perform transformations over the interaction descriptor A with τ = p _test and n _ϕ different ϕ = ϕ _test values within [0, 2π]. Hence, per each 3-tuple ( V ϕ τ A , C ϕ τ A , n ̂ train ) calculated, we generated a set of rays R _pv defined as follows: R p v = a i ″ , ν ̂ i | ν ̂ i = v ⃗ i ‖ v ⃗ i ‖ , a i ″ , v ⃗ i ∈ V ϕ τ A (8) where a i ″ is the starting point and ν ̂ i ∈ R 3 is the direction of each ray. We extend each ray in R _pv by ϵ i p v until collision with M _test as a ″ + ϵ i p v ⋅ ν ̂ i ∈ M test , i = 1,2 , … , n u m p v (9) and compare with the magnitude of each correspondent provenance vector in V ϕ τ A . When any element in R _pv extends further than a predetermined limit max _long , the collision with the environment is classified as non-colliding. We calculate the alignment score κ as a sum difference between extended rays and provenance vectors with κ = ∑ ∀ i | ϵ i p v ≤ m a x long | ϵ i p v − v ⃗ i | (10)

The bigger the κ value, the less the support for the interaction on the p _test . We experimentally determine interaction-wise thresholds for the sum of differences max _κ and the number of missing ray collisions max _missings that permits us to score the affordance capabilities on p _test .

Clearance vectors are meant to fast-detect collision configurations by ray–mesh intersection calculation. Similar to provenance vectors, we generate a set of rays R _cv , whose origins and directions are determined by C ϕ τ A . We extend rays in R _cv until collision with the environment and calculate its extension ϵ j c v . Extended rays with ϵ j c v ≤ ‖ c ⃗ j ‖ are considered as possible collisions. In practice, we also track an interaction-wise threshold to refuse affordance due to collisions max _collisions .

A sparse distribution of clearance vectors on bi-dimensional noisy meshes in a 3D space results in collisions that are not detected by clearance vectors. To improve, we enhance scenes with a set of spherical fillers that pad the scene (see Figure 3). More details are provided in Supplementary Material.

We evaluate the physical plausibility of the compared approaches mainly by following the work of Zhang et al. (2020b) and Zhang et al. (2020c). Given the SDF values of a scene and a body mesh generated, 1) the contact score is assigned to 1 if any mesh vertex has a negative SDF value and is evaluated as 0, otherwise, 2) the non-collision score is the ratio of vertices with a positive SDF value, and 3) in order to measure the severity of the body–environment collision on positive contact, we include the collision-depth score, which averages the depth of the collisions between the scene and the generated body mesh.

We use Amazon Mechanical Turk to compare and evaluate the naturalness of body–environment configurations generated by our approach and baselines. We used only the best version of the compared methods (with optimizer). Each scene in our test set was used equally to select 162 locations around which the compared approaches generate human interactions. MTurk judges observed all human–environment pairs generated through dynamic views, allowing us to showcase them from different perspectives. Each judge performed 11 randomly selected assessments, without repetition, that included two control questions to detect and exclude untrustworthy evaluators. Three different judges accomplished each of the evaluations. Our perceptual experiments include individual and comparison studies for each comparison carried out.

In our side-by-side comparison studies, interactions detected/generated from two approaches are exposed simultaneously. Then, MTurkers were asked to respond to the question “Which example is more natural?” by direct selection.

We used the same set of interactions for individual evaluation studies, where judges rated every individual human–scene interaction by responding to “The human is interacting very naturally with the scene. What is your opinion?” with a 5-point Likert scale according to its agreement level: 1) strongly disagree, 2) disagree, 3) neither disagree nor agree, 4) agree, and 5) strongly agree.

Experiments verify that our approaches can realistically generate human bodies that interact within a given environment in a natural and physically plausible manner. AROS allows us to not only determine the location on the environment in which we want the interaction to happen (the where) but also select the specific type of interaction to be performed (the what).

The number and variety of interactions detected by AROS can easily be increased as a result of its one-shot training capacity. The more trained the interactions, the more the human–scene configuration can detect/generate. Figure 6 shows examples of different affordance detections around single locations.

AROS showed better performance in more realistic environment configurations where elements, such as chairs, sofas, tables, and walls, are presented and must be considered during the generation of body interactions. Figure 7 shows some examples of interaction generated by AROS and baselines over challenging locations.

Alternatively, AROS can be used to concatenate affordances over several positions to generate useful affordance maps for action planners (see Figure 8). This can be used as a way to generate visualizations of action scripts or to plan the ergonomics and usability of spaces beyond individual objects.