The manual method for transoral OP swab sampling is depicted in Figure 1. In this procedure, the testee opens their mouth to expose the OP wall while tilting their head back at an angle of approximately 15°–45°. Simultaneously, the healthcare professional inserts an OP swab into the testee’s mouth and collects a nucleic acid specimen from the OP wall using a pendulum-like movement, guided by visual feedback and force perception through their eyes and hands, respectively.

To replicate the nucleic acid sampling practices of medical workers, the proposed robotic system must provide dexterous manipulation along with visual and tactile force feedback. Based on these requirements, the proposed sampling robot comprises a linear motion module, a flexible series-parallel hybrid mechanism, and a visual-tactile fusion module. The flexible sampling mechanism, mounted on the movable platform of the linear motion module, enables straight reciprocating motion. A series-parallel hybrid mechanism is designed to achieve omnidirectional bending for the pose adjustment of the swab while providing tactile feedback through a force sensor mounted at the distal end. A camera captures an image of the testee’s OP with light assistance while the testee maintains an open-mouth position. A monitor displaying the camera’s live feed is placed in front of the testee to prompt them to expose their posterior pharyngeal wall as fully as possible. Flexible shafts are employed to connect the sampling mechanism to the drive module, serving as a power transmission medium that enables a compact and dexterous spatial layout (Liu et al., 2017). The overview of the OP swab sampling robotic system is presented in Figure 2.

Similar to manual sampling manipulation, the sampling robot utilizes a linear motion module for large-scale movement and employs a flexible series-parallel hybrid mechanism for precise pose regulation. In this study, the series-parallel hybrid mechanism has a diameter of 8 mm and is composed of two bendable segments, each able to achieve a maximum omnidirectional bending angle of 45°. The nomenclature of the robot is depicted in Figure 3, and the kinematics parameters are detailed in Table 1.

Where the kinematics parameters are noted as follows: M_i (i = 0, … ,4): feature point of the origin, the pivot point of the bent joint, and both ends of the distal part. N_i (i = 1, … ,5): pivot point of the universal joint. s: translational journey along axis z . α : polar angle of the distal of the robot under a spherical coordinate system. β : azimuthal angle of the distal of the robot under a spherical coordinate system. θ i (i = 1, … ,4): the bending angle of the bendable segments along axis x or y . L_i (i = 1, … 5): L₁ represents the deviation between the support rod and the origin of the coordinate system along axis x . L₂ represents the deviation between the active rod and the origin of the coordinate system along axis x . L₃ represents the distance between two active rods. L₄ represents the distance between two support rods. L₅ represents the length of the distal part. d_i (i = 1, … 4): d₁ represents the length of the support rod. d₂ and d₃ represent the length of the active rods in segment I, respectively. d₄ and d₅ represent the length of the active rods in segment II, respectively. Each of these rods is designed with a uniform diameter of 2 mm.

During the sampling process, each testee positions their chin on a fixed chin strap, pressing it firmly against an inclined plane tilted at a 30° angle to ensure their head is raised with a backward angle of 30°. The testee then opens their mouth to expose the posterior pharyngeal wall as much as possible, preparing for automatic sampling with robotic assistance. To reduce the risk of cross-contamination, the testee will be provided with a sterile paper, shaped to fit the chin strap, which they will place before the sampling procedure. Additionally, healthcare staff will regularly clean and disinfect the chin strap to ensure hygiene and minimize potential transmission risks. Typically, the mouth opening of an adult measures approximately 50 mm in height. The OP swab head is inserted into the testee’s oral cavity and directed toward the oropharyngeal wall through joint motion facilitated by the linear motion module and the series-parallel hybrid mechanism. In this study’s sampling robot platform, the camera captures a two-dimensional (2D) image without depth information. Therefore, the position of the sampling point in o t - x t y t is identified using visual feature recognition, while the position along the z-axis is determined through tactile force sensing feedback. The details of the position recognition process using the visual-tactile fusion algorithm are discussed in Section 2.3. The relationship among the world coordination system o - x y z , the camera coordination system o c - x c y c z c , and the sampling point coordination system o t - x t y t is shown in Figure 4A.

In order to calculate the workspace of the sampling robot, we define a homogenous transformation matrix as T i − 1   i = R x | y | z T 0 1 (1) where R x = 1 0 0 0 cos θ x − sin θ x 0 sin θ x cos θ x , R y = cos θ y 0 sin θ y 0 1 0 − sin θ y 0 cos θ y , R z = cos θ z − sin θ z 0 sin θ z cos θ z 0 0 0 1 , T = P x P y P z

and θ x , θ y , θ z represent rotating angles along axes x , y , z , respectively. P x , P y , P z represent coordinates along axes x , y , z , respectively.

By substituting the parameters from Table 1 into Equation 1, the forward kinematic chain can be expressed as T 0 8 = ∏ i = 1 8 T i − 1   i (2)

Based on Equation 2 describing the kinematic transformation, the coordinate of the distal of the OP swab is given by P M 4 = T 1,4 0 8 T 2,4 0 8 T 3,4 0 8 T (3)

Therefore, the workspace of the sampling robot can be determined by substituting the kinematic parameters from Table 1 into Equation 3. The calculated workspace is illustrated in Figure 4.

Suppose the coordinate and orientation of sampling point M as P M = M x M y M z , R   o M = u x v x w x u y v y w y u z v z w z (4) where P M represents the coordinate of OP swab tip, and R o M represents the orientation matrix of OP swab in o - x y z coordination system; they can be obtained via Bryan angle transformation. As shown in Equation 5, the coordinate of point o ′ can be calculated as P o ′ = P M + R o M • P Δ (5) where P o ′ represents the coordinate of point o ′ in the o - x y z coordination, and P Δ = 0 0 − L 5 T .

Combining Table 1 and (Equation 1), vector M 1 M 2 → and M 2 M 3 → can be computed as M 1 M 2 → = 2 d 1 s i n θ 2 − 2 d 1 c o s θ 2 s i n θ 1 2 d 1 c o s θ 1 c o s θ 2 , M 2 M 3 → = d 1 c o s θ 4 ⁡ sin θ 2 + θ 3 d 1 − c o s θ 1 s i n θ 4 − s i n θ 1 c o s θ 4 ⁡ cos θ 2 + θ 3 d 1 − s i n θ 1 s i n θ 4 + c o s θ 1 c o s θ 4 ⁡ cos θ 2 + θ 3 (6)

In this study, the series-parallel structure is configured to conform to an arc shape. Therefore, the polar angle of the vector M 1 M 2 → , M 2 M 3 → is α /2 and α respectively. Based on Equation 6, the relation between θ i (i = 1, … ,4) and α , β can be expressed by c o s α 2 = 2 c o s θ 1 c o s θ 2 (7) t g β = − s i n θ 1 c o s θ 2 s i n θ 2 (8) c o s α = − s i n θ 1 s i n θ 4 + c o s θ 1 c o s θ 4 c o s θ 2 + θ 3 (9) t g β = − c o s θ 1 s i n θ 4 − s i n θ 1 c o s θ 4 c o s θ 2 + θ 3 c o s θ 4 s i n θ 2 + θ 3 (10)

Based on the combined use of (Equations 7, 8), θ i (i = 1,2) can be calculated as θ 2 = a r c s i n s i n α / 2 1 + t g 2 β , θ 1 = a r c s i n − t g β ⋅ t g θ 2 (11)

Due to θ 3 and θ 4 coupling with each other, an iterative algorithm is established to calculate θ 3 and θ 4 to reduce the difficulty of the solving process. Combining Equations 9-11, the Pseudo code is presented in Table 2.

By substituting the optimal solution into Equation 3, the translational journey s can then be calculated using the combined application of Equations 3, 4.

The posterior pharyngeal wall is the recommended site for oropharyngeal swab sampling. As illustrated in Figure 1, the camera captures an image of the testee’s oral cavity after they open their mouth to expose the posterior pharyngeal wall. Since the image input into the computer is of a fixed size, the top left corner of the image is defined as the origin. The coordinates of the posterior pharyngeal wall relative to this origin are then calculated in the o t - x t y t frame (Figure 5). To effectively identify the boundaries of the posterior pharyngeal wall, the features of the tonsil glands, posterior pharyngeal wall, and uvula are initially labeled. Subsequently, a recognition model for the posterior pharyngeal wall is developed using a deep learning algorithm. The algorithm steps are as follows.

Figure 8 illustrates the operation of the sampling robot during OP swab sampling. The detailed procedures are as follows: ① the sampling robot moves to its initial straight position (s = 0, θ i = 0, i = 1, … ,4). ② A testee takes a disposable OP swab and sits in front of the sampling robot. ③ the testee inserts the OP swab into the distal holder of the sampling robot. ④ the testee leans their chin against the tilted chin strap and opens their mouth as wide as possible to expose the oropharyngeal wall while self-checking the monitor image suspended in front. The coordinates of the sampling point are calculated using the method described in Section 2.3. The polar angle α and azimuthal angle β of the flexible mechanism are subsequently computed based on the coordinates of the sampling point. ⑤ the translational joint advances forward and stops when the base of the sampling robot reaches fixed point 1, positioning the head of the OP swab near the testee’s oral cavity. ⑥ the distal of the sampling robot is adjusted to posture with parameters ( α , β ). ⑦ the translational joint moves forward until the head of the OP swab has contacted the oropharyngeal wall, then stops moving forward when the contact force exceeds 0.15 N. ➇ the distal of the sampling robot will perform an arcing trail to pose ( α , β +20°), which allows the head of the OP swab to collect enough sampling specimens. ➈ the sampling robot returns to the initial position. ➉ the testee removes the swab and leaves.

A volunteer was invited to participate in this experiment. The volunteer opens his mouth and self-checks the mouth pose on the monitor located at the front. Then, the zone of the posterior pharyngeal wall was identified, and the coordinates of the corner points were calculated by visual processing with a deep learning algorithm. The experimental results during sampling work are shown in Figure 9. The variation of a trace of the robot parameters during sampling operation is presented in Figure 9A, the meanings of labels from ④ to ➈ are the same as that of labels in Figure 8. In this sampling trail, the coordinates of the sampling point in the oral cavity were (−1.67, 149.05) mm and (17.71, 149.05) mm computed by the visual processing method (Figure 9C), then, the polar angle and the azimuthal angle of the sampling robot were calculated as (34.18°, 0.64°) at the left sampling point, which could be further used to resolve the bending angles of the series-parallel mechanism. The OP swab was inserted into the testee’s oral cavity after the distal posture had been adjusted to the angle of (34.18° and 0.64°), and stopped when the tactile force detection had exceeded 0.15 N. Then, the posture of the OP swab was driven to (34.18°, 20.64°) to collect more specimens, mimicking the sampling skill of the medical professional. The tactile force of the sampling robot was shown in Figure 9B, and the value was around 0.15 N during the whole sampling practice.

Nucleic acid testing is an effective method for identifying individuals infected with COVID-19 or seasonal influenza. OP swab sampling is commonly used to collect specimens from the testee’s oral cavity. Despite stringent protective measures, medical staff remain at risk of infection due to the close contact nature of the procedure. In this study, we proposed an OP swab sampling robotic system comprising a series-parallel hybrid structure manipulator and a visual-tactile fusion module for autonomous sampling navigation. The system aims to minimize interaction between medical personnel and the testee, addressing safety concerns and improving sample quality consistency. A monitor positioned in front of the testee replicates the image of the oral cavity captured by the camera, allowing the testee to self-check their oral posture. A deep learning algorithm was developed to identify the sampling position in the posterior oropharynx. The preliminary trial demonstrated the feasibility of the robotic system for OP swab sampling, as shown in Figure 9. However, further research is needed to improve the system’s precision and adaptability in different clinical environments.

We designed a serial-parallel hybrid structure manipulator to perform posture adjustments for OP swab sampling. Compared to existing autonomous sampling systems that rely on limited DOF manipulators [such as the 2-DOF end effector in (Wang et al., 2021) and the 3-DOF end effector in (Hu et al., 2022)], our 4-DOF serial-parallel hybrid mechanism demonstrates superior adaptability to complex oral anatomies, achieving full workspace coverage in simulated oral cavities (Figures 4B–E). Additionally, the successful implementation of the automatic oropharyngeal swab experiment further validates the flexibility in posture adjustment provided by this hybrid mechanism. It can be configured into an arc shape by minimizing the deviation angle between the two bendable segments, and it can also be adjusted into a z shape by driving the two segments to bend in opposite directions. This enables the robotic system to adapt to a variety of oral configurations, reducing the potential for inaccurate sampling caused by rigid manipulators. At the same time, compared to serial-jointed end effectors (Hu et al., 2022; Wang et al., 2021; Li et al., 2021), the serial-parallel hybrid manipulator offers significant potential for OP swab sampling, even in scenarios involving obstacles such as an arched tongue or tooth occlusion. Although promising results were achieved, several challenges remain. For example, in the series-parallel hybrid mechanism, driven by flexible shafts, the rotational drive amount from the motor output is not perfectly transmitted to the distal end. Due to the elasticity and friction of the flexible shafts, one full rotation of the motor does not equivalently drive the distal segment of the flexible shafts to reach the expected drive amount, which makes the intermediate transmission model require more precise construction to ensure accurate system control. Additionally, the complexity of the system increases the time required for each test. While the duration is competitive with manual procedures, it may still vary depending on the testee’s cooperation and the complexity of the oral anatomy. Furthermore, to further optimize the system’s design and functionality, future research will also focus on user experience evaluations. This will include collecting feedback from medical personnel and testees to assess the system’s usability, ease of operation, and comfort during the testing process.

Traditional methods for detecting the posterior pharyngeal wall typically involve first detecting the face and oral cavity, followed by identifying the posterior pharyngeal region within the oral cavity (Sun et al., 2022). In this study, the developed recognition algorithm directly trains on multi-scale features of the posterior pharyngeal wall and is optimized using adaptive anchor strategies, ensuring adaptive recognition of feature regions of varying sizes. Mosaic data augmentation was applied to 42 original images to expand and construct the dataset, resulting in a posterior oropharynx detection accuracy of 97.9%, highlighting the effectiveness of the method. Moreover, the model achieved an average precision (mAP50) of 85.7% for uvula detection, indicating good performance across both target regions. These techniques ensure that the model can better generalize the target location information in images, making it more adaptable to real-world data inputs. Additionally, our approach demonstrates the potential to achieve high performance with a smaller training dataset, which is a key advantage when dealing with the time-consuming and costly image collection process in clinical environments. Furthermore, the model’s fast inference speed of 15 ms per image makes it suitable for real-time applications. While data augmentation helps mitigate the limitation of a relatively small dataset, future work will involve incorporating larger and more diverse datasets to improve the model’s generalization ability and extend its application in a wider range of clinical settings.