In response, this study introduces a structured online platform tailored to streamline the data acquisition and modeling processes vital for building renovation. This platform provides a systematic approach, guiding users through a series of steps that begin with the collection of fundamental geometric data and culminate in the creation of a normalized 3D model compatible with BIM formats, such as. ifc shown below. By automating critical steps and ensuring data integrity, the platform addresses the fundamental limitations of current practices while promoting accessibility for non-expert users.

Enter project name: test01

Enter building name: building01

Enter building height in meters: 12

Enter number of floors: 4

Floor height is: 3.0 m

Enter changed height of each floor: 3

Building height is: 12.0 m

Enter extrusion length in meters: 0.3

The data acquisition platform uses a modular pipeline to generate a normalised 3D representation of building façades, incorporating both user inputs and image-based detection. As shown in Figure 1, the process begins with the collection of two types of information from the user: (1) façade images and (2) basic building metadata, such as the number of floors, total height, and extrusion depth. These are submitted to the platform, where the geometric data is processed alongside a vision-based module that uses pretrained algorithms to extract façade opening features (e.g., windows and balconies). This step produces a. json file containing spatial data on the openings in the façade.

As illustrated in Figure 3, the user-submitted metadata is used separately to generate an initial model structure represented as a. json file without opening information (Process 2). The detected openings are pre-processed and refined (Process 3), after which the two data streams are combined into a unified JSON file containing complete geometric and topological data (Process 4). The platform then generates a. json file with integrated opening information (Process 5) and performs normalisation (Process 6) to produce a final file that complies with BIM standards. This normalised file is structured for compatibility with downstream tools, including FreeCAD, IFC exporters, and CAM modules. The modular nature of this pipeline ensures flexibility, allowing updates to individual steps without disrupting the entire workflow. The result is a high-fidelity digital representation of the building that supports accurate energy modeling and renovation planning. This stepwise approach has been shown to reduce manual intervention and to promote consistency and reproducibility across projects, a critical advantage in large-scale renovation efforts (Ferrandiz et al., 2018; Mainicheva et al., 2017; Chegu Badrinath et al., 2016).

The platform addresses the issue of incomplete user input—especially for building heights—by integrating a height estimation feature grounded in projective geometry. This algorithm utilizes camera parameters and image-based measurements to compute real-world building heights, even in cases where data is sparse. The user is only required to provide the horizontal distance, D. The remaining values, such as sensor size and resolution, are automatically extracted, thereby significantly reducing the barriers to accurate modeling (Gerum et al., 2019).

The platform’s viability was substantiated through a series of pilot studies on buildings of varying complexity. In the case in Figure 4, a building with six facades was processed to verify the platform’s fundamental functionality.

To be mentioned, the input of this platform should be from users, and the output should be the final normalized. json file, which can be transferred into the. ifc file for the visualization. And the approximate external shape of the model and the address information (Munich, Germany) are provided here on the left side of Figure 4.

The renovation process begins with the generation of a 3D model of the building using publicly available facade images and OpenStreetMap (OSM) floor plans. From these inputs, a structured geometric model is created as shown in Figure 5a, which serves as the foundation for the placement of the modules. The FreeCAD-based plugin was developed to generate layout configurations of prefabricated facade elements integrated with photovoltaic and thermal solar panels. This tool allows users to analyze wall geometries, extract facade features (e.g., height, width, corner points, and window locations), and simulate solar panel arrangement strategies, streamlined early-stage planning that is typically time and labor-intensive.

The core goal is to automate the placement of prefabricated modules on the building facade based on geometric feasibility and energy efficiency. A multi-step logic is applied to optimize the number of solar panels while respecting architectural constraints, such as window positions and panel dimensions. The layout generation process shown in Figure 6 is completed in minutes and provides detailed visualizations, facilitating informed decisions by architects and engineers. The modules include structural frames, insulation layers, and wiring pathways, making them ready for robotic assembly.

Once the initial layout is approved, precise on-site measurements are captured using AprilTags positioned on key facade points like window edges and connection anchors as shown in Figure 5b. These measurements are used to update the digital model with higher accuracy. The system then automatically reconfigures the prefabricated module layout to align with the measured geometry, minimizing errors during production and installation, as shown in Figure 5c. This adjustment process addresses the limitations of initial online modeling and enhances the reliability of the final output.

All outputs—including updated FreeCAD models, JSON files with module data, and Excel reports listing connector positions—are structured to support direct integration with robotic fabrication workflows. These outputs also include dimensional data critical for CAM, enabling precise prefabrication of the components.

An example application on a complex 3,000 m² facade (the Milan demo building) demonstrated that what once required over 1,000 h of manual drafting can now be accomplished in under an hour with this system (25 min each for layout definition and realignment). Figure 5d illustrates the updated building model after layout optimization.

This implementation showcases how a data-driven and semi-automated workflow—starting from online modeling and ending in precise on-site layout adjustment—can transform renovation into a scalable, industrialized process, significantly lowering cost and time while preserving architectural integrity and energy performance.

First to be mentioned, the development of a quadcopter with an end effector was undertaken based on Hsu et al. (2024). The following text is an excerpt from the aforementioned source: The quadcopter will approach the target position. The application of the load to the end effector is indicated by the presence of the AprilTags. The reverse side of the tags bears the following inscription: The adhesive is applied in the following manner. Following the establishment of contact between the tags, the surface on which the pressure is to be applied by the quadcopter is hereby designated as the “target surface”. It is imperative that the tags are utilized in order to guarantee a robust binding. The process is indicated as illustrated in Figure 7.

Traditional survey methods, while often precise, tend to be labor-intensive and constrained by logistical and financial limitations. To address these challenges, fiducial markers known as AprilTags have been employed to enable efficient 3D reconstruction of building facades using only digital imaging and straightforward computational techniques (Zhang et al., 2021). These markers, when arranged in systematic grid formations on the surface of a structure, facilitate the derivation of spatial geometry through triangulation methods based on calibrated camera imagery.

A key innovation in the proposed integrative workflow is the use of a best-fitting plane to reduce spatial noise and improve the alignment of coplanar data points derived from AprilTag positions. By applying a least-squares optimization, the reconstructed points are projected onto the most likely planar surface, significantly enhancing the accuracy of the model even in the presence of minor measurement errors or optical distortions (Iturralde et al., 2023b). This correction method is particularly valuable when the resulting geometric model is used for applications with stringent spatial requirements, such as determining optimal solar panel placement or simulating urban light exposure.

Central to the integrative workflow’s accuracy is a robust approach to distance estimation. This is achieved through the use of effective focal length and sensor pixel size, which together allow for precise real-world measurements based on the apparent size of the tags in the image frame. The system leverages camera calibration data and computes distances using a similarity-based geometric model, ensuring that the spatial positions of each tag can be accurately resolved. The experimental setup employed a high-resolution Sony Alpha 7R IV camera, with a computed effective focal length of approximately 13,257.6 in appropriate units, delivering sub-millimeter accuracy in tag distance estimation.

In practical testing, a key factor affecting accuracy is the apparent size of the AprilTags within each image. For instance, when capturing a 15-meter-high building, the 200 mm × 200 mm tags appeared too small and blurred in wide-angle shots. To address this, photographs were taken from multiple viewpoints, with each image containing only four adjacent tags to ensure they appeared large and clear, as shown in Figure 8a. This method allowed for detailed local captures while still covering the entire façade.

Once the distances to the individual tags are known, the system proceeds to determine their 3D coordinates relative to the camera. This is followed by projecting those coordinates onto the best-fitting plane, refining their placement before final distance measurements are made using Euclidean metrics. The AprilTag detection, as shown in Figure 8b, and data processing were implemented using Python libraries such as the AprilTag detector and scikit-spatial for plane fitting, enabling a streamlined and accessible reconstruction integrative workflow with minimal manual intervention.

Additionally, for cooperating with AprilTags, the MK (Matching Kit), which is a set of components that includes a bespoke interface for the purpose of correcting any deviations that occurred during the placement of the AprilTags on the wall, is introduced here with regard to Iturralde et al. (2023c). This MK is not predicated on a specific connector type; rather, it is founded upon a conceptual framework that delineates the interface between the facade and the wall. In previous phases of the research, the MK and its main components were defined. A series of tests was conducted, resulting in enhanced accuracy and reduced time consumption. The MK is comprised of three primary components (see Figure 9).

An integral step in streamlining facade renovation using prefabricated modules is the seamless integration of automated layout generation with CAM systems. In the workflow, once the primary layout of prefabricated facade modules is derived from a 3D Point Cloud, this geometric information is directly linked to CAD/BIM environments and ultimately synchronized with CAM platforms to enable digital fabrication of the modules (Sandberg et al., 2016). This process effectively minimizes manual intervention and ensures that the layout generated from existing building geometries can be translated into manufacturable components with minimal redesign.

The CAM integration is facilitated through interoperability with software such as Revit via Dynamo™ scripts, which convert the semi-automatically extracted polygonal module boundaries into CAD-readable formats (Sandberg et al., 2016). These formats can then be used to define the structure and configuration of prefabricated panels in manufacturing software like Dietrich’s®, which automates the detailing of structural components such as studs, insulation layers, and fireproof barriers once the primary layout is defined (Sandberg et al., 2016; Iturralde et al., 2024). This bridge from data acquisition to fabrication not only enhances design efficiency but also enables mass customization of building envelopes while preserving the geometric fidelity of existing structures.

A crucial advantage of this approach lies in its adaptability to production constraints. The semi-automated layout algorithm takes into account key CAM parameters, such as maximum and minimum module widths, slab positions for connector placement, and alignment with existing window openings (Sandberg et al., 2016). These design rules are incorporated upstream in the layout generation process to ensure that the final output adheres to manufacturing limitations, thus avoiding costly revisions downstream.

However, the success of the CAM integration is strongly dependent on the quality of the initial Point Cloud. As highlighted in the study, inaccurate or incomplete point data can propagate through the entire integrative workflow, resulting in deviations between the generated layout and actual building conditions—especially problematic when modules must fit tightly to existing facades with minimal tolerance (Sandberg et al., 2016).

Based on the rationale above, a CAM generation workbench named ENSNARE_CAD_CAM is created in FreeCAD, allowing geometry properties of individual profiles across an entire module to be exported as a single JSON file. The geometry properties that can be exported include each profile’s dimensions, CoM (Center of Mass), and placement relative to the origin of the module as shown in Figure 10a.

A Python script is implemented to extract the coordinates of each PUP (Pick-Up Point) and derive the optimal sequence based on the JSON file. A ROS wrapper is then implemented to expose this two information as a ROS service, enabling the main controller to request the data at run time. By requesting the assembly sequence, a list of strings containing the names of the modules is returned. Sequentially, quarrying with the name of each module, the PUP coordinates are returned.

With the automated prefabricated facade layout definition process developed in Iturralde et al. (2023b), the properties are automatically extracted from the CAD model and exported to a Excel file, from which a Python dictionary is implemented to index these data in an easy-to-retrieve manner. The same as before, a ROS wrapper exposes the relevant data as ROS services. This new procedure increases data retrieval efficiency, as shown in Figure 10b.

Before updating the layout to match real site conditions, it’s important to make sure the initial design fits the building as it actually is. Since older buildings often have small differences or irregular shapes, this step helps prepare the layout for accurate adjustment. The next part explains how the layout is first created using the building model and design rules.

The architecture implements a data-driven pipeline that systematically processes geometric information from CAD models into executable robot trajectories. The pipeline begins with the Geometry Analyzer, which parses the CAD model to extract critical parameters including component dimensions, PUPs, and optimal assembly sequences based on spatial constraints. These parameters are encoded as transformation matrices and sequential instructions that flow to the Motion Generator. This module performs coordinate frame transformations between the module origin frame MO and the robot’s world frame w using homogeneous transformations, converting design specifications into robot-specific target poses. The resulting pick-and-place coordinates are then processed by the Robotic System Control modules, which decompose the assembly task into synchronized trajectories for the manipulator, linear rail, and gripper subsystems. This modular architecture enables real-time adaptation to component variations while maintaining assembly precision through feedback between the digital model and physical execution.

Once the layout has been corrected based on real measurements, it needs to be turned into files that machines and robots can use. This step is important to make sure each part is made and placed correctly. The next part shows what kind of output files are created and how they are used in the production process.

The successful execution of the assembly task relies on high precision in the installation and calibration of the robotic assembly system. Any offsets between the simulated and real model, for instance, the displacement of the manipulator’s base on the rail or imprecise calibration of the location of input and assembly platforms, lead to deviation of the final assembly result. To eliminate potential errors and avoid cumbersome calibration, a camera is mounted on the wrist of the robot arm to detect the position of the platforms and the profiles at run-time. Advanced ArUco markers (Kedilioglu et al., 2021) are attached to the platforms and the PUP on the profiles. A marker-based vision approach is utilized to accurately detect and calculate the pose relation between the end-effector and the profiles.

As shown in Equation 3, a hand-eye calibration that determines the transformation between the camera optical frame and the end-effector frame T c e e is conducted using MoveIt’s hand-eye calibration tool (MoveIt, 2023b). ArUco marker detection is implemented with OpenCV’s ArUco module library (OpenCV, 2023), which provides pose estimation of the marker in the camera optical frame T m c . Thus, the transformation between the marker (target pose) to the end-effector frame (current pose) can be obtained by: T m e e = T c e e T m c (3)

The target pose in the world frame is, therefore: T m w = T e e w T m e e (4)

By applying the Equation 4, any marker pose in the world frame can be easily obtained if the camera detects it. If the marker is attached to the PUP of each profile, the picking poses for the end-effector can be calculated at run-time instead of using the manually assigned coordinates measured in advance, which can be inaccurate. Moreover, instead of feeding the profiles to the input platform at the same position every time, now the profiles can be casually placed at any point on the input platform where the camera can detect them. Similarly, by attaching a marker on the assembly platform where the module origin should be, we can save the effort of manual coordinate measurements.

Two sets of experiments are conducted to validate the accuracy and efficacy of the presented system. The first one evaluates and compares the accuracy of the pick-and-place poses of a single aluminum part with and without the visual marker detection module, as shown in Figure 14(1). The second experiment evaluates the success rate of the full assembly process of a prefabricated module consisting of four aluminum parts and demonstrates the efficacy of the presented system in real applications.

We first evaluated the accuracy of picking and placing a single aluminum part using the presented system, with and without the visual aid module. In both experiments, as shown in the snapshots in Figure 14(1), an aluminum part is picked up from the input platform on the left side and placed on the assembly platform on the right side. The difference is, in the baseline mode without the visual aid, the pick and place positions are pre-defined in the calibrated workspace, whereas in the visual aid mode, the pick position is marked by the ArUco marker attached on the surface of the aluminum part, and the place position is marked by the ArUco marker attached to the assembly platform, such that the part can be pre-placed at any position on the input platform. We conducted 10 experiments in each mode and measured the average errors of position and orientation, separately. The position error is based on 2D (two-dimensional) Euclidean distance between the actual positions to the reference positions, and for orientation, we measure the yaw angle error wrapped between [ − π , π ] , as defined in Equation 5. e pos = 1 n ∑ i = 1 n x i − x ref , i 2 + y i − y ref , i 2 e ori = 1 n ∑ i = 1 N wrap θ i − θ ref , i (5)

Table 1 shows average errors in the position and orientation of the pick and place in each mode, respectively. Discussion: Without visual aid, small errors for picking and large errors for placing the side. The reason is that calibration is done at the robot’s home position, which is closer to the pick positions. Errors occur when the rail moves since the rail system is not calibrated, so the other side has large errors. With visual aid, the errors mainly come from changes in light conditions and the noise of marker detection. The increase in orientation error observed in the tag-based (visual aid) mode is primarily due to variability in lighting conditions and detection jitter in the ArUco marker pose estimation. While visual localization improves flexibility and position accuracy, small angular deviations may result from inconsistent tag visibility or blur in the image frame. This is more pronounced during placement tasks that require finer rotational alignment, especially when the robot arm is operating far from its calibrated home position.

Furthermore, we compared task durations between traditional manual approaches and the automated system. As shown in Table 2, tasks such as facade layout and clash-checking were reduced from hundreds of hours to under an hour, demonstrating the pipeline’s substantial time-saving potential.