Project Summary

HERON aims to develop an integrated automated system to perform maintenance and upgrading roadworks, such as sealing cracks, patching potholes, asphalt rejuvenation, autonomous replacement of CUD elements and painting markings, but also supporting the pre/post-intervention phase including visual inspections and dispensing and removing traffic cones in an automated and controlled manner. The HERON system consists of:

i) autonomous ground robotic vehicle that will be supported by autonomous drones to coordinate maintenance works and the pre-/post-intervention phase;

ii) various robotic equipment, including sensors and actuators (e.g., tools for cut ad fill, surface material placement and compaction, modular components installation, laser scanners for 3D mapping) placed on the main vehicle;

iii) sensing interface installed both to the robotic platform and to the Road Infrastructures (RI) to allow improved monitoring (situational awareness) of the structural, functional and RI’s and markings’ conditions;

iv) the control software that interconnects the sensing interface with the actuating robotic equipment;

v) Augmented Reality (AR) visualization tools that enable the robotic system to see in detail surface defects and markings under survey;

vi) Artificial Intelligence/AI-based toolkits that will act as the middleware of a twofold role for: a) optimally coordinating the road maintenance/upgrading workflows and b) intelligent processing of distributed data coming from the vehicle and the infrastructure sensors for safe operations and not disruption of other routine operations or traffic flows; and

vii) integrate all data in an enhanced visualization user interface supporting decisions and

viii) communication modules to allow for Vehicle-to-Infrastructure/-Everything (V2I/X) date exchange for predictive maintenance and increase users safety.

HERON aims to reduce fatal accidents, maintenance costs, traffic disruptions, thus increasing the network capacity and efficiency.


The (semi-) automated HERON system relying on improved intelligent control of a multi-degree-of-freedom (MDOF) robotised vehicle, improved CV and AI/ML techniques combined with proper sensors, decision-making algorithms and AR components to perform corrective and preventive maintenance and upgrading of roadworks is considered an advanced solution, which pushes routine roadwork activities quite beyond the state-of-the-art. At the same time, by using advanced data coming from various sources (including V2I and aerial drone surveillance) and well-established methods (from existing know-how from research and industrial projects), the automated system will be able to provide some non-routine (emergency) maintenance operations when required. Towards that direction, HERON targets the development and prototype validation of an innovative, automated intelligent robotic platform for performing the above tasks safely, promptly, reliably and modularly.


HERON targets the following Scientific and Technical Objectives (STOs):

HERON improves Real-Time (RT) perception and cognition of the robotic vehicles, through Simultaneous Localisation and Mapping (SLAM) and novel Machine Learning (ML) techniques. SLAM will provide the concurrent mapping generation of the robotic vehicles’ space (complementary to GPS). The algorithms will be used for the navigation of the robotised vehicles allowing manoeuvres and avoiding collisions. The improved cognition will be based on recent advances in Computer Vision (CV) and ML for detecting objects, signaling, road markings and Point of Interests (Pols), interpreting events and actions, which occur in the area of intervention. Existing robotic vehicles will be used and enhances with all necessary sensors and cognition devices so to perform the maintenance and assessment of the RI.

This requires: a) specialized robotic equipment attached on the vehicle to take care of certain RI maintenance and upgrading works such as modular components installation, cut and filling, surface material placement and compaction, painting, spraying, and b) a control mechanism to transform the high-level processes into low-level robotic manipulation skills. These tools are in close relation with the robotic equipment programming algorithms and the sensing interface. Improved learning-based methods to acquire specialized low-level manipulation and interaction abilities will be developed (e.g., grasping, pulling, pushing, cutting, filling) through a combination of sim-to-real and user-provided demonstrations. These low-level abilities will be further refined into higher-order skills (e.g., pouring, levelling, etc.) by leveraging a symbolic representation framework. Using such representations provides access to powerful planning tools that can be used to generate high-level behaviours such as sealing cracks, patching potholes, painting markings, etc.

In order to achieve a precise, automated maintenance and upgrading task, an accurate inspection of the PoI is needed, as well as wider knowledge of the surrounding area. This is done through the sensing interface installed on the robotic vehicle, the drones and on the RI. Surface 3D mapping and modelling5 are some of the salient parts of HERON sensors aiming at extracting precise details on 3D geometry of a road defect and thus stimulate the proper control mechanisms for its repair or the activation of pre-fabricated solutions (such as the Demountable Urban Roadway -CUD- elements) to modularize the planning and accelerate the maintenance and upgrading tasks. CV tools will be applied towards a traffic management for a safe flow and working conditions of the maintenance personnel (access to PANOPTIS outcomes2). Regarding the communication tools, HERON will support any module (e.g., 4G/5G, WiMAX, BLE4, etc.) that will be integrated into the robotic vehicle (as in STO-5), allowing the robotised vehicle(s) to have different configurations for communicating to the each other, the field maintenance team and the control center.

The AI tools will be developed on novel deep ML algorithms and receive as input information from all data sources, installed either on the robotic platform or in the RI or being inputted from external sources (proper V2I data exchange for predictive maintenance, Copernicus, etc.), fuse this information and extract high-level conceptual decisions on the maintenance process. In addition, image analysis and CV algorithms will be applied to allow a continuous traffic flow during the maintenance works and guarantee a safe execution of routine works on the RI by the expert-personnel. Convolutional Neural Networks (CNNs), Long Short-Term Memory Networks (LSTMs), and Generative Adversarial Networks (GANs) will be exploited as Deep Learning (DL) tools, which will be aligned with vision-based object detection algorithms, defect detection schemes and traffic flow analysis modules. Finally, HERON supports a distributed AI architecture including the widely publicized paradigm of Federated Learning6, which is the epitome of collaboration between edge nodes (Unmanned Ground Vehicles-UGVs) and cloud for meeting privacy and application performance requirements. HERON will focus on optimizing a cloud/edge solution for a big class of robotic applications in RIs.

Robust communication will be developed for keeping the network organized even in case of a link/node failure; the HERON network will be self-healing with reconfiguration around a broken path or alternative connection, ensuring continuous connectivity. Our proposed architecture aims to operate in a seamless and cooperative manner with existing human maintenance and technical crews of the roads, as well as integrating HERON to the existing maintenance/upgrading mechanisms. It will act as the technological backbone to provide improved Data Fusion (DF), seamless interconnection and interoperability between the system layers minimising data ambiguity.

We will integrate all the information to be provided by the various tools, the sensor data, etc. as different layers in a unified enhanced visualisation User Interface (UI) generating the Common Operational Picture (COP), as a complementary support to the prevailing Transport Management Systems (TMS) and Closed-Circuit Television (CCTV). The IMS will provide the information of facilities, equipment, personnel, procedures and communications. Assisted by the DSS tool, IMS will manage seamlessly routine incidents as well as emergency situations in the RIs. The decisions and response procedures per reported incident will be turned to actions and resource proposals by building related incident process workflows to support the decision making of RI operators. The visualization will be enriched with AR components to allow for the user to have an in-situ supervision of the automated maintenance and upgrading operation process.

The cost, features, benefits and reaction time of the proposed system and incurred procedures will be compared to conventional (pre-project) procedures and means of motorway operation.

Pilot 1

The pilot will be deployed in the A2 Motorway stretch maintained by the company (R2–CM42
stretch, coming from Madrid, and finishing in the limit between the provinces of Guadalajara and Soria, Spain), and
in the traffic control centre of the stretch located near the village of Torija The motorway is owned by the Spanish
National Road Authority and the section selected has a length of 77.5 km, The section has 4 lanes (2 per traffic
direction) and crosses a region with Continental-Mediterranean climate, with long and severe winters, long, dry
and hot summers and high heavy traffic levels, so pavement is exposed to severe requirements and maintenance
is crucial to preserve the optimum pavement conditions required. A2 is one of the main motorways in Spain,
connecting Madrid with Barcelona, it is part of the Trans-European Transport Network (TEN-T) and the CEF corridor.

Pilot 2

Transpolis is a proving ground of more than 80ha, which has been created by 5 entities among
which UGE and which has been opened officially in 2019. It is typically used to test autonomous vehicles in a secure
and controlled environment, also by assessing the V2I communication possibilities (several types of Road-Side-
Units- RSUs- and communication means) are already installed on site. It also is composed of several kilometers of
road and all reinforced concrete buildings. Many types of V2X and I2V (Infrastructure to Vehicle) communication
means are available, as well as camera monitoring, all of them will be used during the HERON activities.

Pilot 3

OLO has undertaken the traffic management and routine maintenance of the Elefsina-Korinthos-
Patra motorway (in the heart of the Greek highway networks), which has 202 km total length and includes more
than 25 km tunnels and a large number of bridges, culverts and ancillary structures. It includes corrective and
preventive maintenance both of civil works equipment and Early Equipment Management (EEM) of open road and
tunnels. OLO will provide a part of the motorway, where extensive tests of the automated vehicle can take place,
issuing the necessary permits in cooperation with the relevant Authorities (Public Service and Traffic Police) and
ensuring safety conditions both for the road users and the people working for the project. The area that will be
examined during the pilot program is the ELKO section, which is a dual carriageway with three lanes (3.5m width
left lane, 3.75m width middle and right lane) and emergency lane (varies from 2.5 to 3.5m) per direction with
concrete New Jersey safety barriers in the central axis of the motorway. Some major technical features of ELKO
section: ⁕ Total length: 64 km; ⁕ Interchanges (I/C): 12; ⁕ Bridges: 16; ⁕ Tunnels: 5 (total length of 4,473 km).

Deliverables Progress

Starting Date: 1st of June, 2021