2.1 Equity in shared control applications

Whilst the notion of shared control has been around for quite some time, we need to reflect on what we are really trying to achieve. In a world where we hear about inequality, are we striving for equality, where mass production treats every user of a human-machine system the same? We would argue no; instead, as control engineers, we have the power to strive for equity. Equity is about fairness. For example, within the healthcare sector, instead of distributing resources equally between all users or regions, equity is about distributing according to need [19]. Translating this concept into shared control applications, this means we should not necessarily give every user the same level of automation assistance. By contrast, we should give each user the level of assistance they require as an individual, to be afforded the same control opportunities to succeed in undertaking their desired task (Figure 3). And I argue that in many cases, even in mass production, this equity could be achieved through software, by adapting system behaviours to meet the needs of the individual.

Figure 3:

Equity is about fairness. For shared control applications, this means we should not give every user the same level of automation assistance. Instead, we need to give different users the level of assistance they require as individuals, to be afforded the same control opportunities to succeed in undertaking the desired task (illustration CC BY SA 4.0, interaction institute for social change | artist: Angus Maguire, https://interactioninstitute.org ).

On the surface, some of our early work in shared control seemed to be achieving this, as we applied a linear-blending method to simply quantize the user’s input signal gradually towards a safe trajectory that was generated by a robotic planner [6]. Whilst this did provide different levels of assistance to each user, enabling them to achieve the same task, it neglected to consider each individual user’s own preferences. Consequently, although the approach resulted in a successful and safe task completion from the automation perspective, it was not subjectively acceptable to all the participants, nor was it necessarily efficient [20]. This was attributed to the users sometimes “fighting” against the shared control as it deviated from their personal comfort zones (for example in terms of speed, acceleration or clearance from an obstacle). In this example, despite both the human and the machine having the same overall goal, they may not have had well-matched value functions for the optimisation process.

When moving around, there are of course many possible safe trajectories, so whilst the automation may plan one safe trajectory, the user may be attempting to take an alternative – yet equally safe – route. When we take a linear blending approach to resolving this conflict, as Trautman showed, it can result in a suboptimal, and sometimes unsafe, trajectory [21]. Therefore, we moved away from linear blending and instead developed a probabilistic shared control (PSC) framework, whereby the automation can consider the probability that a user would select each of a multitude of different safe trajectories generated by a robotic planner. We then also fit a likelihood function to the instantaneous user input. For example, if a user is proficient with joystick manipulation you could apply a delta function to the current joystick vector (target translational and rotational velocity vector). However, if the user is less proficient – or, for example, has a tremor – you could instead apply a Gaussian distribution over a range of vectors, centred on the current joystick position, thus accounting for potential noise – or uncertainty – in the user input signal. This capability becomes even more pertinent when you consider discrete inputs such as head arrays, or sip-and-puff switches, which would otherwise result in a more bang-bang style of control (Figure 2). Then, instead of a naïve linear blend of the human and automation control signals, you simply select the trajectory that maximises the joint probability distribution between those proposed by the robotic planner and those indicated by the actual user input [22]. Whilst this approach allowed users to follow their preferred path more closely, and experimentally improved acceptability to some extent over the linear blending method, there was still a mismatch between exactly how the user wanted to reach the goal and the resultant style of driving. Therefore, we should further explore how to estimate the user’s own value function and how their individual preferences could be (automatically) accounted for during the control optimisation process.

2.2 Personalised control optimisation

We begin by drawing upon lessons from the automotive industry. For years, most car manufacturers have given drivers the option to choose between preset vectors of control parameters that suit different driving and performance styles [23], [24], [25]. This is normally achieved by allowing them to manually press a button or change a default driver profile setting to switch preset modes between e.g. eco, normal and sport etc. Whilst this goes some way to personalising the driving experience, in most cases the parameters being optimised are still very much determined by the designer, during the design phase, e.g. fuel burn, acceleration etc. If we switch to the assistive technology domain, we can consider a similar approach for powered wheelchairs, and indeed, many manufacturers do offer different drive profiles that change the mapping between the joystick input and maximum velocity (and acceleration curves), e.g. the Dynamic Controls DX/DX2 Power Wheelchair Control System^[2] and the Curtiss-Wright R-net Control System.^[3] These can be targeted towards specific scenarios, for example a more gentle indoor profile and a higher speed outdoor profile. Moreover, these parameters can indeed be tailored to the individual wheelchair user by a specially trained wheelchair technician. However, once these parameters have been configured, they tend to remain the same for the lifetime of the wheelchair, unless the user is experiencing any particular difficulties.

Again, in the automobile industry, over the last couple of decades, more complex Advanced Driver Assistance Systems (ADAS, such as adaptive cruise control and lane-keeping assistance) have also become prevalent. In turn, this has led to the need to personalise the assistance, because almost all these systems can be switched on/off and will therefore only enhance safety if drivers actually choose to use them [26]. More recently, there has been a shift from automated assistance towards a more cooperative approach between the driver and the vehicle, but this raises questions about authority and how to ensure the driver is able to regain or influence the control of the vehicle. One approach to dealing with such recurring problems is to take a human-system pattern approach, and balance the capabilities of the driver and vehicle automation through the concept of confidence horizons [27]. An alternative approach uses linear-quadratic differential games to identify the human behaviour and adapt the automation correspondingly, in a shared control manner. Regardless of the approach, the human factors element, with respect to how the user develops and maintains an evolving mental model of the assistive control system, is critical to maintaining safety and risk mitigation [28].

In the assistive technology domain of smart wheelchairs, we can think of some pseudo-equivalent functions to the ADAS and the more recent cooperative approaches used in highly automated vehicles. In the literature, there are some fairly prevalent examples of highly-automated wheelchair assistance functions, including: obstacle avoidance, wall following, door passing, kerb following, table-docking etc. [8]. In this case, the parameter space becomes much larger, as personal preferences are not only about the driving style, but can include more nuanced topics, like proxemics: how to approach other people and objects for interactions, which should consider notions such as personal space and angles of approach etc. [29], [30]. Moreover, these preferences are often context-dependent, so for an individual they are likely to change in different situations, for example, depending on whether or not the people who are interacting are familiar with one another. Since this parameter space is so large, it is difficult for a designer to explicitly consider all the variables that a user may wish to optimise. Furthermore, even if such a complete parameter space could be enumerated, it would then become a near impossible task for a technician to tune such a large array of parameters that, in all likelihood, most users would not be able to take the advantage of the full capabilities of such a system.

Therefore, we have taken a different approach: to try to learn a range of typical wheelchair driving styles and interaction preferences, from a wide range of people. Then we can cluster these into distinct profiles (which could be given notional labels – e.g. novice, timid, aggressive, sporty etc.). These user profiles can in turn be used as a starting point for transfer learning and subsequently the large parameter space can continue to be fine-tuned on the fly. To achieve this, we developed a reinforcement learning based probabilistic shared control (RL-PSC) framework [31], which built upon a generative adversarial imitation learning (GAIL) approach [32]. To date we have implemented and tested this approach in a simulated environment.

We found that whilst our earlier incarnation of probabilistic shared control (PSC) and our newly-proposed reinforcement learning-based probabilistic shared control (RL-PSC) were both successful in reducing the number of collisions, compared with no shared control (No-SC), RL-PSC was able to achieve this without the usual cost in terms of overall task completion time (c.f. Figure 4). We attribute this to the fact that the shared control is offering assistance that is more aligned to the user’s natural driving style, which means they no longer feel the need to “fight” against the controller when it subtly deviates from their input signals. This leads to a better understanding of the common workspace [33] and consequently an improved joint task performance. Perhaps more importantly, this in turn leads to better user acceptance of the shared control system, compared with the early linear blending approach that was only accepted in particular scenarios (e.g. by expert users when facing particularly high-workload situations; or by complete novice users [20]). Indeed, user acceptance and the previously mentioned “human-out-of-loop” problems were the starting points for the development of shared control systems, so this next step of personalisation was the natural evolution of such systems.

Figure 4:

Both probabilistic shared control (PSC) and reinforcement learning-based probabilistic shared control (RL-PSC) reduced the number of collisions, compared with no shared control (No SC). However, RL-PSC achieved this, without the usual cost in terms of overall task completion time (adapted with permission from [31], IEEE license number 5873121128373, license date Sep 20, 2024).

One challenge that remains to really harness the power of the learning approach is to collect a sufficiently large and diverse data set in the real world. There are currently limited smart wheelchairs in the world, with diverse configurations, although there are efforts to improve interoperability, for example, through modular approaches [7] and with standardised interfaces and data formats, such as those provided through ROS (Robot Operating System) [34]. Furthermore, access to incredibly useful location and video data can often be impeded through policies, such as the General Data Protection Regulation (GDPR),^[4] which, despite being very important and well-meaning when it comes to protecting our rights to privacy, place a high burden on innovators of emerging technologies [35]. Consequently, we are investigating to what extent data collected in simulated environments can be extrapolated for use in real-world applications [36] (the so-called sim-to-real problem [37]).

2.3 Understanding interactions with the environment

The robotics field has advanced rapidly in terms of sensing capabilities to understand the environment in which a robot is operating. Early sensors were fairly rudimentary and restricted to linear distance measures to objects via ultrasonic (sonar) or infrared (IR) emitter-receiver pairs [38]. Then laser scanners and LiDAR were able to give us a much more precise 3D representation, leading to simultaneous localisation and mapping (SLAM) as well as being able to incorporate notions of higher-level reasoning in path-planning [39], [40], whilst stereoscopic and RGB-D cameras are able to co-register colour and depth. When fused, this wealth of information has allowed the robotics field to develop satisfactory planning algorithms for autonomous robots to operate in real-world environments [41], [42].

The majority of these path planners will generate safe (and often optimal, with respect to pre-defined criteria) trajectories that will lead to a specified goal (location or pose). In doing so, these planners will automatically attempt to avoid collisions with obstacles. For mobile robotics, an obstacle is usually defined as anything that is not navigable space (floor/ground), and in the case of wheeled robots, including wheelchairs, this can also include the notion of negative obstacles [43]. Nevertheless, sometimes a detected object may not actually be an obstacle per se, but rather a target. For example, when a wheelchair user is navigating, they may wish to drive around a table that is in their way, or they may wish to dock to a table to work at it, or eat lunch. Therefore, we have worked on table detection, which can then be integrated into our planning algorithms to automatically line up and dock, or avoid, depending on the user input signals during the approach [44]. Similarly, a person may be treated as an obstacle that the user wishes to avoid, or a target with whom the user wishes to interact. In this case, higher level semantic information can be given to the planner by e.g. people detectors and people trackers [45], [46].

Another challenge is that the majority of robotic path planners neglect to take into account the effect of the robot itself on the environment [47]. Whilst this is fine for relatively static environments, many assistive robotic devices such as smart wheelchairs will need to be used in often densely populated areas. There has been extensive work modelling how crowds of people behave, to predict crowd flows and crowd dynamics [48], [49], which could theoretically be incorporated into a robot planner. However, there has been very little work that seeks to predict how the people in a crowd will react to the presence of a robot, and in turn the effect that has on the crowd dynamics. This is clearly important for any planner to incorporate, if it is going to enable a robot to move effectively in a crowded environment, without having to constantly re-plan in reaction to unexpected (yet potentially predictable) changes in crowd dynamics.

Therefore, we took a two-pronged approach to the challenge. First, we experimentally discovered how crowd dynamics are affected by different types of robot (including a smart wheelchair) [50]. This enabled us to modify state-of-the art crowd dynamics models [51], [52] to incorporate these newly observed behaviours and use this in our robotic path planner [31]. Second, we undertook some qualitative research with wheelchair users, to understand how they would like to be supported in navigating dynamic crowded environments [53]. The findings from this study not only reiterated that the driving assistance must be adapted to the individual user’s driving preference, but also highlighted the two interaction loops that need to be supported: (1) between the user and the smart wheelchair; and (2) between the combined user-smart wheelchair entity with the people in the crowd. This led to three design areas that should be considered when developing an equitable shared control system:

1. Empathy, embodiment and social acceptance

2. Situational awareness and adaptability

3. Selective information management.

2.4 Maintaining control authority

Although users can often retain authority in blended shared control schemes, they usually do this by correcting automation mistakes after they have been executed. By contrast, the H-mode [18] naturally supports a method for negotiating (and maintaining) control authority through the use of haptic interaction. In such a paradigm, the automation is blended with the user input at the physical interface, which allows for smooth transition in control authority: the user can choose to be guided by the automation, or if they don’t like what the automation is proposing, they can simply push harder against it [54].

Whilst haptic shared control can work well for physical interactions, there are other cases when this might not be so easy to implement, for example, in brain-machine interfaces (BMI). Nevertheless, we were inspired by the concept of haptic shared control, so were compelled to see if we could imitate it to some degree in a BMI context. First we decomposed a joint human-BMI-robot task into the different levels of interaction at which we can cooperate [55]. Then, again considering the operational level, we attempted to emulate haptic supports, by dynamically modulating the level of mental effort required to deliver a command, as opposed to varying the physical effort. In this way we demonstrated that BMI users could benefit from automation assistance, but in cases of conflict were now able to proactively override the assistance and assert their control authority, instead of having to correct errors after they occurred [56]. We have since built upon this concept to better support the initial BMI learning process, by dynamically introducing diminishing levels of bias into our classifiers [36].