With all the emerging new day-to-day uses of AI such as ChatGPT and other virtual assistants, it is only logical to imagine robots doing the same in our everyday lives. A robot butler, that sets the table before you eat can already be achieved quite easily, supposing that the table, dishes and everything else stays in the same spot as when the robot was programmed to do it, and nothing obstructs the robot’s ability to accomplish the task. But how close are we to robots that could accomplish this and other assisting tasks seamlessly? For example, if you wanted your robot butler to bring you that specific cup with a reindeer on it, that your grandma gave you for Christmas all those years ago, or your phone that was right next to you seconds ago? Not even you always know where it’s located, so how can your mechanized butler do it?
Local LLM
In the last few years, Large Language Models (LLMs) have become one of the central elements of research for Robotics applications. These models have been shown to be versatile tools that can be implemented to perform various crucial tasks necessary for an autonomous robotic system. However, the most powerful LLMs are bound to data centers, which in turn means that whatever function they are performing is being done over an active internet connection by a third-party service provider. A physical manipulator can suddenly lose its ability to perform its expected task if this connection is interrupted or lost. The third-party provider may change the parameters of their services or, in the worst-case scenario, stop providing this service at all.
LLMs can be used to provide valuable functions such as Natural Language Processing (NLP), which allows the creation of an intuitive interface between freely speaking users and structured language-using machines. High-Level Planning (HLP) is one such opportunity enabled by LLMs – by being able to generate a sequence of actions based on provided information about the environment and restrictions. This approach is currently being developed within the VCD4.1: “Cognitive Mobile Multi-Agent Platform” of EdgeAI project. Additionally, such models can perform autonomous reasoning, code generation, corrective evaluation and various other complex yet useful tasks.
Ideally, such abilities could be placed directly on a robotic platform with minimal footprint, but the practical reality is that edge applications for these models are impeded by various obstacles. Chief among them is related to GPUs which currently are the fastest inference platforms available. These GPUs eat up a lot of energy, a lot of space and even then, do not possess enough memory for mid-range models. However, even if these models must be deployed to locally run servers, that already is a major improvement in regard to autonomy, oversight and privacy. Our current implementation of the experimental system within VCD4.1 is able to rely on a consumer-grade GPU (with 12GB of available VRAM) as its LLM inference provider. This means that our system can potentially be contained to two elements: the physical manipulator and the computer unit.
Larger models can process larger prompts and as a result of both, more parameters and more context, can solve more complex tasks. For smaller models it would appear that “prompt chaining” is a fundamental method – smaller, more specialized prompts work together to break down and solve a task it otherwise could not. Finer control of the process comes at a cost of increase in the processing time, as each prompt adds its own overhead. As these chains grow more complex and more time has been invested in the development of the prompt, the system may become bound to a specific set of weights, as the performed experiments has shown that attempting to switch the system to newer and seemingly better weights does not guarantee improvements of results without adjustments to the prompts. This complicates long term maintainability of such systems, as ensuring robustness of an LLM based system is one of the main hurdles that developers of such systems must overcome.
A major direction of study within LLMs is grounding – how to inform the model of the current state of the external world so that its generation is relevant to the environment. This includes both passing collected information about the world into the model, which is usually textual (multi-modal models can input images, audio, etc.) and deciding on how to act upon the world. For this, the model needs to know what it (or the robot it is planning for) can do. While some have built methods that directly generate executable code or coordinates in the environment, a more robust approach and more appropriate for compact models may be the use of predefined actions that the model is condition on how to use as the basic building blocks of a plan.
Action primitives
Having a task plan available is half the battle, as it also needs to be executed. Comparing a robot system to a human, existing fully autonomously would mean being able to execute tasks with minimal prior knowledge about the environment. For example, given the task of setting the table, a person without prior knowledge about the location of the dishes would start looking for it. A stereotypical place one might find dishes is in the kitchen so that would be a good start to the search. This thought process can be implemented for any task in any environment. The usual approach for a robot system would be to preprogram such information in the robot’s world model. This can be a lot of manual labor, so it would not be feasible as this would be a requirement for each new environment the robot encounters. Instead, the robot could use the same thought process – searching by relation to the objective’s stereotypical information. But to utilize this type of approach, the robot requires to have context-adaptive functions, which, in this case, are called action primitives.
Action primitives are robot functions, created in a generalized form. Similarly to how humans know the act of picking up without knowing what will be picked up, an action primitive for picking would determine how to do this, depending on the object. This concept can be divided hierarchically in three stages, primitives being the simplest form, actions being sequential combinations of primitives and activities consisting of actions and primitives to fulfil specific tasks. Looking at the previously mentioned task of setting the table, a robot butler might have such action primitives as move_arm, move_body, control_gripper, search etc. Action primitives move_arm and control_gripper could be combined into actions such as pick and place. move_body and search, on the other hand, could be combined into the action locate_object. These actions could then be combined in the activity bring_object, that would allow the robot to autonomously search the environment, find the object of interest and bring it to the task giver. For setting a table, this activity could be called multiple times, or a new activity could be defined, where something similar is implemented.
Using action primitives gives great potential to a system, but the essence for high adaptability is how they are created. They too can be programmed, which, in turn, would mean that reaching the highest potential adaptability relies on how many edge cases the engineer creating them can think of. Another approach might be to use methods such as imitation or reinforcement learning. A system that could learn primitives by demonstration or even be left to find the correct execution by itself.
Combined usage potential
Robotics and edge computing are strongly interlinked – performing real-world manipulation requires fast processing of data for the quickest reaction to changes in the environment. To do this reliably, it has to be computed as close to the robot as possible. To execute more complex tasks, the resources available at the micro edge might not be enough, but being able to bring this computing power on-board is already a major step.
Action primitives provide two major enticing benefits for a resource constrained system. First, they provide modularity in both the design of an individual robot and of the system at large, as with such an approach one can think of each robot as a client that tells the planner what it can do. Second, they align with the prompt chaining approach of breaking more complex tasks down into parts, which appears to be necessary for using the less powerful, yet more memory compact models. Action primitives therefore allow not only new abilities to be introduced as they are developed, but whole robots with a full range of novel skills to be quickly integrated into the workflow.
An LLM-based High Level Planner is enticing from the standpoint of a general application such a model enables. A system adaptable to various environments and capable of interfacing directly with the user in natural language. By receiving an outline of a user’s request and an instruction of use from the robot, the HLP bridges the gap between these two domains and ensures a Human-Robot Interface that can cooperate to achieve ever increasingly harder tasks.
While inference for natural language and processing may be too costly to be placed on certain smaller chassis, with the action primitive approach it is possible for several specialized robots to rely on a single planning unit, saving both energy and compute resources. Thanks to Edge AI approaches this inference can be contained to mobile computing units meaning that the whole system can be kept together, easing deployment to new worksites.
LLMs have contributed to major developments in NLP and HLP, which enable advanced Human-Machine Interaction, while action primitives enable the creation of robots that can integrate with such a system to eventually create a general-purpose robot capable of reliably assisting people in solving the complex tasks of the rapidly evolving and ever more demanding world.
Blog signed by: EDI team