In 2017, a team from Australia won the Amazon Robotics Challenge to build an automated robot, including hardware and software, to successfully pick, and stow items in a warehouse. Amazon is one of the largest robotics companies in the world and requires the technology to quickly package and ship millions of items to customers from their global network of fulfilment centres. The commercial technologies to solve automated picking in unstructured environments are still being developed.

Using an in-house cartesian robot (Cartman) built for only $AU10,000, the team from the Australian Centre for Robotic Vision applied a novel few-shot learning algorithm to place first in the competition. The challenge combined object recognition, pose recognition, grasp planning, compliant manipulation, motion planning, task planning, task execution, error detection, and error recovery. The robots were scored by how many items they successfully picked, and stowed, in a fixed amount of time. They were also challenged by being given 16 unseen items just 45 minutes before the competition began.

Cartman can move along three axes at right angles to each other, like a gantry crane, and featured a rotating gripper that allowed the robot to pick up items using either suction or a simple two-finger grip. Cartman’s vision system was the result of hours of training data, and training time, but the team also had to create a robust vision system to cope with the unseen items. One feature of the vision system was that it worked off a very small amount of hand annotated training data. Cartman needed just seven images of each unseen item to be able to reliably detect them.

The robot was only unpacked and reassembled out of suitcases a few days before the event, and broke a wrist during the competition, which had to be quickly re-engineered and a replacement part 3D printed.