Smart Item Detection Skill

Detection of different objects.

Overview

Similar to the Smart Vacuum Picking Skill, the Smart Item Detection Skill can also be used for vacuum gripper applications, but it differs fundamentally in terms of its structure and thus its application. This is a segmentation skill that determines instances of objects. Based on these segments, pick points are determined and displayed in the user interface. Through the segment recognition, an initial recognition of the position and orientation is possible. Longitudinal and transversal axes of the objects are recognized and can be used for process design in the robot program. The red axis of each possible pick point represents the x-axis of the object. With the help of this object coordinate system, the removal to be performed can be influenced by introducing offsets along and perpendicular to this axis of the gripping point. Besides the red one, there is also the green y-axis and a blue axis in z-direction. The center point is typically shown as a green circle, only with the next pick point displayed in light blue.

Thus, this skill as well as the other Vacuum Skill (Smart Vacuum Picking Skill) are ideal for warehouse applications. By identifying the different instances of the objects and returning the respective dimensions, it is possible to use these in the separation process in a more intelligent way.

Boxes

Cylinders

Random

Skill Result Information

Position

Position of the grasp point relative to the robot’s coordinate system (in m)

Orientation

Rotation of the grasp point relative to the robot’s coordinate system (in m)

Dimensions

Object dimensions, aligned with the pick pose axes

Specifications

Conditions

Camera mount:

Dynamic
Static

Camera distance:

35 – 45 cm

Parts dimensions:

5 – 15 cm

Parts material:

dull - low reflection

Specs

Avg. recognition time:

< 1 seconds

Supported grippers:

Vacuum
Magnet
Single contact point

Features

Measures
Collision detection
Multiple instance recognition
Generalized to a variety of object types
Picking from bulk

Parameter Example

To ensure accurate identification of various types of objects, the skill parameters can be easily adjusted to fit your specific needs. Here are some recommendations to help you find the perfect parameters for your application. For your convenience, we've only included descriptions of parameters that differ from the default settings.

Used Objects: small cardboard boxes (70x50x50mm), matt surface
Camera Distance: 550mm
Camera Mount: 30° angle
Skill Parameters:
- Pose Estimation: Bounding rectangle (due to the shape of the objects)
- other parameters: default values

Technical Parameter Description

Parameter

Name

Minimum coverage

Parameter

min_coverage

Type

float

Description

This parameter is only useful in combination with the bounding rectangle/ellipse/plane fitting methods for pose estimation. For the rectangle and ellipse fitting it represents the intersection over union between the detected mask and the expected shape. For the plane-fitting method it sets a limit on the minimum fraction of the detected mask covered by the fitted plane.

Name

Pose estimation method

Parameter

pose_method

Type

string

Description

Method for the estimation of pick poses for each detected mask

Pose Estimation Types

Type

Description

pca

Principal components use the orientation and the centroid of the mask as position of the pick pose

rectangle

Bounding rectangles, best used for planar rectangular items or cuboids when observed directly from above

ellipse

The ellipse fit, best for planar elliptic and spherical items

plane

Plane fitting selects the plane with the smallest angle towards the camera axis, useful for selecting a pick pose on the plane facing the camera when an item is observed from an angle. Works best for cuboids.

Name

Minimum instance quality

Parameter

min_score

Type

float

Description

Used to filter out detections with poor confidence. The impact of this parameter can differ between AI models.

Name

Detection class

Parameter

det_cls

Type

int

Description

Used to select a specific class ID for detections, detections with other class IDs are rejected. Valid IDs differ between models. The default value of -1 enables all classes.

Name

Maximum allowed IoU

Parameter

max_iou

Type

float

Description

Maximum allowed intersection over union between two detection masks, allows to resolve situations where multiple detections cover the same item. In conflicts, the mask with the highest score is selected.

Name

Offset for pick point

Parameter

offset_x, offset_y

Type

float

Description

Allows to shift the pick position. The position can only be shifted within the x-y plane of the pick pose's coordinate frame. The x-axis is depicted as a red line and the y-axis as green.

Name

Distance tolerance

Parameter

distance_tolerance

Type

float

Description

Only used by the plane-fitting method for pose estimation, refers to the distance tolerance for inliers. All points that are closer to the fitted plane than this parameter are considered to be on the plane.

Name

Minimum inlier fraction

Parameter

min_fraction

Type

float

Description

Only used by the plane-fitting method for pose estimation, the minimum fraction of inliers a fitted plane must provide to be considered valid.

Name

Optimize gripper orientation

Parameter

orientation_optimization

Type

boolean

Description

Enables the optimization of the gripper orientation. If enabled the pick pose might be rotated by 180° around the approach axis to minimize the robot rotation between the capture and pick pose.

Name

Enforce GPU usage

Parameter

force_gpu

Type

boolean

Description

Enforces GPU usage for this skill (could disable GPU usage for other skills or cause performance problems.)

Detections

Type

pose (Transformation)

Description

Pick pose for the gripper

Type

cls (int)

Description

ID of the predicted class

Type

dimensions (list)

Description

Object dimensions, aligned with the pick pose axes

Type

coverage (float)

Description

Amount of detected shape covered by mask, changes depending on pose estimation method

Type

optimization_angle (float)

Description

Only present if the parameter optimization_orientation is set to true. The parameter represents the rotation angle in radians that was applied to the original pick pose during the orientation-optimization step.

{
  "detections":[
    {
      "pose":{
        "x":0.03015468326308962,
        "y":-0.4181476095144873,
        "z":0.3146591919146398,
        "rx":0.5325534651400073,
        "ry":0.003906282923994998,
        "rz":0.006591706153745695
      },
      "cls":2,
      "dimensions":[
        0.09554422282330793,
        0.03654570469274869,
        0.050999999046325684
      ],
      "coverage":0.8020935960591133,
      "optimization_angle":0.0
    },
    {
      "pose":{
        "x":-0.2244455008984334,
        "y":-0.17618821237008492,
        "z":0.47046944385142553,
        "rx":0.5094861206521137,
        "ry":0.18745539973711622,
        "rz":-0.6672058153990215
      },
      "cls":2,
      "dimensions":[
        0.07480855861880528,
        0.0324312542979934,
        0.020000031123772077
      ],
      "coverage":0.6429383670684558,
      "optimization_angle":0.0
    }
  ]
}

The following code snippet demonstrates how you can access and use the detection data in a Universal Robot program.

Get the dimension of the first / best pick point along the x- and y-axis:

dim_x := rb_get_attr_float("detections[0].dimensions[0]")
dim_y := rb_get_attr_float("detections[0].dimensions[1]")

Get the number of detected pick points:

n_dets := rb_get_attr_int("detections.length()")

PreviousSmart Vacuum Picking Skill NextSmart Parallel Picking Skill

Last updated 1 year ago