KLT & Load Carrier Detection Skill

This skill enables the detection of predefined containers in the camera's field of view. The focus is on the detection of one KLT, but other types of small load carriers are also possible. The prerequisite for this skill is that the edge of the container can be detected by the camera. With this skill, various information is also determined and displayed to the user on the left side of the screen, which helps in the creation of a robot program. The skill offers two ways of providing position information. One is that the container position and orientation (center of the container) can be displayed, which can be used for a dynamic workspace definition. In addition, intelligent and dynamic processes can be realized with this approach. For example, in the automotive industry, parts often arrive at the gripping station in KLT on a conveyor belt. This does not ensure that the container is in the exact same position. To compensate for this uncertainty, this skill can be used to optimally measure the ROI (region of interest).

Another way is to display a transformation with respect to a reference position or container. By a matrix multiplication of the reference position with a teach-in position of the bin's handle, it is possible to calculate a handle point in relation to the bin. With this approach, boxes can be gripped by a conveyor and stacked on a pallet. This is a classic example of when the empty bins need to be sent for cleaning.

Euro container

Small load carrier

Skill Result Information

Position	Position of the detected bin position in the robot's coordinate frame
Orientation	Rotation of the detected bin position in the robot's coordinate frame
Item fraction	Estimated filling level
Bin empty	Decision if a bin is empty or not; based on the previous parameter and an adjustable threshold
Changed reference pose	After saving a reference position of the bin, the skill also calculates the distance and rotations between the saved position and the new detection.
Distance to reference	Translational deviation from the reference pose
Rotation relative to reference	Rotational deviation from the reference pose

Specifications

Conditions

Camera Mount:

• Dynamic

• Static

Supported type:

• KLT

• Euro contianer

Specs

Avg. recognition rime:

< 1 seconds

Supported grippers:

• Parallel

• Vacuum

Features

• KLT / load carriers position detection

• Rotated position recognition

• Single instance recognition

Parameter Example

To ensure accurate identification of various types of load carriers, the skill parameters can be easily adjusted to fit your specific needs. Here are some recommendations to help you find the perfect parameters for your application. For your convenience, we've only included descriptions of parameters that differ from the default settings.

Euro container

KLT & Load Carrier Detection Skill for standard KLT

• Used Object: Industry-standard load carrier (398x297x171 mm)

• Camera Distance: 690mm

• Camera Mount: 30° angle

• Skill Parameters:

  • Bin dimensions: according to size of load carrier (0.398x0.297x0.171m). It's recommended to start with the longest edge as your x-axis

  • Edge width: according to size of load carrier (0.015)

  • Reference Pose: no change

  • Other parameters: default values

Small Box

KLT & Load Carrier Detection Skill for small boxes

• Used Object: Small load carrier (200x150x120 mm)

• Camera Distance: 450mm

• Camera Mount: 30° angle

• Skill Parameters:

  • Bin dimensions: according to size of load carrier (0.2x0.15x0.12m). It's recommended to start with the longest edge

  • Edge width: according to size of load carrier (0.015)

  • Reference Pose: no change

  • Other parameters: default values

Technical Parameter Description

Parameter

Bin settings

Name

Bin dimensions

Parameter

bin_x, bin_y, bin_z

Type

int

Description

The dimensions of the bin.

Name

Edge width

Parameter

edge_width

Type

float

Description

The width of the edge of the bin. If the edge does not have the same width on all four sides, the smallest width should be used. However, some bins have small gaps on the edge. In that case, better results are obtained if these gaps are ignored for the determination of the edge width.

Fit settings

Name

Minimum quality for bin detections

Parameter

min_score

Type

float

Description

Minimum prediction quality for pixels on a bin edge. This score can be used to tune the detection of bin edges. False detections can be suppressed by increasing the value, while decreasing the minimum quality can increase the detected fraction of the edge for some bins.

Name

Downsampling size

Parameter

downsampling

Type

float

Description

Defines the minimum distance between 3D points for the skill. All points within a cube with the specified size are replaced by one point. Smaller values for the downsampling size lead to better resolutions while larger values lead to faster runtimes. Best results are achieved if the specified edge width is a multiple of the downsampling size. Check the documentation for some example measurements.

Name

Maximum item coverage

Parameter

item_threshold

Type

float

Description

Maximum fraction of pixels containing items for empty bins. This limit is used to tune the decision if a detected bin is empty or not. The Skill determines the fraction of pixels within a detected bin that show items within the skill (purple shaded region in image). If this fraction is below the specified limit (maximum item coverage) the detected bin is marked as empty. Note that, while this feature works for a large variety of items, it might not work for all possible items.

Name

Maximum distance to reference pose

Parameter

distance_threshold

Type

float

Description

Maximum allowed distance to the specified reference position. The bin-detection skill was designed to be as robust as possible allowing ,e.g., the detection of a bin also if more than a single bin is present in the view of the camera or if the bin is not fully visible. However, this robustness could also lead to a wrong detection (e.g., not the desired bin, or a wrong orientation). To reject undesired results the maximum distance to a reference position can be used.

Name

Maximum rotation relative to reference pose

Parameter

rotation_threshold

Type

float

Description

Maximum allowed rotation relative to the reference pose.

Reference pose

Name

Change reference pose

Parameter

change_reference

Type

string

Description

Selects the method to change the reference pose.

Name

Position of reference bin

Parameter

reference_pose_x, reference_pose_y, reference_pose_z

Type

float

Description

Position of the reference bin.

Name

Orientation of reference bin

Parameter

reference_pose_rx, reference_pose_ry, reference_pose_rz

Type

float

Description

Orientation of the reference bin.

Reference Pose Types

Type	Description
no_change	The reference pose remains unchanged.
fit_result	The result of the next execution of the skill is used as reference pose for future executions. If this option is selected, the distance and rotation relative to a previous reference pose are not checked. Use this option to turn off the comparison of the detection result with the reference pose.
manual	The reference pose is provided as skill parameter. Caution: The provided reference pose has to be in the camera's coordinate frame.

Expert

Name

Enforce GPU usage

Parameter

force_gpu

Type

boolean

Description

Enforces GPU usage for this skill. Enforcing GPU usage for one skill can disable GPU usage for other skills.

Detections

Detection Data

Type

pose (Transformation)

Description

The pose of the bin.

Type

bin_empty (bool)

Description

True if the bin is empty.

Type

item_fraction (float)

Description

Fraction of pixels containing items.

Type

reference_trafo (Transformation)

Description

Transformation relative to the reference pose.

Detection Example

Reference Pose Types: fit_result

{
   "detections":[
      {
         "pose":{
            "x":-0.056031276694279916,
            "y":-0.44327393228068174,
            "z":0.5726675775572347,
            "rx":-0.12817415969395238,
            "ry":0.9263117430484452,
            "rz":-2.9538682613260026
         },
         "bin_pose":{
            "x":-0.06337235849379225,
            "y":-0.3978340441934568,
            "z":0.5072357421203973,
            "rx":0.06617629091108139,
            "ry":-2.918019947534546,
            "rz":-0.9150699708718468
         },
         "bin_empty":true,
         "item_fraction":0.0,
         "reference_trafo":{
            "x":0.0,
            "y":0.0,
            "z":0.0,
            "rx":0.0,
            "ry":0.0,
            "rz":0.0
         }
      }
   ]
}

Reference Pose Types: no_change

{
   "detections":[
      {
         "pose":{
            "x":-0.07200788946578035,
            "y":-0.4221961998324571,
            "z":0.5629487695216557,
            "rx":0.04372156978230481,
            "ry":0.8291113470163821,
            "rz":-2.990767713921765
         },
         "bin_pose":{
            "x":-0.07064287213105912,
            "y":-0.3809863358438283,
            "z":0.4943930707234182,
            "rx":-0.058720708358277335,
            "ry":2.9997349483453495,
            "rz":0.831597275889207
         },
         "bin_empty":true,
         "item_fraction":0.0,
         "reference_trafo":{
            "x":0.005281489005689269,
            "y":0.006600885911974622,
            "z":0.02074063723356856,
            "rx":0.06900890793539619,
            "ry":-0.10688376692974187,
            "rz":-0.024829240766247828
         }
      }
   ]
}

UR Programming Example

The following code snippet shows how you can use the load carrier skill to detect boxes in the camera's FOV and adjust the workspace accordingly.

 'Select the Bin Detection Skill and 'World' workspace for detection'
     robobrain Pick Pose
       If rb_has_result()
         'Overwrite the workspace configuration'
         bin_pose≔rb_get_attr_pose("detections[0].bin_pose")
         rb_set_ws_config("pos_x", bin_pose[0])
         rb_set_ws_config("pos_y", bin_pose[1])
         rb_set_ws_config("pos_z", bin_pose[2])
         rb_set_ws_config("rot_x", bin_pose[3])
         rb_set_ws_config("rot_y", bin_pose[4])
         rb_set_ws_config("rot_z", bin_pose[5])
         'Select your item picking skill and a random workspace of the right size here
         robobrain Pick Pose