lfd package

lfd.alignment module

The alignment.py module contains a variety of methods and classes used to align Demonstrations captured during the LfD process.

exception lfd.alignment.AlignmentException(expression, message)

Base class for exceptions in this module.

class lfd.alignment.DemonstrationAligner(demonstrations, vectorize_func)

Demonstration aligning class to align demonstrations, ensuring uniform constraint transitions across all demosntrations.

Attributes:

demonstrations : list

List of demonstraions to align.

vectorize_func : func

A function used to vectorize the dictionary data of a demonstrations observations.

align()

Alignment is performed using the FastDTW algorithm. It first separates trajectories that are constraint annotated, and aligns those first. Secondly, arbitrarily uses one of those trajectories as a reference against which to align all the remainin trajectories captured during demonstrations. If there are no

Returns:

self.demonstrations : tuple

Returns the demonstrations each having a new parameter called aligned_observations.

lfd.alignment.vectorize_demonstration(demonstration)

Vectorizes a demonstration’s observations through the union of the robot’s postion and robot’s joints.

Parameters:

demonstration : Demonstration

Demonstraions to vectorize.

Returns:

vectors : list

List of observation vectors.

lfd.analysis module

The analysis.py module contains a variety of methods and classes used to analyze specific structures and objects in the Cairo LfD ecosystem. This includes KeyframeGraphs, MotionPlans, and Constraints.

class lfd.analysis.ConstraintAnalyzer(environment)

Constraint analysis class to evaluate observations for constraints.

Attributes:

enivornment : Environment

Environment object for the current LFD environment.

applied_constraint_evaluator(observations)

This function evaluates observations for constraints that were triggered during the demonstration. It will label a demonstration’s entire list of observations with the constraints that were triggered and wheter or not they are still applicable.

New constraints are those where the triggered constraints are different from the previously applied constraints:

triggered = observation.get_triggered_constraint_data() new = list(set(triggered)-set(prev))

The evaluated constraints are those that are still valid from the previous observation’s applied constraints.

evaluated = self.evaluate(constraint_ids=prev, observation=observation)

The current observation’s applied constraints are the union of the evaluted constraints and new constraints.

applied = list(set(evaluated).union(set(new)))

Parameters:

observations : int

List of observations to be evaluated for constraints.

evaluate(constraint_ids, observation)

This function evaluates an observation for all the constraints in the list constraint_ids. It depends on being able to access the constraint objects from the self.environment object. Every constraint object should have an ‘evaluate()’’ function that takes in the environment and the observation.

Parameters:

constraint_ids : list

List of constraint id’s to evalaute.

observation : Observation

The observation to evaluate for the constraints.

Returns:

valid_constraints : list

Returns the list of valid contraints evaluated for the observation.

class lfd.analysis.DemonstrationKeyframeLabeler(aligned_demonstrations, constraint_transitions)

Keyframe labeling class.

This class depends on constraint aligned demosntrations. This means that all demosntrations should have the same sequence of constraint transitions. Without such alignment, the class functions will fail ungracefully.

Attributes:

demonstrations : list

List of demonstraionts. These must be constraint aligned

constraint_transitions : list

A 2D list containing the set of constraint transitions that are applicable to all of the aligned demosntrations.

label_demonstrations(divisor=20, keyframe_window_size=8)

This function serves to take each demonstration and create a list of observations labeled with keyframe_ids. For each demonstation, the function gets the observation grouping and then iteratively calls _get_labeled_group() from which it extends a list using the function’s returned labeled_group. This list becomes the labeled_observations list of observation obejcts assigned to the demonstration object.

Parameters:

group_divisor : int

The divisor used by the _get_keyframe_count_per_group function

keyframe_window_size: string

The size of the window of each split used to capture a subset of data for the keyframe.

Returns:

demonstrations : tuple

Returns the classes demonstrations attribute each of which will have a new parameter assigned with a list called ‘labeled_observations’.

class lfd.analysis.KeyframeGraphAnalyzer(task_graph, sawyer_moveit_interface, observation_vectorizor)

Class with methods to support the analysis of a KeyframeGraph.

Attributes:

graph : KeyframeGraph

KeyframeGraph class obejct on which to perform analysis

interface : robot_interface.moveit_interface.SawyerMoveitInterface

Sawyer Robot interface.

vectorizer : function

Function that vectorizes Observation object.

cull_keyframes(threshold=-10000)

Culls consecutive keyframes sequentially until the one such keyframe’s observations has an average LL is below a threshold as scored by the current keyframes model.

Parameters:

threshold : int

The threshold average log likilhood.

evaluate_keyframe_occlusion(keyframe_observations)

Evaluates a given list of keyframe observations for occlusion using the interface’s check_point_validity() function.

Parameters:

keyframe_observations : list

List of keyframe observations to evaluate. Expects Observation objects.

Returns:

(free_observations, occluded_observations) : tuple

A tuple containing a list of free observations as the first element and a list of occlueded observations as the second element.

max_mean_ll(model, observation)

Given a set of observations, generate the mean log likilhood and max log liklihood of the set scored by a given model.

Parameters:

priod_model : object

An modeling.models class object that has a score_samples() method.

observations : list

List of keyframe observations to evaluate their score.

Returns:

(max_ll, mean_ll) : tuple

Returns the maximum log liklihood of scored samples as well as the average of all the sample scores.

class lfd.analysis.MotionPlanAnalyzer(environment)

Class with methods to support the analysis of a motion plan.

Attributes:

enivornment : Environment

Environment object for the current LFD environment.

evaluate(constraint_ids, observation)

This function evaluates an observation for all the constraints in the list constraint_ids. It depends on being able to access the constraint objects from the self.environment object. Every constraint object should have an ‘evaluate()’’ function that takes in the environment and the observation.

Parameters:

constraint_ids : list

List of constraint id’s to evalaute.

observation : Observation

The observation to evaluate for the constraints.

Returns:

valid_constraints : list

Returns the list of valid contraints evaluated for the observation.

evaluate_plan(constraint_ids, plan_observations)

Evaluates plan observations to ensure that the constraints specified in the list of constraint ids is valid throughout the entire plan. If one observation in the plan violates the constraints, the plan is invalidated.

Parameters:

constraint_ids : list

List of constraint id’s to evalaute.

plan_observations : lsit

The observation to evaluate for the constraints.

Returns:

: boolean

Returns true if the given constraints are valid throughout the entirety of the plan, false otherwise.

lfd.analysis.get_observation_joint_vector(observation)

Vectorizes a Observation object by obtaining joint configuration data.

Parameters:

observation : Observation

Observation object to vectorize.

Returns:

: list

Returns list of joint configuration data. Formatting is dependent on robot DOF etc,.

lfd.analysis.get_observation_pose_vector(observation)

Vectorizes a Observation object by obtaining pose data.

Parameters:

observation : Observation

Observation object to vectorize.

Returns:

: list

Returns list of pose data as the following [x, y, z, q_x, q_y, q_z, q_w]

lfd.constraints module

The constraints.py module contains a classes that encapsulate predicate classifiers used to evaluate binary value conceptual constraints.

class lfd.constraints.HeightConstraint(constraint_id, item_id, button, reference_height, threshold_distance)

HeightConstraint class to evaluate the height predicate classifier assigned to a given item.

height() returns true if object distance from reference_height is greater than threshold distnace.

A positive threshold distance mean height above

A negative threshold distance means height below.

Attributes:

id : int

Id of the constraint as defined in the config.json file.

item_id : int

Id of the item on which the constraint can be applied.

reference_height : int

The reference or starting height to compare an objects height distance against the threshold_distance.

threshold_distance : int

The distance from reference (positive: above; negative; below) to compare an object’s distance from reference.

button : string

String of a button for the intera_interface.Navigator().get_button_state(self.button) function to check the trigger.

check_trigger()

This function evaluates whether the constrain has been triggered. In this case, this class’s trigger uses the cuff buttons of Sawyer.

Returns:

: int

Boolean value of trigger result.

evaluate(environment, observation)

This function evaluates an observation for the assigned constraint of the class. It differentiates betweeen Sawyer (end-effector) and general items (blocks etc,.).

Parameters:

environment : Environment

The Environment object containing the current demonstrations environment (SawyerRobot, Items, Constraints) and helper methods.

observation : Observation

The observation to evaluate for the constraint.

Returns:

: int

Boolean value of constraint evaluation for the height constraint.

class lfd.constraints.UprightConstraint(constraint_id, item_id, button, threshold_angle, axis)

Upright Constraint class to evaluate the upright predicate classifier assigned to a given item.

upright() returns true if object distance is within a threshold angle from its defined upright orientation, pivoting around a given axis.

Attributes:

id : int

Id of the constraint as defined in the config.json file.

item_id : int

Id of the item on which the constraint can be applied.

threshold_angle : int

The angle within which the assigned item’s (from item_id) current orientation must be compared with its defined upright position.

axis : int

The axis from which angle of deviation is calculated.

button : string

String of a button for the intera_interface.Navigator().get_button_state(self.button) function to check the trigger.

check_trigger()

This function evaluates whether the constraint has been triggered. In this case, this class’s trigger uses the cuff buttons of Sawyer.

Returns:

: int

Boolean value of trigger result.

evaluate(environment, observation)

This function evaluates an observation for the assigned constraint of the class. It differentiates betweeen Sawyer (end-effector) and general items (blocks etc,.).

Parameters:

environment : Environment

The Environment object containing the current demonstrations environment (SawyerRobot, Items, Constraints) and helper methods.

observation : Observation

The observation to evaluate for the constraint.

Returns:

: int

Boolean value of constraint evaluation for the associate constraint and item.

lfd.data_io module

The data_io.py module contains a classes that contain methods for importing and exporting data.

class lfd.data_io.DataExporter

Data exporint class with a variety of methods to support importing trajectory/observation data from csv, json etc.

export_to_json(path, data)

Exports dictioanry data to a .json file.

Parameters:

path : string

Path of directory to save .json file.

dict_data : dict/JSON serializable type

Data to be serialized to json.

class lfd.data_io.DataImporter

Data importing class with a variety of methods to support importing trajectory/observation data from csv, json etc.

import_csv_to_dict(path)

Import trajectories stored as .csv files into a Ordered Dictionary. In this case, each file represents a single trajectory/demonstration.

This method expects a directory path and will automatically import all files with an approporaite .csv file signature.

Parameters:

path : string

Path of directory containing the .csv files.

Returns:

entries : OrderedDict

Dictionary of trajectories. Trajectories themselves are dictionaries of observations (rows of the .csv file).

import_csv_to_list(path, exclude_header=True)

Import trajectories stored as .csv files into a list of trajectories. In this case, each file represents a single trajectory/demonstration.

This method expects a directory path and will automatically import all files with an approporaite .csv file signature.

Parameters:

path : string

Path of directory containing the .csv files.

exclude_header : bool

If true, first entry for each trajectory will be removed i.e. ignores the header.

Returns:

entries : list

List of trajectories. Trajectories themselves are lists of observations (rows of the .csv file).

import_json_to_dict(path)

Import trajectories stored as .json files into a Ordered Dictionary. In this case, each file represents a single trajectory/demonstration.

This method expects a directory path and will automatically import all files with an approporaite .json file signature.

Parameters:

path : string

Path of directory containing the .json files.

Returns:

entries : OrderedDict

Dictionary of trajectories. Trajectories themselves are dictionaries of observations.

load_json_file(path)

Import JSON file as a Python dictionary.

Parameters:

path : string

Path of directory containing the .csv files.

Returns:

entries : dict

Dictionary representation of the JSON file.

load_json_files(path)

Import JSON files as a Python dictionary from .json files in the directory signified by the path..

Parameters:

path : string

Path of directory containing the ..json files.

Returns:

entries : dict

Dictionary representation of the JSON file.

lfd.environment module

The enviornment.py module contains core data container classes and retrieval methods. The core classes are the Environment, Demonstration, and Obsevation used throughout Cairo LfD’s codebase.

class lfd.environment.Demonstration(observations, aligned_observation=None, labeled_observations=None)

Demonstration object to contain list of osbervations and various methods to perform on those observations.

Attributes:

observations : list

List of Observation objects representing raw observations from demonstration. (Requried)

aligned_observation : SawyerRobot

List of Observation objects representing DTW aligned observations.

labeled_observations : list

List of Observation objects representing keyframe labeled observations.

get_applied_constraint_order()

Returns the applied constraint order of a Demonstration’s aligned observation list.

Returns:

constraint_order: list

List of list where each element is ordered by the sequence of the applied constraints and represents the set of constraints applied.

get_observation_by_index(idx)

Returns an raw observation by its index.

Parameters:

idx : int

Index integer

Returns:

: Observation

The retrieved observation object.

class lfd.environment.Environment(items, robot, constraints)

Environment container class that houses various objects (items, robots, constraints) relevant to conducting LfD experimentation.

Attributes:

items : list

List of environment items/objects i.e. blocks, spatulasm, cups etc,.. of the AbstractItem class.

robot : SawyerRobot

AbstractItem extended class object representing the robot.

constraints : list

List of constrain class objects representing the available constraints for the Demonstration.

check_constraint_triggers()

Checks all constraints for their trigger. A triggered constraint might be a button press on Sawyer’s cuff or a natural language dicated constraint.

Returns:

triggered_cosntraints: list

List of the id’s of all the constraints currently triggered

get_constraint_by_id(constraint_id)

Retrieves a constraint from the Environment by it’s id.

Parameters:

iconstraint_id : int

The id of the constraint to retrieve.

Returns:

: Constraint class

Constraint class for the given id.

get_item_info()

Retrieves the Environment’s items information.

Returns:

entries : list

List of dictionaries for each of the Environment’s items (exluding the robot item)

get_item_states()

Retrieves the Environment’s items states.

Returns:

entries : list

List of dictionaries for each of the Environment’s items (exluding the robot item)

get_robot_info()

Retrieves the robot item’s configuration information.

Returns:

entries : dict

Dictionary representing robot’s information. See SawyerRobot class.

get_robot_state()

Retrieves the robot item’s current state

Returns:

entries : dict

Dictionary representing robot’s state. See SawyerRobot class.

class lfd.environment.Observation(observation_data)

Observation object to contain raw dictionary data for an observation and retrieval methods for that data.

Attributes:

data : dict

Dictionary of raw data collecting item, robot and constraint states.

get_applied_constraint_data()

Get the applied constraints of an object.

Returns:

: list

List of constraint id’s.

get_item_data(item_id)

Gets an items information based on its id

Parameters:

observation_data : dict

Dictionary of raw data collecting item, robot and constraint states.

Returns:

: float

Integer timestamp in milliseconds representing time from epoch.

get_joint_list()

Get the joint data of the observation as a list of numerical values.

Returns:

: list

list of numerical joint values for the given robot.

get_keyframe_info()

send back tuple of keyframe num and type

Returns:

: tuple

(keyframe_id, keyframe_type)

get_pose_list()

Get the pose data of the observation as a list of numerical values.

Returns:

: list

combined data from position and orientation

get_pose_msg()

Get pose data and return a pose msg type

get_robot_data()

Get the observation’s robot data.

Returns:

: dictionary

dictionary of robot data

get_timestamp()

Get’s osbervations timestamp

Returns:

: float

Integer timestamp in milliseconds representing time from epoch.

get_triggered_constraint_data()

Get the triggered constraints of an object.

Returns:

: list

List of constraint id’s.

classmethod init_samples(pose, orientation, joints)

Parameters:

pose : array of pose data

orientation: array of orientation data

lfd.environment.import_configuration(filepath)

Wrapper function around json.load() to import a config.json file used to inform the Environment object.

lfd.items module

The items.py module contains container classes for items uses in the Cario LfD ecosystem. These could include robots, constraints, and environment objects such as a cup etc,.

class lfd.items.AbstractItem

Abstract Base class for represent items in an Environment.

get_info()

Abstract method to get the Item’s info.

get_state()

Abstract method to get the Item’s state.

class lfd.items.ConstraintFactory(constraint_configs)

Factory class that builds LFD constraints. These items are defined in the config.json file. The class field in the configuration determines which constraint class to use.

Attributes:

configs : list

List of configuration dictionaries.

classes : dict

Dictionary with values as uninitialized class references to constraint classes i.e. HeightConstraint

generate_constraints()

Build the constraint objects defined in the configuration dictionaries of self.configs.

Returns:

robots : list

List of constraint objects.

class lfd.items.RobotFactory(robot_configs)

Factory class that builds robot items. These items are defined in the config.json file. The class field in the configuration determines which AbstractItem robot class to use.

Attributes:

configs : list

List of configuration dictionaries.

classes : dict

Dictionary with values as uninitialized class references i.e. SawyerRobot

generate_robots()

Build the robots defined in the configuration dictionaries of self.configs.

Returns:

robots : list

List of AbstractItem robot objects.

class lfd.items.SawyerRobot(robot_id, upright_pose)

Class representing the Saywer robot in the LFD Environment object.

Attributes:

id : int

Id of robot assigned in the config.json configuration files.

upright_pose : dict

Dictionary with position and orientation fields indicating the upright orientation of the Sawyer end-effector.

_limb : object

Intera SDK class object that provides controlling functionality of the Sawyer Robot.

_cuff : object

Intera SDK class object that provides controlling interface of the cuff bottons of Sawyer robot.

_navigator : object

Intera SDK class object that provides controlling functionality of the button/wheel interface on the Sawer Robot.

_gripper : object

Intera SDK class object that provides controlling functionalirty of the Sawyer Robot gripper.

get_info()

Get’s the robot item’s information.

Returns:

state : dict

The info of the robot item

get_state()

Get’s the current state of the robot.

Returns:

state : dict

The state of the robot

lfd.processing module

The processing.py module contains classes and methods for manipulating data/objects into difference forms.

class lfd.processing.SawyerSampleConverter(interface)

Converts raw samples generated from models into Observation objects.

Attributes:

interface : object

SawyerMoveitInterface to help run forward kinematics.

convert(sample, run_fk=False, normalize_quaternion=False)

Converts raw samples generated from models into Observation objects.

Parameters:

sample : list

Raw sample to convert. Either joint configuration or pose [x,y,z,x,y,z,w] as a list.

run_fk : bool

Flag indicating whether to run foward kinematrics or not.

normalize_quaternion : bool

Flag indicating whether to normalize the values of the quaternion entries.

Returns:

obsv : lfd.environment.Observation

Observation object constructed from the converted sample.

lfd.processing.convert_data_to_pose(position, orientation)

Converts raw position and orientation data to a ROS message Pose object.

Parameters:

position : list

List of numerical values representing x,y,z position.

orientation : list

List of numerical values representing the x,y,z,w values of a quaternion.

normalize_quaternion : bool

Flag indicating whether to normalize the values of the quaternion entries.

Returns:

pose: geometry_msgs.msgs.Pose

The Pose object

lfd.record module

The record.py module contains classes and methods for recording data during live demonstrations.

class lfd.record.SawyerRecorder(rate)

Class to record state data from the ReThink Robotics Sawyer robot for capturing demonstrated data for LfD experimentation.

Attributes:

_raw_rate : int

The rate at which to capture state data.

_rate : int

The ROScore rate.

_start_time : float

Current time in ROScore.

_done : bool

Termination flag.

done()

Return whether or not recording is done.

Returns : bool

The _done attribute.

record_demonstrations(environment)

Records the current joint positions to a csv file if outputFilename was provided at construction this function will record the latest set of joint angles in a csv format.

If a file exists, the function will overwrite existing file.

Parameters:

environment: Environment

The Environment object of the current LfD experiment.

Returns:

demosntrations : list

List of Demonstration objects each of which captures Observations during user demonstrations.

stop()

Stop recording by setting _done flag to True.