Demo on Companion Robots for AAL

Important: this annex may be refined with the feedback of the competitors and distributed to the competitors by means of the contest(at)evaal.aaloa.org mailing list and the web site.

Technologies

Research on robotics is progressively focusing its attention to the design and development of companion robots to help seniors and disable people living independent and safe in their own homes. There are many challenges in this research field related to the interaction and cooperation between robots and humans, and the navigation of robots in unstructured environments. The companion robots should be able to autonomously navigate in safe and reliable manner over indoor and crowded environments. the ability of the robot to manipulate the environment is a very important challenge to provide effective services to the users, like the objects transportation (e.g. boxes, shopping bags, food). The possibility to recognize people, objects, foods, drinks, furniture and electric appliance is an important feature of the emerging companion robots, in order to develop new applications and provide innovative services. Robotic vision and intelligent systems for object recognition are  very challenging topics. Companion robots should be able to recognize the world around and properly interact with it. Proper an safe interaction between robots and the environment, and robots and humans is a fundamental aspect to design acceptable companion robots to be used in real domestic environments. The Robot design in terms of materials, colors and shape affects the acceptability of the new human companion. This edition of the EVAAL competition focuses its attention on the navigation capability of companion robots in real unstructured and populated indoor environments. The companion robots will face several challenges in this competition, regarding the navigation accuracy and reliability, the capability of safely approach the user and other people  in a proper manner to provide acceptable robotic services.

Each team should deploy a companion robot to provide services in the field of the AAL, such as communication services, lift support, reminders, drug delivery, etc. Restrictions about robot dimensions and shape are related to the doors and rooms width of the Peccioli Living Lab: in particular, the robot should be able to cross 0.90m width doors (please refer to the Peccioli Living Lab description at the end of the document for details and contact EvAAL staff in case of doubts). Competitors are requested to integrate their robotic solution in the Living Lab, including the ability to move over tile and parquet floors as well as get on a carpet or move near tables and chairs. Main information about furniture and furniture’s locations in the environment will be reported on  time before the competition in a technical annex.

This competition is focused on the evaluation of different kind of companion robotic platforms designed to provide services to seniors and disabled people. 

For the competition will be recreate different types of scenarios where robots provide real services to persons in a real Smart Environment. Main topics of the challenge are Robot/Smart Environments interaction and robot/user cooperation to provide high quality services to the user in an effective manner. For that reason the robotic platforms are expected to exchange data with the Smart Environment in order to a-priori know the user position in the environment and cooperate with the user to perform complex task that the user or the robot by themselves could not fulfill.

During the competition, several kind of scenarios will be simulated and Innovative technical solutions will be appreciated as well as the capability to interact and exchange information about user position and home status with distributed sensors over the Smart Environment. Robot/Smart Environment data exchange will be simulated for this competition, using the benchmark software provided by EvAAL organization. In this edition the events occurring in the domestic environment will be simulated and data about the occurring events will be provided to the robots using WI-Fi communication.

Benchmarking system

A simple software integration with the benchmarking system is requested. The robot has to be able to send and receive messages through TCP sockets. The benchmarking software runs on a dedicated machine and the communication with the robot works as follows:

• the robot has to send localization data to the benchmarking system at 5Hz;
• the benchmarking system can send domestic event data to the robot at any time.

Localization data is identified by the position of the robot (x,y and theta, the origin of the coordinate system is map-centered) and a timestamp.

Domestic Event data is identified by an event (i.e. door is open, gas leakage...) and the current position of the user.

Detailed instructions about the integration with the benchmarking system will be provided on a dedicated manual.

The robot and the benchmarking system must be synchronized. The competitors must come with a NTP client installed on the robot and they must run it before starting the benchmark. The organizer will verify that the clocks are actually synchronized.

Activities to do

Remote controltex

Description of activity:

Robot is remotely controlled and has to move in a typical domestic environment (apartment on a single floor), especially in specific and narrow path. Path is not provided before but is described during the day of perform. Typical path includes features of a common domestic environment as:

• carpets
• doors (open doors, like a narrow passage)
• quiet roughly floor (e.g. tiles)
• other narrow spaces (e.g. corridor)

Path does not include stairs or other significative steps.

Each robot has to perform two different paths. User is not allowed to directly see/control the robot (aim of the task is a simulated completely remote environment, also about way of communication / protocol)

The task is evaluated both with a daily light (light that is in the environment with open window during the day) and a completely dark environment, simulating a power blackout during night.

Specific rules:

• robot is moved by a member of team, as an “expert caregiver”
• robot is “alone” (without other persons) in home
• map of apartment is provided before; teams are allowed to make their own map during the setup procedure

Autonomous movement started by a environment alert

Description of activity:

An alert is sent to robot, together with user position. Robot has to navigate toward the user, alerting him/her about critical situation in the house.

Robot has to discern about two different kind of alert:

• main door open
• gas leakage in the kitchen

and has to behave in two different ways,te clearly recognizable by user.

The robot has to move in a typical domestic environment (apartment on a single floor) with carpets, doors (open doors, like a narrow passage), other narrow spaces (e.g. corridor).

During the task, the trajectory of robot is crossed by another person in the house, to test obstacle avoidance capabilities. The cross is trajectory is done, as much as possible, when robot is moving on a straight line.

Autonomous movement started by a user request

Description of activity:

Robot is called by user and has to navigate toward him/her.

Conversely of previous task, how user can call robot (e.g. remote microphone, app on smartphone, etc.) is up to teams. In this phase, usability and acceptability are evaluated.

The robot has to move in a typical domestic environment (apartment on a single floor) with carpets, doors (open doors, like a narrow passage), other narrow spaces (e.g. corridor).

During the task, the trajectory of robot is crossed by another person in the house

Specific rules:

• NO artificial landmark or markers are allowed
• every obstacle in the environment is visible at height between 20cm and 50cm from floor
• there are not transparent obstacles/walls
• walls can include elements that are not “flat” (e.g. bookshelves)
• map of apartment is provided before; teams are allowed to make their own map during the setup procedure
• at the start of experiment both robot and user are not in a fixed position but they can be located in one of the position described in a prefixed list

Typical starting points for robot are: in the middle of corridor, near main door, etc..

Typical starting points for user are: on sofa, on bed, near cooker, etc..

Maximum time

Each team has up to 1 hour for setup its system. There is only a single phase of setup (not one for each task).

Remote control

Remote control task is divided in two distinct phases (two different paths); for each path, task “following the entire path” has to be accomplished in less than 5 minutes. Every path is at most  20 m long.

Autonomous movement

Each task (start by an environment alert and start by a user request) has to be accomplished in less than 15 min. For every task, path is at most 40m long (considering the shortest path as reference).   

Evaluation criteria

In order to evaluate the robots, some criteria will be applied.

For each criterion, a weighted numerical score will be awarded and added to the overall score. Each criterion goes from 0 to 10 points.

• End point accuracy (weight 0.20)
• Path and trajectory performance criteria (weight 0.15)
• Remote control performance (0.10)
• Setup complexity (weight 0.10)
• User acceptance(weight 0.20)
• Availability (weight 0.15)
• Interoperability (weight 0.10)

EC0 End point accuracy [weight 0.20]

It will be calculated considering the error between the robot position and the expected position. It will be evaluated only for autonomous movement.

Accuracy score =

     10         if          D      7.5        if          0.25m < D      5           if          0.5m < D      2.5        if          1m 2m

Considering two different tasks of autonomous movement, points about End point accuracy are calculated as the mean between the two tasks.

As before, points about smoothness are given based on the best performance provided by competitors as the smallest value about smoothness.

  10    if   110% < T
  7.5   if   120% < T   5      if   150% < T
  0      if   T > 150% of smallest smoothness

Obstacle Clearance is a qualitative parameter that describes the security of the trajectory (i.e. the minimum distance of the robot and the obstacles). Obstacle Clearance  parameters are described in the following table:

For each collision a penalty of 1 points will be subtracted to points of Obstacle Clearance (until a sum of 0 points).

The overall path and trajectory performance is calculated as the sum of all the above parameters.

(Path length + Time + Smoothness + Obstacle Clearance)/40*10

Considering two different tasks of autonomous movement, points about Path and trajectory performance criteria are calculated as the mean between the two tasks.

Note. If the installation on the robot of the StarGazer system (www.hagisonic.com) is technically possible, it will be used for the metric calculation. Otherwise data about position will be taken directly from robot.

EC2 Remote control performance [weight 0.10]

Remote control performance is a qualitative parameter which takes in account the ease of use in a remote control way. Qualitative evaluation considers:

• maneuverability;
• quality of audio/video of telepresence.

EC3 Setup complexity [weight 0.10]

Effort required to setup and configure the robot to be able to operate in the domestic environment. It will be measured by the evaluation committee as a function of the person-minutes of work needed to complete the installation. This phase include also a map building procedure if needed.

The time T is measured in minutes from the time when the competitors enter the living lab to the time when they declare the installation complete (no further operations/configurations of the system will be admitted after that time), and it will be multiplied by the number of people N working on the installation, considering the first person as entire (1) and the other ones as half (0.5). For example, if two people take 10 minutes to complete the setup process, total score is computed taking in account T=10, N=1+0.5.

The parameters will be translated in a score (ranging from 0 to 10) according with the following formula:

            Setup Complexity

EC4  User acceptance [weight 0.20]

The acceptability is a primary feature of a robot companion. This feature expresses how much the robot companion is accepted by the user and by the caregiver (as user of remote control mode) and how much invasive the robot is in the user’s daily life, and in the home environment, and thereby the degree of potential discomfort or pleasantness of use perceived by the user.

An assessment protocol will evaluate a set of elements, listed below:

• usability (ease of use of the human-robot interface, speech recognition,…);
• utility;
• impact on daily habits;
• aesthetics;
• maintenance.

Furthermore the protocol aims at determining objective and quantitative parameters that represent information directly traceable by means of numerical values.

This parameters will be evaluated by answering a questionnaire.

The score will be calculated by dividing those accumulated points P by the maximally reachable points and multiplying by the maximum score 10.

User Acceptance score = P/20*10

Usability (0-5)

EC4.1 Is the robot easy to be used for the tasks it is designed? (2)

EC4.2 Are the software interface and/or the speech recognition simple and effective? (2)

EC4.3 Does the user need no or short training for the correct use of the robot?  (1)

Utility (0-4)

EC4.4 Is the companion robot useful for the tasks it is designed? (2)

EC4.5  Does the robot improve the Activities of Daily Life of the user? (1)

EC4.6 Does the robot enhance the safety of the user and the house? (1)

Impact on daily habits (0-4)

EC4.7 Is the companion robot well integrated in the house (low obtrusiveness)? (2)

EC4.8 Is the robot not invasive in the user’s daily life? (1)

EC4.9 Does the robot need other installations in the home environment (including for example cameras, sensors, etc.)? (1)

Aesthetics (0-3)

EC4.10 Is the external appearance of the robot familiar and friendly? (1)

EC4.11 Is the robot design anthropomorphic? (1)

EC4.12 Does the user feels relaxed and no pain using the companion robot?  (1)

Maintenance (0-4)

EC4.13 Does not the robot requires the exchange of parts during routine maintenance(e.g. batteries)? (2) If yes go to EC4.15.

EC4.14 Does the robot work reliably if no routine maintenance is performed for at least three months? (1)

EC4.15 Does the robot work reliably without re-calibration? (2) If yes go to EC5.

EC4.16 Does re-calibration of the system only need to be carried out every three months or even less often? (1)

EC5     Availability [weight 0.15]

The availability evaluates the correct functioning of the robot companion during the different tasks.

The correct functioning of the navigation system of the robot will be evaluated by the measurement of the ratio between the number P of produced localization samples and the number of expected samples E, during the time duration of the task. Competing companion robots are expected to provide samples at the frequency of 5 Hz. Excess samples are discarded.

The values of availability A will be translated into a score (ranging from 0 to 10) according to the following formula:

Availability score = 10*P/E

EC6     Interoperability [weight 0.10]

It evaluates the degree of interoperability of the companion robot in terms of openness of the SW, adoption of standards for both SW and HW and  use of commercial parts.

This parameter will be evaluated by answering a questionnaire. The overall score will be given by summing up all weighted criteria.

Metrics will be used are shown in the following list:

• use of open source solutions;
• use of standards;
• availability of tools for testing/monitoring the robot;
• availability of libraries for development (API, documentation, tutorial, sample application);
• integration with standard protocols.

Evaluates the degree of interoperability of the solution in terms of openness of the SW, adoption of standards for both SW and HW, replaceability of parts of the solution with other ones. This parameter will be evaluated by answering the following questionnaire. The overall score will be given by summing up all weighted criteria.

Interoperability score = P/14*10

Code integration (0-6)

EC6.1. Do you provide any API for integrating your robot system into others? (2)

EC6.2. Will you provide it in the future? (0.5)

EC6.3. Do you provide any written documentation for this API (code documentation, tutorial, and/or sample application)? (1)

EC6.4. Will you provide it in the future? (0.5)

EC6.5. Do you publish your code as open source? (2)

EC6.6. Do you use any well-known application-level protocol which would allow an external system to plug into the robot system without requiring any further specification about the protocol? (e.g. SOAP, XMPP, XML-RPC, DPWS, but not TCP/IP, Ethernet) (1)

Testing and configuration (0-6)

EC6.7. Do you provide any tool for testing/monitoring the robot? (2)

EC6.7.1 Is the tool easy to be used? (1)

EC6.8. Will you provide it in the future? (1)

EC6.9. Do you provide any tool for configuring/calibrating the robot? (2)

EC6.9.1 Is the tool easy to be used? (1)

EC6.10. Will you provide it in the future? (1)

HW (0-2)

EC6.11. Can any part of your robot be substituted by another commercially available one without any modification on the HW and/or the firmware? (1)

EC6.12. Is your firmware publicly available? (1)