Technical Annexes
Technical Annexes2011
Version 1.0
09/03/2011
Important: this version of the annex will be refined with the feedback of the competitors. Refined versions will be timely distributed to the competitors by means of the contest(at)evaal.aaloa.org mailing list.
Technologies
Each team should implement a localization system to cover the area of the Living Lab, without limits to the number of devices. Localization systems can be based on different technologies, including (but not limited to): measurements on radio communications (for example RSS, angle of arrival etc.) based on standard radios (such as IEEE 802.11, IEEE 802.15.4, or IEEE 802.15.1), RFID, or ultrawideband, infrared sensors, active infrared break beams, ultrasound, camera and optical systems etc.. The proposed systems may also include combinations of different technologies. Other technologies may be accepted provided they are compatible with the constraints of the hosting living lab. To this purpose competitors wishing to check such compatibility may inquire with the organizers by e-mail ( info(at)evaal.aaloa.org).
The teams should consider possible restrictions related to the availability of power plugs, cable displacement, attachment of devices to walls/furniture in the Living Lab, etc. The requirements of the proposed localization systems should be communicated at an early stage in order to make the necessary on-site arrangements. However the Technical Program Committee (TPC) may exclude localization systems if their deployment is incompatible with the living lab constraints.
Evaluation criteria:
In order to evaluate the competing localization systems, the TPC will apply the evaluation criteria listed in this document. For each criterion, a numerical score will be awarded. Where possible the score will be measured by direct observation or technical measurement. Where this is not possible, the score will be determined by the Evaluation Committee (EC). The EC will be composed of some volunteer members of the Technical Program Committee TPC, and will be present during the competition at the Living Lab.
The evaluation criteria are:
- Accuracy – each produced localization sample is compared with the reference position and the distance is computed; accuracy is a statistics on the distances, such as the median or the root mean square.
- Installation complexity– a measure of the effort required to install the AAL localization system in a flat, measured by the evaluation committee as the total number of man-minutes of work needed to complete the installation.
- User acceptance– expresses how much the localization system is invasive in the user’s daily life and thereby the impact perceived by the user; this parameter is qualitative and will be evaluated by the evaluation committee.
- Availability– fraction of time the localization system was active and responsive. It is measured as the ratio between the number of produced localization data and the number of expected data (the number of expected data is dependent on the sampling rate required by the benchmarks, see Section Benchmark Test). The weight of this criteria is not high (we don’t want to penalize too much solutions that are still prototypes)
- Integrability into AAL systems– Metrics used are: use of open source solutions, use of standards, availability of libraries for development, integration with standard protocols.
The following table presents the scoring scale for the individual criteria. Each criterion has a maximum of 10 points, and these are awarded as whole numbers only (no half points etc.). The weightings shown will be applied to the individual scores in order to determine the overall score:
Evaluation criteria | Weight | Reference values |
Accuracy (a statistics on the distances between the system measurement and the reference position) This criterion has been modified (see Refinement of Criteria) | 0,25 | - less than 0.5 m 10 points |
Installation complexity | 0,2 | Score from 10 to 0 depending on the time required to install the system: |
User acceptance This criterion has been detailed (see Last Minutes Clarifications) | 0,2 | Score from 0 to 10 given by the evaluation committee: |
Availability This criterion has been modified (see Refinement of Criteria) | 0,15 | Fraction of available localization data. Ranges from 0 to 10: |
Integrability in AAL systems | 0,1 | Score from 0 to 10 given by the evaluation committee. The score is as composed of: |
Benchmark Testing
The score for measurable criteria for each competing artefact will be evaluated by means of benchmark tests (prepared by the organizing committee). For this purpose each team will be allocated a precise time slot at the living lab, during which the benchmark tests will be carried out. The benchmark consists of a set of tests, each of which will contribute to assessment of the scores for the artefact. The EC will ensure that the benchmark tests are applied correctly to each artefact.
The evaluation process will also assign scores to the artefact for the criteria that cannot be assessed directly through benchmark testing.
When both benchmark testing and EC evaluation have been completed, the overall score for each artefact will be calculated using the weightings shown above. All final scores will be disclosed at the end of the competition, and the artefacts ranked according to this final score.
The time slot for benchmark testing is divided in three parts. In the first part, the competing team will deploy and configure their artefact in the living lab. This part should last no more than X minutes, to be defined in the next revisions of this document.
In the second part, the benchmark will be applied. During this phase the competitors will have the opportunity to perform only short reconfigurations of their systems. In any case, this part should be concluded in Y minutes (strict deadline, to be defined in the next revisions of this document). In the last part the teams will remove the artefact from the living lab in order to enable the installation of the next competing artefact (also this part will have strict deadlines)
Competing teams who will fail to meet the deadlines in parts 1 and 3 will be given the minimum score for each criterion related to the benchmark test. Furthermore, artefacts should be kept active and working during all of the second part. If benchmark testing in the second part is not completed, the artefact will be awarded a minimum score for all the missing tests.
During the second part, the localization systems will be evaluated in two phases:
- Phase 1. In this phase each team must locate a person inside an Area of Interest (AoI). The AoI in a typically AAL scenario could be inside a specific room (bathroom, bedroom), in front of a kitchen etc. The AoI will be disclosed to the competitors before the competition, as early as possible.
- Phase 2. In this phase a person that moves inside the Living Lab must be located and tracked (we plan only 2D localization and tracking here). During this phase only the person to be localized will be inside the Living Lab.
In this phase each localization system should produce localization data with a frequency of 1 new item of data every half a second (this will be also used to evaluate availability). The path followed by the person will be the same for each test, and it will not be disclosed to competitors before the application of the benchmarks.
The environment will be made as much as possible similar to a house. This means that, if possible, there will typical appliances on, neighbours wifi AP on, cellular phones on etc. The kind of environmental noise will be defined during the shepherding phase of the competition.
In order to evaluate the accuracy of the competing artefacts, the organizers will compare the output of the artefacts with a reference localization system. Currently, the reference localization system, the number and lengths of the paths used in the tests, the number and position of AoIs and the algorithm used to evaluate the accuracy are being defined by the TPC. Details will be made public to the competitors as soon as the decisions in merit will be taken by the TPC.
Environment
Definition
The CIAmI Living Lab is an approximately 90 m2 infrastructure that simulates the real environment of a citizen´s home combined with the existence of Information and Communications Technologies (ICT) massively distributed across the physical space, but as much invisible as possible to the people living in it.
Location
• CIAmI Living Lab is placed on the Valencia´s Industrial Park (Parque Tecnológico de Valencia, Paterna) on Spain.
• CIAmI Living Lab has been built inside the private company Soluciones Tecnológicas para la Salud y el Bienestar industrial bay.
Infrastructure
CIAmI Living Lab occupies the ground floor (level sub-1) on the building (please, find the document “CIAmILivingLabPlans.pdf” where you can see the CIAmI Living Lab plans in high resolution).
- Next to the CIAmI Living Lab there are some laboratories and devices working with RF wireless technologies at 2.4 GHz.Near to the CIAmI Living Lab there is a server room with servers and communications equipments (WiFi routers, switches, etc) in 7/24 operation mod
- The EvAAL competition candidates can install its indoor localization systems only in the area reserved for the challenge (please, see the figure number 3.
- CIAmI Living Lab is composed by a control room, a 3D immersive room, a bathroom, a kitchen, a living room and a bedroom. The indoor localization system could be installed in either.
- CIAmI Living Lab has been designed with an open distribution (like a loft). It´s very important to note that only the control room, the 3D immersive room and the bathroom are closed rooms.
- CIAmI Living Lab frontview is higher than backview.
- CIAmI Living Lab frontview include the drop (or suspended) ceiling and the technical floor.
- The real CIAmI Living Lab wall height (without drop ceiling and technical floor) is 252 cm.
- CIAmI Living Lab infrastructure was building with a prefabricated modules of plaster material (Pladur is the concrete trademark of it).
- Installing devices over drop ceiling or under technical floor is quite complex (non immediate almost) and consumes a lot of time.
Architectural requirements and materials:
- The industrial bay´s outdoor walls are reinforced concrete.
- The industrial bay´s indoor walls and the CIAmI Living Lab´s walls are dry walls or prefabricated modules of plasterboard (Pladur®).
- Easily removable floor (600 x 600 mm width and 40 mm thickness) wood laminated.Easily removable plaster ceiling (please, find attached the document “Anex2evAALCompetitionCIAmILivingLabPlans.pdf” where you can see the CIAmI Living Lab plans in high resolution).
- Bathroom walls and floor are covered with ceramics tiles, and floor is not removable, only a small part.
- It´s allowed installing devices on the floor, the ceiling and the CIAmI Living Lab walls without drilling in it (double-sided tapes are allowed for temporary installations).
- The installation above the removable ceiling in the 3D immersive room is more difficult and complex than in the other rooms because the distance between the floor and the removable ceiling is higher due to the 3D immersive environment restrictions (please, find the document “CIAmILivingLabPlans.pdf” where you can see the CIAmI Living Lab plans in high resolution).
- There are only two traditional doors in the CIAmI Living Lab manufactured with wood: the main entrance to the laboratory and the bathroom´s door.
- The control room and the kitchen are separated by a unidirectional vision glass.
- The light in the control and the 3D immersive rooms are fluorescent tubes but in the other rooms are low consumption halogen lamps.
- There is direct communication between the control room and the other rooms (except in the 3D immersive rooms) therefore passing wires is simple and fast.T
- hanks to the unidirectional glass vision between the control room and the kitchen there is direct vision in the entire laboratory (except in the bathroom for ethical reasons).
- CIAmI Living Lab is a laboratory with home look where anyone could live with total comfort and safety.
Communications requirements
• Ethernet sockets and WiFi capabilities across the CIAmI Living Lab.
• High number of power sockets around the CIAmI Living Lab.
• Appliances with home automation capabilities (Maior-Domo® by Fagor).
• Heating, Ventilating and Air Conditioning (HVAC) system.
• Devices could be installed hidden to the user´s senses.
• Oversized wiring pipes and electric boxes (inside the walls).
Accessibility requirements
• Universal design applied in all areas of home construction.
• In compliance with architectural accessibility rules.
• Accesible ramps at all entrances of the CIAmI Living Lab.
• High reconfigurability of all elements in the laboratory.
• Wide corridors (the spaces have been designed to allow anyone pass including those people with they are in wheelchairs or with other assistive products).
• Interaction elements of the laboratory have been signalized and organized to minimize risks and errors.
• Handle openers require a minimum effort to be used.
• Pushers and switchers have enough size so they can be used with fingers, the fist or other parts of the body and they have been situated in places with enough space and at a properly height so they can be reached by any user, independently if the user is sitting or standing.
Available technologies
• Software and hardware development tools for several technologies like Bluetooth, NFC or MHP for instance.
• Full list of home automation system´s sensors and actuators (KNX, X10 and Zigbee Home Automation Profile).
• Media Center with high performance connected to a 37 inches television.
• IP radio with DLNA capabilities.
• Digital television set top boxes with IP and MHP capabilities.
• Smartphones and tabletsPC with Android OS.
• IP cameras.
• Digital frame with NFC support.
• Environmental microphones with high performance.
• Wearable smart sensors.
• Several panel and touch PCs distributed around the laboratory.
• Two 3D auto stereoscopic screens.
• User-friendly and intuitive interaction devices integrated with the environment (for instance, some Nabaztag rabbits and a Roomba vacuum cleaner).
• Neurofeedback equipment for mind peak performance training.
• Inmersive 3D virtual reality system.
More information
• http://www.ciami.es
• If you have some questions or need some additional information about the CIAmI Living Lab facilities you can send an email (please, pay attention that only questions by email will be accepted) to the laboratory´s technical manager (Ángel Martínez, anmarca6(at)itaca.upv.es ).