Work on v3 controller

Designer 2.5
#1

Hi

I've cleaned up and moved the core code of controller v2 into v3:

https://github.com/openremote/openremote/tree/master/controller/src

Deployment of controller.xml and rules is working in tests:

https://github.com/openremote/openremote/blob/master/controller/src/test/groovy/org/openremote/test/rules/SimpleRuleTest.groovy

My main goal was to get the "Climate Control" demo setup working (still looking for a better name). Currently the rules compile and we are ready for writing tests:

https://github.com/openremote/openremote/blob/master/controller/src/test/groovy/org/openremote/test/rules/ClimateControlTest.groovy

Rule definitions are 99% compatible with v2, it's mostly package renaming and introducing more Java utilities for writing rules.

Other major tasks moving forward, also see the TODOs in code:

1. Every PullCommand (previously ReadCommand/StatusCommand) needs a new thread, any PushCommand (previously EventListener) might also start its own thread. The Climate Control demo starts more than 100 threads. The current event processing chain is the same as a Camel route, the event processing context is the same as a Camel exchange. Replacing these bits with Camel will also resolve the issues with Pull/PushCommand scaling.

2. More commands and especially Velbus need to be ported to v3. I've done VirtualCommand and DateTimeCommand. The API is still 99% compatible with v2, only how commands are build has changed. The VirtualCommand is a bit of a mystery for me, especially how its arguments/parameters are used/not used, some cleanup and better docs should be done. On the southbound side we also need a simulator, probably implemented as a command?

3. There is no proper data type handling for sensor values. We have from/to string conversions all over the place. Value range checks for min/max constraints are inconsistent. We have rules logic based on whether a value is "OFF" or "off". Not sure how this can be fixed without breaking backwards compatibility, maybe we want to rethink the Sensor model. Then we could also resolve the ambiguities in unique device/sensor/command IDs and names.

4. The StatusCache is now the DataContext, where the current state of sensors is stored. I've dropped the old change polling mechanism. We need a (messaging) solution for sharing state updates with in-process and external services. This also has to handle ExecutableCommands and later device discovery. I'm working on this task and how the in-process and remote API of the controller will look like northbound.

My proposal would be to get more tests working first. Especially Climate Control should be extensively tested from a business logic perspective. This gives us stability when we introduce Camel, improve models and data type handling, etc.

Cheers,
Christian

#2

Status update: Controller was renamed to Agent in v3 and I've cleaned it up some more. The main API is now AgentContext, start browsing the code from here if you are interested:

https://github.com/openremote/openremote/blob/9cf842d5a4091fc6ceadacc55cb03553c8a2deb8/agent/src/main/java/org/openremote/agent/context/AgentContext.java

Thread handling still needs a major redesign, and the Sensor class hierarchy should be looked at.

Next I'll work on the Manager and remove the prototype (and now obsolete) Agent code that connected to a v2 controller through HTTP. It never really worked well.

My goal for the Manager is to get multiple instances of the new Agent running within the same process as the Manager. As for running the Agent standalone on a gateway, we have to discuss how remote communication should work.

Cheers,
Christian

#3

Would there happen to be an ‘online demo’ version up or is it still in too early a stage?

For rules logic with “OFF” or “off”, would it simplify things to force-format values to all uppercase (or lowercase) when expecting booleans?

As for data type handling, while it might be a bit of extra luxury, would it be possible to have built-in hex to dec (or dec to hex) conversion?

Cheers,

Patrick

#4

Would there happen to be an ‘online demo’ version up or is it still in too early a stage?

I’m afraid you won’t see anything useful on a demo right now, more work has to be done to integrate the controller/agent changes. Some security related tasks are also still open and I think they should be resolved before we can install a public demo system.

Having said that, the project should be reasonably easy to build and run if you know some Docker, see https://github.com/openremote/openremote/wiki

For rules logic with “OFF” or “off”, would it simplify things to force-format values to all uppercase (or lowercase) when expecting booleans?

As for data type handling, while it might be a bit of extra luxury, would it be possible to have built-in hex to dec (or dec to hex) conversion?

This is the entry point of the agent datatype system:

https://github.com/openremote/openremote/blob/master/agent/src/main/java/org/openremote/agent/sensor/Sensor.java#L134

Protocol implementations return a string representation of their internal sensor value. How can a protocol return binary data? Base64?

The user then picks a class of the Sensor hierarchy to transform the value into the desired SensorState. We have SensorState and SensorState. Not sure why SwitchSensorState isn’t boolean, this would help with the “on” “OFF” issue.

Next the rule LHS expressions are operating with SensorState: foo == “bar”, foo == 123. I don’t know the automatic conversion rules when the operands are of different type. Maybe the explanation why we don’t use actual boolean types is related to this.

Finally the SensorState will end up in the asset database as current and historical state of the sensor. This database is build with JSON storage, so the SensorState must ultimately be transformed to a JSON value. Currently we have SensorState#serialize(), which transforms the value into a string.

I guess JSON could play a more prominent role overall. If protocols return a JSON value, combined with JSON expressions in rules…

Christian

#5

Following up on this, here is a proposal for the Agent/Thing asset model.

Assets form a tree structure, several assets can be grouped together under the same parent. The root of an asset tree is always a Tenant asset.

An asset can be of well-known type (which means we have specific functionality for that asset) or Custom user-defined type (which means it’s a domain asset handled in a generic fashion). The Agent and Thing are well-known asset types:

https://github.com/openremote/openremote/blob/master/manager/shared/src/main/java/org/openremote/manager/shared/asset/AssetType.java

Besides type, every asset has a few static properties such as asset name and geolocation coordinates. Furthermore, each asset can have a set of dynamic attributes. How the dynamic attributes are used depends on the asset type. For example, a Building asset might have a street and zip code, a Floor asset might have a floor plan, etc.

The value of a dynamic asset attribute must be one of the defined types:

https://github.com/openremote/openremote/blob/master/manager/shared/src/main/java/org/openremote/manager/shared/attribute/AttributeType.java

Finally, each asset attribute can have additional metadata items. Examples are a pretty label and description of the attribute, or how a string attribute value should be formatted when shown to the user, or value constraints such as integer ranges and regex patterns.

Attribute meta items are a simple name/value list where duplicate names are allowed for multi-valued items. There are no restrictions on the value of a meta item, any JSON type is supported, and how the value is used depends on the meta item name. Again, we support some well-known meta items by URN but users can add arbitrary custom meta information to any attribute:

https://github.com/openremote/openremote/blob/master/manager/shared/src/main/java/org/openremote/manager/shared/asset/AssetAttributeMeta.java

Given this asset model, how can we represent Agents and their Things? This is an example definition of an Agent and a Thing:

https://gist.github.com/christianbauer/92ac576a4be8f28f32cab6e4f9277681

This Agent connects a Hue Bridge and a ZWave USB stick to the system. We may have many Agents and each Agent can have many of these protocol bindings. Protocol specific configuration, such as the Hue Bridge host and username or the ZWave USB port can be set at the Agent level. Other (global) Agent configuration including rulesets would also be configured as Agent attributes. Whether deployed in-process or on a remote gateway, each Agent instance has its own data context and therefore rule facts.

The Thing might be a child asset of an Agent, for example, if it’s a device that is automatically discovered and added to the asset tree by the Agent. A user can however also create a Thing manually anywhere in the asset tree and configure it as desired.

The example Thing asset is a light fixture. The current, known state of the light is represented with asset attributes: Is it turned on, its dim level, color, and power consumption. To implement this, add an agent link meta item to the Thing’s attribute. This is a marker for “bind this attribute to a device or service” using the identifier of the Agent and a protocol binding attribute name.

The linked protocol handles south-bound read and write of the attribute value: If the user writes a new value into the Thing asset attribute, the protocol translates this value change into a device (or service) action. If the actual state of the device (or service) changes, the linked protocol writes the new state into the attribute value and the user is notified of the change.

Protocol-specific meta items required for the link can be added to Thing attributes, such as the Hue light identifier or the ZWave node and command.

TBC…

#6

Why is geolocation or even name considered to be static?

What is geolocation of cars and/or things in mobiles? What is geolocation of things in a space vehicle traveling to Mars? Even though, 2D geolocation on 3D Earth is too limiting.

Can’t we rename things? Are anonymous things not possible?

Kind regards,

Michal

#7

Why is geolocation or even name considered to be static?

Because every thing in the world has a name, so we can identify and talk about it, and a location in space. The static Asset type captures that.

Geolocation model types and possible operations are currently build on https://en.wikipedia.org/wiki/JTS_Topology_Suite

Can’t we rename things? Are anonymous things not possible?

Yes, things can be renamed. No, you can’t work with something that you can’t even identify. The terms “static” and “dynamic” in my description refer to the type system of the programming language. It doesn’t mean something is constant or mutable.

#8

JTS "provides an object model for Euclidean planar linear geometry”, which is much less than location in space, i.e. it is only location on a plane.

By the way, for the location of an asset do you expect that each asset has absolute coordinates or relative to it’s parent?

#9

Making progress on the implementation of this design and in-process Agent/Thing deployment.

The sample data for Agent and Thing and the model is the same as in my earlier proposal:

https://github.com/openremote/openremote/blob/master/manager/server/src/main/java/org/openremote/manager/server/DemoDataService.java#L285

When a Thing asset is modified (created, updated, deleted), its attributes are examined and linked to and unlinked from the configured protocols:

https://github.com/openremote/openremote/blob/master/agent/src/main/java/org/openremote/agent3/protocol/Protocol.java

A Protocol implementation must have a unique name in a VM, this is however currently not enforced. OpenRemote protocols are in the “urn:openremote:protocol” namespace.

The Protocol implementation can receive AttributeValueChange messages on the ACTUATOR_TOPIC. It can produce AttributeValueChange messages on the SENSOR_TOPIC. The linked attributes provide the model for the Protocol to perform these operations. How attributes and value changes map to actual device and service calls is up to the Protocol implementation.

Datatype conversion is also delegated to the Protocol implementation: If an attribute has a particular AttributeType and therefore a certain JsonValue, the Protocol must receive and send value change messages with values of that type. Some kind of converter system can be introduced later, although I find JsonValue#asXXX() is already a good utility for protocol implementators.

See the AbstractProtocol and SimulatorProtocol for a concrete implementation.

A simple test of this can be found here:

https://github.com/openremote/openremote/blob/master/manager/test/src/test/groovy/org/openremote/test/agent/AgentDeploymentTest.groovy

Next steps are:

  • Finishing the implementation of the SimulatorProtocol with a web or command-line interface

  • Implementing other protocols (Yahoo Weather because it should be easy, Velbus because we need it) to proof the design

  • Finding out how we move old Connector/Sensor to the new API, and how rules are handled

  • Replacing the old connector/agent editor UI in the Manager client, finalizing Asset client API