[MQTT] To update several attributes with a single message

[adrianocarvalhodtx]

Hello,

To save power, using MQTT, we would like to update several attribute with a single message. Unfortunately, the MQTT API allows only one attribute per mesage.

To workaround this limitations, we are publishing a JSON payload to an “inbound” attribute and using AttributeLinks to update individual attributes. This, however, seems a bit awkward, especially the setup.

Do you plan to support this (i.e., to update several attributes with a single message) in the near future?

We though of adding support for this with a special attribute name (in {realm}/{clientId}/writeattributevalue/{attributeName}/{assetId}). However, there seems to be an issue with this.

Because access control seems to be performed in DefaultMQTTHandler::canPublish, if we authorize such a publication, such a JSON payload may be used to update an attribute whose access is not authorized.

To workaround this, we thought of:

1. Accepting all such publications (with the special attribute name).
2. But, whenever there’s a publication, of using canPublish for each individual attribute and only updating the attribute if access is authorized.

Any thoughts?

Thank you.

Best regards,
Adriano Carvalho

[Rich]

Hi,

Good thinking but as you say AttributeLink is processed as an internal request (i.e. if you authorise the creation of the AttributeLink then that implies who/whatever updates the attribute is OK to update the linked attribute(s) also.

I think we’ve discussed before about using a custom MQTTHandler; this mechanism would give you complete control over what data is in the payload and how you process it.

MQTT is by design quite limited; we cannot support multiple attributes in one payload as the topic is all we can use to authorise the publish and therefore we couldn’t authorise the payload itself (although the system could still disallow un-authorised attribute publishes further down the processing pipeline).

[dcompent]

This is a very important real-world requirement that should not easily be dismissed or ignored simply because of to the current internal structuring of OpenRemote’s MQTT extensions. As @adrianocarvalhodtx mentioned, keeping power requirements low is often extremely important for some systems. In addition to that, cellular systems are more often than not billed by the MB (if not the KB)–so every “little” savings becomes very substantial.
Especially if there are several hundred attributes to update. It’s even worse if each Attribute is 1-3 characters (think “Byte” status, 0-255): the overhead of the MQTT topic string alone at >80 bytes per Attribute simply cannot be ignored. (This isn’t even including all of the network stack overhead.)

I am tinkering with an implementation modification to the “DefaultMqttHandler” that hopefully could be an upgrade to the current default, providing the oft-requested extensibility of being able to update more than one Attribute with a single MQTT Publish.

As of right now, I do see where the Attribute permissions are tied to the MQTT topic. (The result of the permissions query is cached to reduce database queries–but said cache is also tied to the MQTT topic.)

My plan:

• Implement a slightly different endpoint, namely “writeattributevalues” (i.e. just plural, not singular), where the “{attributeName}” part of the topic is omitted (only 4 topic tokens, not 5). In other words, “{realm}/{clientID}/writeattributevalues/{assetID}”.
• All “root level” JSON keys are treated as AttributeNames, with any value (including JSON arrays and objects) being assigned to said Attribute.
• If the named Attribute cannot be found in the specified Asset, it is ignored. (This avoids database clogging with unspecified JSON payloads.)
• We have to bypass the “authorizeEventWrite” functionality in “canPublish”, as there is no “{attributeName}” in the topic. This means that any Publish request is granted, providing the User has sufficient permissions to access the Asset.
• It is relatively trivial to semi-duplicate the “authorizationCache” implementation in the JSON root key iterator loop: check for cached authorization on each existing Attribute. If not found, then confer with “authorizeEventWrite” (append the Attribute name to the “key”) → save the result to the “authorizationCache” for the following time.

So far, I’ve gotten everything except the secondary “authorizationCache” step implemented. Here’s a test with the following:

• topic: master/mqttx_xxxxxxxx/writeattributevalues/3mEmxDh7VyCSnmbRFHCEoV
• publish: {“notes”: “Idle”, “battery”: 38.12, “cells”: {“vals”: [13,0,98,2,6]}, “void”:0}

(note that I added several of these Attributes in the UI for test purposes)

Here’s the console output:

After I get the permissions per topic re-implemented (with caching) in the JSON iterator loop, is this something that could be considered as a PR for upstream OpenRemote?

[panos]

First of all, thank you for taking a look into the inner workings of OpenRemote and making a plan of how this could look like.

While I do understand why people may require this functionality, they’re encouraged to go on and implement this. But, before jumping into the handler code and typing away, we should think of how this functionality should play out in general for a project.

As someone that has been doing a deep dive into communicating with MQTT-enabled devices that may contain hundreds of Attributes, I thought how this could be developed for maximum flexibility, efficiency, and interoperability with different devices. Also, I am not able to format my devices’ messages to my liking. As such, I decided to create my own, specialized MQTT handler.

Creating an MQTT handler proved to be exceedingly easy once you understand how it works, and you have a lot of room for doing what your payload requires you to. You also have the possibility of sending MQTT messages, which proved to be useful for my implementation.

Using an MQTT handler for your devices allows for maximum control. You can use JSON schemas and parsers for handling your data as a normal class. You could even represent your payload altogether as an attribute with all of the functionality that goes along with it. Split your payloads into different attributes, perform data analysis, create events, add listeners, and make your implementation as robust as possible.

So while that functionality, writeattributevalues, does make sense for certain use cases, I believe that the overwhelming majority of use cases would benefit a lot more from using custom-built MQTT handlers, given their ease of implementation, use, the robustness they provide, the functionality they unlock. Investing a couple of hours to implement your own MQTT handler will definitely prove more useful for your instance than a normal writeattributevalues will ever be.

I’m on mobile so I can’t find it right now, but I have a quite good forum post with someone here where I help them implement their own MQTT handler. There is a multitude of examples, including in my fleet-management repository in the OpenRemote org. The resources are there, I’m here for any questions, and I hope you guys will see what I’m rambling on about

[dcompent]

@panos

As someone that has been doing a deep dive into communicating with MQTT-enabled devices that may contain hundreds of Attributes, I thought how this could be developed for maximum flexibility, efficiency, and interoperability with different devices. Also, I am not able to format my devices’ messages to my liking. As such, I decided to create my own, specialized MQTT handler.

My point of view is not one of “trying to make existing commercial MQTT products work with OpenRemote”, as the chances of them matching every aspect of the OpenRemote MQTT infrastructure is between nil and zero (especially if we’re talking self-provisioning.) In this respect, we completely agree.

However, the niche I am trying to address, is one addressed by the OP in the initial post in this thread–and several others dotted across the forums: DIYers, programmers, or project managers wanting to design a product around the default OpenRemote infrastructure. Where the goal is not to reinvent the wheel (i.e. donning a Java hat and writing a custom handler), but rather to leverage what is already present. And what is present is extremely data-inefficient for updating more than a single Attribute.

Yes, you can cobble together OpenRemote JSON parsers and such in the UI–but it does not appear (as far as I yet know) that these are geared towards (or even usable) in auto-provisioning scenarios. (For one-off manually-setup and manually-configured implementations, though, they are a viable option.)

@panos

Investing a couple of hours to implement your own MQTT handler will definitely prove more useful for your instance than a normal writeattributevalues will ever be.

“a couple of hours”–that is, for an experienced Java programmer who already has the entire ecosystem installed and properly configured.

For me personally, if I didn’t have an experienced Java programmer to sort out the multitude of issues with tooling I have never before used–much less figure out tooling version issues resulting from incorrect/insufficient OpenRemote documentation, etc., etc.–this would be an impossible task. (Even with all of the help, it still took me a full 3 days to get to the point of the above screenshots.)

@panos

I’m on mobile so I can’t find it right now, but I have a quite good forum post with someone here where I help them implement their own MQTT handler. There is a multitude of examples, including in my fleet-management repository in the OpenRemote org. The resources are there, I’m here for any questions, and I hope you guys will see what I’m rambling on about

Links, please? The only example I was able to find anywhere online was the Teltonika MQTT handler–and it was an extremely valuable resource. Without real-world examples (to say nothing of documentation), those outside of the inner OpenRemote programming team stand very little chance of figuring out how to implement anything.

Took me the better part of a day to figure out how to read an Attribute’s metadata (checking for the “ACCESS_RESTRICTED_WRITE” entry)–and it wasn’t after several dozen keyword searches across the entire OpenRemote Github repository that I found a usable code snippit that actually fit the need. I’m a C/C++ programmer: Java is a completely different planet!

TL;DR: We embedded systems developers are looking for a multi-Attribute-write OpenRemote MQTT solution that does not involve recompiling OpenRemote from source! One that can be leveraged for projects specifically designed to work with OpenRemote.

[panos]

@dcompent

However, the niche I am trying to address, is one addressed by the OP in the initial post in this thread–and several others dotted across the forums: DIYers, programmers, or project managers wanting to design a product around the default OpenRemote infrastructure. Where the goal is not to reinvent the wheel (i.e. donning a Java hat and writing a custom handler), but rather to leverage what is already present. And what is present is extremely data-inefficient for updating more than a single Attribute.

I understand where the need for this might emerge, especially for users that may not be that well-versed with software development. But, given that OpenRemote is an open-source project, you are more than welcome to create a new pull request!

@dcompent

Yes, you can cobble together OpenRemote JSON parsers and such in the UI–but it does not appear (as far as I yet know) that these are geared towards (or even usable) in auto-provisioning scenarios. (For one-off manually-setup and manually-configured implementations, though, they are a viable option.)

I was talking about using parsers within OpenRemote code, within the custom MQTT handler.

@dcompent

Links, please? The only example I was able to find anywhere online was the Teltonika MQTT handler –and it was an extremely valuable resource. Without real-world examples (to say nothing of documentation), those outside of the inner OpenRemote programming team stand very little chance of figuring out how to implement anything.

Yeah, I wrote that MQTT handler, it’s still a work in progress . It is a good place to start if I may say so myself, but also the MQTT Handler abstract class, that contains documentation for (I think) all of the methods you may want to implement. There’s also the forum thread about an MQTT sparkplug integration, where I go a bit more in-depth about how to create an MQTT handler.

I would also like to mention that I do not want to make a custom MQTT handler appear to be a one-stop-shop for all OpenRemote users with an MQTT device. I think that the pros for it outweigh its cons, if we remove the concept of required knowledge or time requirements. I just wanted to show this functionality as a very useful tool, which enables me to create integrations for devices at a level that has never been seen before.

@dcompent

keeping power requirements low is often extremely important for some systems. In addition to that, cellular systems are more often than not billed by the MB (if not the KB)–so every “little” savings becomes very substantial.
Especially if there are several hundred attributes to update. It’s even worse if each Attribute is 1-3 characters (think “Byte” status, 0-255): the overhead of the MQTT topic string alone at >80 bytes per Attribute simply cannot be ignored. (This isn’t even including all of the network stack overhead.)

Here you mention how power and bandwidth limitations are also quite significant for your use case. To me, this sounds like this could lead to real issues pertaining to cost. It also sounds like an implementation like this could be mission-critical for a project you may be taking on. This is one of those use-cases where I do believe that a custom MQTT handler would just make your life a lot easier. If people are thinking of network overhead for IoT devices creating unnessessary cost, then I do consider it to be an issue that deserves its own MQTT handler as a solution.

In total, I do understand that need of some users for writeattributevalues, but I also need to highlight the importance of custom MQTT handlers for use-cases like these. In any case, opening a pull request is free, and it would improve OpenRemote as a whole!

[dcompent]

@panos

In total, I do understand that need of some users for writeattributevalues, but I also need to highlight the importance of custom MQTT handlers for use-cases like these.

And I do agree, in that custom MQTT handlers are a requisite for existing commercial products. But I’m trying to broaden the MQTT horizons for projects specifically being developed for use with OpenRemote.
Much like you can just get tires for a car at a tire shop–as opposed to having to build a rubber mold and make your own tires for the (otherwise complete) open-source car you just assembled.

@panos

In any case, opening a pull request is free, and it would improve OpenRemote as a whole!

Yes, but my reasoning for trying to discuss it on the forums vs “just opening a PR”…is to avoid wasting everyone’s time if the OpenRemote team is just going to reject it as “not relevant to our goal” no matter what I do.

Which the vibe I’m getting is that a PR would most likely be rejected, i.e. a waste of time to try.

[Rich]

Hey guys,

Seems like a good discussion so far and I see value in having a topic that supports multiple attribute events in one payload and the path you suggest would be the way to go.

The final piece of authorisation that would need to be addressed would be restricted users and ensuring that they are firstly linked to the asset and secondly that the attribute(s) have a ACCESS_RESTRICTED_WRITE meta item.

We would need to just drop events that don’t pass authorisation, logging them is the obvious thing to do.

We are always grateful for community contributions so a PR is the way to go and we can collaborate on getting the finer details figured out.

[dcompent]

@Rich

The final piece of authorisation that would need to be addressed would be restricted users and ensuring that they are firstly linked to the asset and secondly that the attribute(s) have a ACCESS_RESTRICTED_WRITE meta item.

I managed to get that implemented, so the function will only allow writing to Assets that are linked to the authenticated User…and only Attributes that have the “ACCESS_RESTRICTED_WRITE” meta tag.

Here’s a debug screenshot with the “ACCESS_RESTRICTED_WRITE” check:

Finding whether a User is linked to an Asset wasn’t easy, and probably can/needs to be optimized.
You can see in the screenshot the UUID and internal Username for Users; the internal Username is not exposed on the web UI.

[pdt590]

+1. It is also neccessary for me if a MQTT/HTTP agent link also can send out a message with multiple attributes inside

[dcompent]

I’m providing my progress thus far in case it helps someone in the future. (Note that it has a TXT extension to get around the forum preventing a JAVA file uploaded.)

MultiAttributeMQTTHandler.java.txt (12.2 KB)

Obviously, to utilize these these does require building OpenRemote from source–which is not something for the faint of heart.

There’s just too many little things stacking together for me to consider OpenRemote an out-of-the-box solution: lots of little things that need changed/tweaked/corrected before it can be used. (I could go into specific detail, but will only do so if asked.) For now, I’m evaluating Thingsboard–which effectively has identically what I wrote above for the default MQTT handler (i.e. multi attribute writes in a single JSON payload.)

[pdt590]

Thank you for your info. However, my suggestion is that core team should develop a new buit-in operation e.g merge attributes on Flow Rules. Then new output attribute of the flow rule is a combination of multi other attributes. So, we can use Agent to send out the new attribute to external device/system.