new version of paho-mqtt breaks mqtt messaging within pywws

[rick165]

It seems the new version of paho-mqtt (v2.0.0) breaks the mqtt messaging within pywws.

pywws.service.mqtt:Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details
Work around is to install v1.6.1 of paho-mqtt

sudo pip install paho-mqtt==1.6.1

[GeekyTim]

I'll put that on my list to fix, alongside the self-cert issue.

[GeekyTim]

I've got a REALLY simple fix to this paho-mqtt version issue, as well as allowing self-certs. I'll send a pull request once I've tested a few more things.

[cromemcos]

I just installed 24.8.0 and I was so hopeful ... but nothing changed. mqtt still broken. :-(

1:22:54:pywws.service.metoffice:data: {'dateutc': '2024-09-22 20:22:51', 'winddir': '315', 'windspeedmph': '0.00', 'windgustmph': '1.57', 'humidity': '97', 'tempf': '61.5', 'baromin': '29.7898', 'rainin': '0.0000', 'dailyrainin': '0.0000', 'dewptf': '60.7', 'softwaretype': 'pywws-24.8.0', 'siteid': 'a4f1ade3-29b2-ed11-9ac4-201642ba4e0a', 'siteAuthenticationKey': '535251'} 21:22:54:pywws.weatherstation:delay 0, pause 27.4161 21:22:57:pywws.service.mqtt:Unsupported callback API version: version 2.0 added a callback_api_version, see docs/migrations.rst for details Traceback (most recent call last): File "/home/m/.local/pipx/venvs/pywws/lib/python3.11/site-packages/pywws/service/__init__.py", line 180, in run OK = self.upload_batch()

[GeekyTim]

I will take a look and see if there are issues.

[GeekyTim]

Curious. I have that version installed on Python 3.9 with paho-mqtt 2.1, and it works fine for me.

Could you please take a look at /home/m/.local/pipx/venvs/pywws/lib/python3.11/site-packages/pywws/service/mqtt.py and check that the first line after 'def def session(self):' is:

session = mosquitto.Client(
            mosquitto.CallbackAPIVersion.VERSION1, self.params['client_id'], protocol=mosquitto.MQTTv31)

Thanks

[cromemcos]

So yes I checked the file and mine looks a little different Here is a numbered copy paste from a vi edit

`

240 for line in template.splitlines():

241 if line:

242                 result.append(pprint.pformat(line, width=256))
243         return '(\n' + '\n'.join(result) + '\n)'
244 
245     @contextmanager
246     def session(self):
247         session = mosquitto.Client(
248             self.params['client_id'], protocol=mosquitto.MQTTv31)
249         if self.params['password']:
250             session.username_pw_set(
251                 self.params['user'], self.params['password'])
252         elif self.params['user']:
253             session.username_pw_set(self.params['user'])
254         logger.debug(('connecting to host {hostname:s}:{port:d} '
255                       'with client_id "{client_id:s}"').format(**self.params))
256         if self.params['tls_cert']:
257             selfcert = False

"mqtt.py" line 246 of 301 --81%-- col 5
`

`

m@domo:/.local/pipx/venvs/pywws/lib/python3.11/site-packages/pywws/service $ grep -i Callback *.py
m@domo:/.local/pipx/venvs/pywws/lib/python3.11/site-packages/pywws/service $ grep -i mosquitto *.py

mqtt.py:* Mosquitto (a lightweight broker): http://mosquitto.org/
mqtt.py:See https://mosquitto.org/man/mosquitto-tls-7.html for information on how to
mqtt.py:http://www.steves-internet-guide.com/mosquitto-tls/ for a step-by-step
mqtt.py:This has been tested with the Mosquitto Open Source MQTT broker, running
mqtt.py:import paho.mqtt.client as mosquitto
mqtt.py: session = mosquitto.Client(
mqtt.py: self.params['client_id'], protocol=mosquitto.MQTTv31)
`

I do apologise that I am crucifying the formatting, inserting code in the tool seems not to do what I expected.

[GeekyTim]

I can see there that it is the old version of the file, so needs to be updated.

I think @jim-easterbrook wold be better at answering why it didn't get upgraded correctly.

[cromemcos]

I just redid the installation and show the output here for clarity

pipx install pywws==24.8.0
'pywws' already seems to be installed. Not modifying existing installation in '/home/m/.local/pipx/venvs/pywws'. Pass '--force' to
force installation.
m@domo:~/.local/pipx/venvs/requests $ pipx install pywws==24.8.0 --force
Installing to existing venv 'pywws'
⚠️  Note: pywws-hourly was already on your PATH at /usr/local/bin/pywws-hourly
⚠️  Note: pywws-livelog was already on your PATH at /usr/local/bin/pywws-livelog
⚠️  Note: pywws-livelog-daemon was already on your PATH at /usr/local/bin/pywws-livelog-daemon
⚠️  Note: pywws-reprocess was already on your PATH at /usr/local/bin/pywws-reprocess
⚠️  Note: pywws-setweatherstation was already on your PATH at /usr/local/bin/pywws-setweatherstation
⚠️  Note: pywws-testweatherstation was already on your PATH at /usr/local/bin/pywws-testweatherstation
⚠️  Note: pywws-version was already on your PATH at /usr/local/bin/pywws-version
  installed package pywws 24.8.0, installed using Python 3.11.2
  These apps are now globally available
    - pywws-hourly
    - pywws-livelog
    - pywws-livelog-daemon
    - pywws-reprocess
    - pywws-setweatherstation
    - pywws-testweatherstation
    - pywws-version
done! ✨ 🌟 ✨

pywws-version
24.8.0

[GeekyTim]

Does it work now?

[cromemcos]

Correct me if I am wrong but on Github it does not have the API V1 call ??

[cromemcos]

Does it work now?

So when I reinstalled then no it still does not work and this is presumably because the mqtt.py file is as the github shows the code that does not include the V1 API call.

[GeekyTim]

You are absolutely correct. It seems to have been reverted somehow.

I'll have to ask Jim.

[jim-easterbrook]

The commit history of mqtt.py doesn't show anything odd.
https://github.com/jim-easterbrook/pywws/commits/master/src/pywws/service/mqtt.py

The version on my local machine, from which the packages uploaded to PyPI were made, has these lines:

       session = mosquitto.Client(
            self.params['client_id'], protocol=mosquitto.MQTTv31)

I just redid the installation and show the output here for clarity

Note: pywws-hourly was already on your PATH at /usr/local/bin/pywws-hourly

Does pipx have write access to /usr/local/bin? I suspect the pywws stuff there is a hangover from an earlier installation, and pipx is warning you that the ones it has installed somewhere else won't get used. (I don't use pipx so don't know much about it.)

[cromemcos]

Ah okay, thanks Jim for replying. So your comment relates to stuff in /usr/local/bin. Indeed I checked and that was the result of an old installation.

so it was a

`
sudo rm -fv /usr/local/bin/pyww**

and then a reinstallation
which now looks like this`

`

pipx install pywws==24.8.0

'pywws' already seems to be installed. Not modifying existing installation in
'/home/m/.local/pipx/venvs/pywws'. Pass '--force' to force installation.
m@domo:/usr/local/bin $ pipx install pywws==24.8.0 --force
Installing to existing venv 'pywws'
installed package pywws 24.8.0, installed using Python 3.11.2
These apps are now globally available
- pywws-hourly
- pywws-livelog
- pywws-livelog-daemon
- pywws-reprocess
- pywws-setweatherstation
- pywws-testweatherstation
- pywws-version
done! ✨ 🌟 ✨
`

BUT and BUT the issue still remains pywws is unable to talk to mqtt

and I check the logs ... still bused

[GeekyTim]

Somehow the change I submitted was there, then wasn't. I'm not sure why (since Jim did the pull request). I need to get some time to spend on this; I had it working here until I took certs off my MQTT server because of issues with other software!

We'll get it sorted.

[cromemcos]

Oh thanks to GeekyTim I am now rather happy it is now working ....

Alexa, what is the temperature of the outdoor, NOW WORKS
(the mqtt messages lands at our mqtt server and is broadcast, smarthings picks it up and then it gets to Amazon)

[GeekyTim]

Did you make that change manually? Or has the build changed?

[cromemcos]

Tim, I just made a quick vi edit, recycled pywws and hey presto, it is working. Like I say very happy, but yes pls push into the codebase.

[samus-homebridge]

Apologies if this is not the right place for this, but I am struggling to get the mqtt service working. I suspect the issue is TLS - I haven't been able to configure it correctly with my MQTT installation (I'm a novice with this, especially with SSL/TLS) and I don't really need it for my purposes. Is there a way to disable TLS in the service configuration? When I leave the relevant parameters blank, pywws.hourly gives me "RuntimeError: "tls_ver" not set in weather.ini" so I suppose not. Many thanks!

[jim-easterbrook]

Apologies if this is not the right place for this, but I am struggling to get the mqtt service working.

You might reach more MQTT users by asking on the pywws mailing list.

[samus-homebridge]

Thanks! I'll give that a go if there is no way to use the Pywws mqtt service with TLS disabled -- which it sounds like is the case now?

Thanks for the project overall as well, of course -- much appreciated!

[GeekyTim]

I used to have my MQTT broker (Mosquitto) secured but decided to open it because it is within my own network. For mosquito, create a .conf file e.g. /etc/mosquitto/conf.d/insecure.conf with the following:

listener 1883
allow_anonymous true

which will open the broker on the default non-TLS port and not expect usernames/passwords.

At the end of the pywws config file I have added the following:

[mqtt]
topic = weather/current
hostname = mqttpi
port = 1883
client_id = WeatherPi2
retain = False
auth = False
template_txt = (
        '#idx          \'"idx"         : "%Y-%m-%d %H:%M:%S",\'#'
        '#wind_dir     \'"wind_dir"    : %.0f,\' \'\' \'winddir_degrees(x)\'#\n'
        '#wind_ave     \'"wind_ave"    : %.2f,\' \'\' \'wind_mph(x)\'#\n'
        '#wind_gust    \'"wind_gust"   : %.2f,\' \'\' \'wind_mph(x)\'#\n'
        '#hum_out      \'"hum_out"     : %.d,\'#\n'
        '#hum_in       \'"hum_in"      : %.d,\'#\n'
        '#temp_in      \'"temp_in_c"   : %.1f,\'#\n'
        '#temp_out     \'"temp_out_c"  : %.1f,\'#\n'
        '#abs_pressure \'"pressure": %.4f,\'#\n'
        '#calc \'rain_hour(data)\' \'"rain": %g,\'#\n'
        '#calc \'rain_day(data)\' \'"dailyrain": %g, \' #\n'
        '#calc \'wind_chill(data["temp_out"], data["wind_ave"])\' \'"chill": %.1f, \' #\n'
        '\n')
multi_topic = False

I hope that helps.

Tim

[samus-homebridge]

Tim, that's fantastic -- now receiving the mqtt message perfectly! Thank you

For anyone else reading this in the future: it seems the key is "auth = False" in the pywws config and dropping the TLS-related fields from the example given in the documentation.