Converting Shiny weather data with the Metadata processor: bugs and issues #687

CsillaVG · 2024-12-13T21:26:34Z

Hi Team,

There comes my concise write-up.
Thanks for all the work!
Best,
Csilla

Shiny weather data offers four output formats, out of which two could be relevant to UMEP.

1 .CSV

At the moment, under all my os-QGIS variations, loading the csv file is not possible. I receive the error of "If invalid data was detected such as strings or other non-numrical characters, these data points could result in that the MetdataProcessor will fail to create your UMEP-formatted inputdata." and the interface remains locked (without the possibility to assign the column names to various meteorological parameters).
This is most likely caused my the formatting of the file that starts with 11 lines of header lines that begin with a hash sign. Unfortunately, setting the 'Number of header lines' in the interface to 12 or higher does not resolve the issue. See attached screenshot.

Attached is a csv file for reference:
Chicago2020LT.csv

2 .EPW

The epw conversion works, apart from the hard-coded year of 1985 (noted earlier). While one could replace the year in the resultant file with the help of a text editor, the problem becomes more complex when the supplied Actual Meteorological Year (AMY) is a leap year. Put simply, the current code results in duplicate datetime labels, when leap year is fed in, eventually, causing a 24-hour shift in the data following the leap day. Fixing this can be challenging to those without solid Excel or scripting skills.

Attached is an UMEP-converted epw file for reference:
Chicago2020LTepw.txt

biglimp · 2024-12-16T09:06:56Z

Thanks Csilla,
Your first point is actually a bug that I just found will fix this. The 1985 hardcode should noew be fix (in the development release).

Response to #687 and minor fix in SUEWS DB Prepare

biglimp · 2024-12-16T14:52:04Z

Both issues should now be fixed. Check out new development release.

CsillaVG · 2024-12-16T16:34:23Z

Fantastic, thank you!

CsillaVG · 2024-12-16T19:12:12Z

The attempt to install the development release lands in a SyntaxError: (f-string: unmatched '[')

Here is the full error message ::
Couldn't load plugin 'UMEP-SuPy-QGIS3' due to an error when calling its classFactory() method

SyntaxError: f-string: unmatched '['
Traceback (most recent call last):
File "/Users/vuk/miniconda3/envs/fooqgis/share/qgis/python/qgis/utils.py", line 426, in _startPlugin
plugins[packageName] = package.classFactory(iface)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vuk/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/UMEP-SuPy-QGIS3/init.py", line 42, in classFactory
from .UMEP import UMEP
File "/Users/vuk/miniconda3/envs/fooqgis/share/qgis/python/qgis/utils.py", line 966, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vuk/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/UMEP-SuPy-QGIS3/UMEP.py", line 68, in
from .suews_database_manager.suews_database_manager import suews_database_manager
File "/Users/vuk/miniconda3/envs/fooqgis/share/qgis/python/qgis/utils.py", line 966, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vuk/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/UMEP-SuPy-QGIS3/suews_database_manager/suews_database_manager.py", line 56, in
from .utilities.suews_SS import setup_SUEWS_SS_creator
File "/Users/vuk/miniconda3/envs/fooqgis/share/qgis/python/qgis/utils.py", line 966, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vuk/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/UMEP-SuPy-QGIS3/suews_database_manager/utilities/suews_SS.py", line 165
QMessageBox.information(None, 'Succesful', f'New edit {spartacus_dict['Name']}, {spartacus_dict['Origin']} added to your local database')
^^^^
SyntaxError: f-string: unmatched '['

Environment ::
macOS Ventura 13.2
QGIS version: 3.40.1-Bratislava
Python version: 3.11.11
UMEP 4.3.2
UMEP for processing 2.0.35
SuPy version: 2024.5.24.dev

biglimp · 2024-12-17T07:29:04Z

Try now.

CsillaVG · 2024-12-17T15:54:01Z

Unfortunately, I got the same exact error.
I'll try to install QGIS into a 3.12 python environment and see if this helps with the issue, or just opens a new can of worms...
It does work under Windows 11.

biglimp · 2024-12-18T08:34:58Z

Ok, Let us know if you find a solution for Mac.

CsillaVG · 2024-12-20T02:12:20Z

Installing QGIS into a Python 3.12 environment on MacOS indeed solves the above SyntaxError. The development release of UMEP, with the Metadata processor, runs. However, 1985 is still wrote into the file. This is the case with Windows as well (I forgot to check the results the other day).
To double check the results, UMEPdev was downloaded and installed once again, and file conversion attempted multiple times. Still, no change regarding 1985.

Off: This set up comes with the unresolved issue of the faulthandler that becomes disabled due to conflicts between QGIS and MacOS (esp. M1), see discussions e.g. here and here. Nevertheless, one can still use QGIS and UMEP with minor caveats.

biglimp · 2024-12-20T08:19:47Z

I cannot replicate your error regarding 1985. I keep the year from the epw-file if I tick in the box "Keep year..."
Can you send me your epw-file so that I could try it.

CsillaVG · 2024-12-20T15:31:22Z

Sure. Here it comes. ERA5-ERA5n_2020LTtg.zip
The file was generated by Shiny Weather app. I have only added ground temperatures to the header.
Attaching the original epw as well.
ERA5-ERA5n_2020LT.zip

biglimp · 2024-12-23T08:26:11Z

Tried with your file, I got the correct year...

%iy  id  it imin   Q*      QH      QE      Qs      Qf    Wind    RH     Td     press   rain    Kdn    snow    ldown   fcld    wuh     xsmd    lai_hr  Kdiff   Kdir    Wd
2020 1 1 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.50000 89.00 7.00 100.80 0.00 0.00 -999.00 334.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 231.00
2020 1 2 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.90000 87.00 7.10 100.80 0.00 0.00 -999.00 338.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 226.00
2020 1 3 0 -999.00 -999.00 -999.00 -999.00 -999.00 1.10000 88.00 7.10 100.80 0.00 0.00 -999.00 345.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 210.00
2020 1 4 0 -999.00 -999.00 -999.00 -999.00 -999.00 1.20000 88.00 6.80 100.70 0.00 0.00 -999.00 343.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 188.00
2020 1 5 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.60000 88.00 6.60 100.70 0.00 0.00 -999.00 341.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 171.00
2020 1 6 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.50000 88.00 6.50 100.70 0.00 0.00 -999.00 338.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 160.00
2020 1 7 0 -999.00 -999.00 -999.00 -999.00 -999.00 0.80000 90.00 6.10 100.80 0.00 0.00 -999.00 329.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 192.00
2020 1 8 0 -999.00 -999.00 -999.00 -999.00 -999.00 1.40000 89.00 6.20 100.80 0.00 0.00 -999.00 341.00 -999.00 -999.00 -999.00 -999.00 0.00 0.00 209.00
...

CsillaVG · 2024-12-23T22:57:59Z

It must be a midwinter magic! :)
But seriously. I am still getting 1985s written into the file. Both on my MacOX and Windows machines. Attached is the screen capture from the mac.
Happy holidays!

epw_conversion.mov

biglimp · 2025-01-07T09:09:05Z

Truly strange...
Try uninstall and reinstall again by physically removing the UMEP plugin from your plugin library.

In the file metdata_processor.py your lines 260-270 should look like this:

    def epw2umep(self, met_old):
        met_new = np.zeros((met_old.shape[0], 24)) - 999

        # yyyy
        if self.yyKeep == 0:
            met_new[:, 0] = 1985
            met_new[met_old.shape[0] - 1, 0] = 1986
        else:
            met_new[:, 0] = met_old[:, 0]

        # hour
...

CsillaVG · 2025-01-08T20:00:26Z

I did that.

Downloaded yet again the development version.
Located the installation folder.
Observed the UMEP-SuPy-QGIS3 folder to be completely removed then rebuilt during the un- and re-installation process.
Restarted QGIS between the un- and re-installations.
Checked the relevant section of the code: it is as it should be.

Still, I cannot make UMEP to retain the year/formatting** of the data.

NOTE: Beforehand, I have removed all additional QGIS installs, except the LT release installed in another conda environment. To make sure it doesn't interfere with the one I use, I have uninstalled there all UMEP plugins.

** There is a loop (repetition of datetime values) over February 29 in the converted 2020 epw file.

biglimp · 2025-01-09T08:46:45Z

I have no clue why it does not work for you. I can also test on an Ubuntu installation next week but until then I suggest you just manually change the year column. You can also try to set in two print commands on lines 266 and 268 (see below) and see where the code goes.

    def epw2umep(self, met_old):
        met_new = np.zeros((met_old.shape[0], 24)) - 999

        # yyyy
        if self.yyKeep == 0:
            met_new[:, 0] = 1985
            met_new[met_old.shape[0] - 1, 0] = 1986
            print('1985')
        else:
            met_new[:, 0] = met_old[:, 0]
            print('User spec. year')

        # hour
...

You need to have your Python console open to see the prints

biglimp added a commit that referenced this issue Dec 16, 2024

Version 4.3.2

69a4d6d

Response to #687 and minor fix in SUEWS DB Prepare

biglimp closed this as completed Dec 16, 2024

biglimp mentioned this issue Dec 16, 2024

Download data (ERA5), comments #686

Closed

biglimp reopened this Dec 17, 2024

CsillaVG mentioned this issue Dec 20, 2024

Download data (ERA5) error, OSX binary install #683

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Converting Shiny weather data with the Metadata processor: bugs and issues #687

Converting Shiny weather data with the Metadata processor: bugs and issues #687

CsillaVG commented Dec 13, 2024

biglimp commented Dec 16, 2024

biglimp commented Dec 16, 2024

CsillaVG commented Dec 16, 2024

CsillaVG commented Dec 16, 2024

biglimp commented Dec 17, 2024

CsillaVG commented Dec 17, 2024 •

edited

Loading

biglimp commented Dec 18, 2024

CsillaVG commented Dec 20, 2024 •

edited

Loading

biglimp commented Dec 20, 2024

CsillaVG commented Dec 20, 2024 •

edited

Loading

biglimp commented Dec 23, 2024

CsillaVG commented Dec 23, 2024

biglimp commented Jan 7, 2025

CsillaVG commented Jan 8, 2025 •

edited

Loading

biglimp commented Jan 9, 2025

Converting Shiny weather data with the Metadata processor: bugs and issues #687

Converting Shiny weather data with the Metadata processor: bugs and issues #687

Comments

CsillaVG commented Dec 13, 2024

1 .CSV

2 .EPW

biglimp commented Dec 16, 2024

biglimp commented Dec 16, 2024

CsillaVG commented Dec 16, 2024

CsillaVG commented Dec 16, 2024

biglimp commented Dec 17, 2024

CsillaVG commented Dec 17, 2024 • edited Loading

biglimp commented Dec 18, 2024

CsillaVG commented Dec 20, 2024 • edited Loading

biglimp commented Dec 20, 2024

CsillaVG commented Dec 20, 2024 • edited Loading

biglimp commented Dec 23, 2024

CsillaVG commented Dec 23, 2024

biglimp commented Jan 7, 2025

CsillaVG commented Jan 8, 2025 • edited Loading

biglimp commented Jan 9, 2025

CsillaVG commented Dec 17, 2024 •

edited

Loading

CsillaVG commented Dec 20, 2024 •

edited

Loading

CsillaVG commented Dec 20, 2024 •

edited

Loading

CsillaVG commented Jan 8, 2025 •

edited

Loading