Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OpenMapTiles vector map #4042

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

zdila
Copy link

@zdila zdila commented May 23, 2023

This pull request adds OpenMapTiles vector tiles among the featured layers. To ensure compatibility with the current stack based on Leaflet, it uses maplibre-gl-leaflet binding.
The main advantage for users is support for labels in multiple languages: users can easily switch by changing language preferences in settings.

196365626-243c978f-256c-4a53-a6a7-aee2d63f6c74

An official email with a link to this PR was sent, together with an edit of the “Featured tile layers” wiki page.

@tomhughes
Copy link
Member

I thought we had agreed that adding support for vector tile layers should be separate from adding any specific layer - certainly at the least I'd like them to be separate commits.

The API key should probably be in the configuration file as well, as with the thunderforest layers, rather than hardcoded.

What are all the 3D terrain bits about?

@zdila
Copy link
Author

zdila commented May 23, 2023

I thought we had agreed that adding support for vector tile layers should be separate from adding any specific layer - certainly at the least I'd like them to be separate commits.

Support for vector tiles is actually just adding maplibre-gl and @maplibre/maplibre-gl-leaflet to package.json.

If you agree, I can prepare such PR.

The API key should probably be in the configuration file as well, as with the thunderforest layers, rather than hardcoded.

I'll look at it.

What are all the 3D terrain bits about?

It is a link to a demo site that shows full capability of vector maps (map rotation, animation, 3D terrain) which is not possible with Leaflet and maplibre-gl-leaflet library.

@tomhughes
Copy link
Member

tomhughes commented May 23, 2023

So are you actually adding two layers here then? The main vector tiles layer and the 3D one? Only the 3D one doesn't seem to be global so I don't think it's going to pass OWG review.

I can't look at the other one because it's not actually working currently.

@zdila
Copy link
Author

zdila commented May 23, 2023

This PR adds only one layer - vector map with carto-like style. Most of the code is actually about updating the link URL in the attribution of this map so that it would point to the same view of the external 3D map demo.

@zdila
Copy link
Author

zdila commented May 23, 2023

You can see it live at http://osm.openmaptiles.org/

@tomhughes
Copy link
Member

Ah yes it does work over HTTP but all the other links were HTTPS which doesn't work.

I think the 3D stuff should be separated out - most of the code is only actually there to do that I think and I think that's going to be a lot more contentious that the rest of it.

@zdila
Copy link
Author

zdila commented May 23, 2023

Do I understand correctly that the issue we are talking about is the "3D map ⛰️." link in the footer?

We added it there to allow visitors to see the OSM in 3D. We would love to have it directly on osm.org. However, it is not technically possible because the site runs on Leaflet. Therefore, we added it at least as an external link.

@tomhughes
Copy link
Member

Yes that's what I'm talking about - it wouldn't be acceptable as a layer on the site anyway as it doesn't appear to cover the whole world and I think that also makes linking to it questionable which is why I'd rather separate that decision/discussion from the question of adding the vector tile layer to the site.

@zdila
Copy link
Author

zdila commented May 23, 2023

Can you please report an area that is missing terrain/map? Because I am checking it on my side and the coverage is global - even in the remote places.

@tomhughes
Copy link
Member

tomhughes commented May 23, 2023

Maybe I just don't understand how to operate it then... I clicked the bottom button to switch back to 2D mode, panned to my home area, then tried to switch back to 3D mode by pressing the same button again and nothing happened.

Specifically I was looking at https://labs.maptiler.com/showcase/osm-3d-terrain/#style=openstreetmap&lang=en&mode=3d&position=15.46/51.76264/-0.00868 which is 2D and I can't find any way to switch to 3D there?

@tomhughes
Copy link
Member

Oh if I hold on that button and move my mouse I can make it tilt and pan...

@zdila
Copy link
Author

zdila commented May 23, 2023

You can also drag the map with the right mouse button, or with Ctrl key pressed. Anyway, that area is not very hilly.

@gravitystorm
Copy link
Collaborator

Great to see progress on this, thank you for the PR.

The main advantage for users is support for labels in multiple languages: users can easily switch by changing language preferences in settings.

This is great! Do you have any guidance for us as to how it handles regional languages, e.g. pt-PT vs pt-BR? Or sr vs sr-Latn? And what happens if you pick a language that is not supported by GL-based renderings, like my (Burmese)?

I want to make sure that the translations work for all of the languages that our user interface works in, hopefully so that there's no mismatch between what you can pick in the UI and what then happens with this layer.

I also noticed on the wiki that it says "map labels translations from Wikidata" - can you clarify this? How does this interact with the translated labels available in OSM? If a contributor spots a mistake on the map, can they edit OSM to fix it, or do they have to edit something on Wikidata?

We added it there to allow visitors to see the OSM in 3D. We would love to have it directly on osm.org. However, it is not technically possible because the site runs on Leaflet. Therefore, we added it at least as an external link.

I don't think the layer attribution is the right place for linking to external sites to view other renderings.

If we want to link to external sites to view other renderings, which haven't been through the new layer approval process, then let's discuss this separately. And if we want to adapt the site to support alternative (i.e. non-leaflet) tech, then let's discuss that separately too. But I don't think such links should be part of the attribution for another layer, and I definitely don't think that should be part of this PR.

@zdila
Copy link
Author

zdila commented May 25, 2023

This is great! Do you have any guidance for us as to how it handles regional languages, e.g. pt-PT vs pt-BR? Or sr vs sr-Latn? And what happens if you pick a language that is not supported by GL-based renderings, like my (Burmese)?

Supported languages of OMT are documented at https://docs.maptiler.com/schema/planet/#languages

Comparison
OSM OMT Language
af Afrikaans
aln Gheg Albanian
am Amharic
ar ar Arabic
arz Egyptian Arabic
ast Asturian
az az Azerbaijani
ba Bashkir
be-Tarask Belarusian (Taraškievica)
be be Belarusian
bg bg Bulgarian
bn Bengali
br br Breton
bs bs Bosnian
ca ca Catalan
ce co Chechen, Corsican
cs cs Czech
cy cy Welsh
da da Danish
de de German
diq Zazaki
dsb Lower Sorbian
el el Greek
en-GB English (UK)
en en English
eo eo Esperanto
es es Spanish
et et Estonian
eu eu Basque
fa Persian
fi fi Finnish
fit Tornedalen Finnish
fr fr French
fur Friulian
fy fy West Frisian
ga ga Irish
gcf Guadeloupean Creole French
gd gd Scottish Gaelic
gl Galician
gsw Swiss German
he he Hebrew
hi hi Hindi
hr hr Croatian
hsb Upper Sorbian
hu hu Hungarian
hy Armenian
ia Interlingua
id id Indonesian
is is Icelandic
it it Italian
ja ja Japanese
ja-Hira Japanese (Hiragana)
ja-Latn Japanese (Latin)
ja_rm Japanese (Romanization)
ja_kana Japanese (Kana)
ka ka Georgian
kab Kabyle
kk-cyrl kk Kazakh (Cyrillic)
km Khmer
kn kn Kannada
ko ko Korean
ko-Latn Korean (Latin)
ksh Colognian
ku-Latn Kurdish (Latin)
ku Kurdish
la Latin
lb lb Luxembourgish
lt lt Lithuanian
lv lv Latvian
mk mk Macedonian
ml Malayalam
mo Moldovan
mr Marathi
ms Malay
mt Maltese
my Burmese
nb Norwegian Bokmål
nds Low German
ne Nepali
nl nl Dutch
nn Norwegian Nynorsk
no Norwegian
nqo N’Ko
oc oc Occitan
pa Punjabi
pl pl Polish
pnb Western Punjabi
ps Pashto
pt-BR Portuguese (Brazil)
pt-PT Portuguese (Portugal)
pt pt Portuguese
rm Romansh
ro ro Romanian
ru ru Russian
sat Santali
sc Sardinian
scn Sicilian
sco Scots
sk sk Slovak
skr-arab Saraiki (Arabic)
sl sl Slovenian
sq sq Albanian
sr-Latn sr-Latn Serbian (Latin)
sr sr Serbian
sv sv Swedish
ta ta Tamil
te te Telugu
th th Thai
tl Tagalog
tr tr Turkish
tt Tatar
uk uk Ukrainian
vi Vietnamese
xmf Mingrelian
yi Yiddish
yo Yoruba
zh Chinese
zh-CN Chinese (China)
zh-HK Chinese (Hong Kong)
zh-TW Chinese (Taiwan)

Currently if the language is not supported then local language is used - same as on Carto map - value from the name key of the tag.

I want to make sure that the translations work for all of the languages that our user interface works in, hopefully so that there's no mismatch between what you can pick in the UI and what then happens with this layer.

I will discuss it with my team. In any case current implementation doesn't fallback to the other languages in the list but I will implement it into this PR. Also falling back to main locale if sub-locale is not available (pt-BR -> pt).

I also noticed on the wiki that it says "map labels translations from Wikidata" - can you clarify this? How does this interact with the translated labels available in OSM? If a contributor spots a mistake on the map, can they edit OSM to fix it, or do they have to edit something on Wikidata?

We have decided to use names primarily from wikidata because there are more translations than in OSM. If the name is not in wikidata then one from OSM is used.

We added it there to allow visitors to see the OSM in 3D. We would love to have it directly on osm.org. However, it is not technically possible because the site runs on Leaflet. Therefore, we added it at least as an external link.

I don't think the layer attribution is the right place for linking to external sites to view other renderings.

If we want to link to external sites to view other renderings, which haven't been through the new layer approval process, then let's discuss this separately. And if we want to adapt the site to support alternative (i.e. non-leaflet) tech, then let's discuss that separately too. But I don't think such links should be part of the attribution for another layer, and I definitely don't think that should be part of this PR.

We will adjust it in this PR.

@pnorman
Copy link
Contributor

pnorman commented May 26, 2023

Currently if the language is not supported then local language is used - same as on Carto map - value from the name key of the tag.

What about when the language of the name tag is in a script Maplibre does not support?

@zdila
Copy link
Author

zdila commented May 26, 2023

What about when the language of the name tag is in a script Maplibre does not support?

Not sure if I understand you correctly, but for unsupported languages their local name will be displayed.

If you meant something else then please give me some example. You can try switching to (unsupported) language by setting it as primary in your browser (until I add a fallback to other languages your browser accepts - then try just with a single language configured).

@zdila
Copy link
Author

zdila commented May 26, 2023

Meanwhile I have removed 3D map link in the attribution.

@zdila
Copy link
Author

zdila commented May 26, 2023

The API key should probably be in the configuration file as well, as with the thunderforest layers, rather than hardcoded

done

@jachym
Copy link

jachym commented May 26, 2023

Maybe I just don't understand how to operate it then... I clicked the bottom button to switch back to 2D mode, panned to my home area, then tried to switch back to 3D mode by pressing the same button again and nothing happened.

Specifically I was looking at https://labs.maptiler.com/showcase/osm-3d-terrain/#style=openstreetmap&lang=en&mode=3d&position=15.46/51.76264/-0.00868 which is 2D and I can't find any way to switch to 3D there?

@tomhughes The map is 3D, but London is rather flat, sending better position:
https://labs.maptiler.com/showcase/osm-3d-terrain/#style=openstreetmap&lang=en&mode=3d&position=12.51/55.4648/-3.3975/-4.80/85.00

As metioned, you hold Ctrl key and can change the view angle.

@pnorman
Copy link
Contributor

pnorman commented May 27, 2023

Not sure if I understand you correctly, but for unsupported languages their local name will be displayed.

Maplibre doesn't support some scripts (e.g. Burmese, Lao, and other Brahmic scripts), so it can't display the local name.

@ZeLonewolf
Copy link

ZeLonewolf commented May 27, 2023

Maplibre doesn't support some scripts (e.g. Burmese, Lao, and other Brahmic scripts), so it can't display the local name.

That's not exactly accurate. The issue is that MapLibre doesn't support ligatures and combining characters correctly, and instead displays them anywhere from slightly to terribly wrongly. However, OSM-Carto also renders some of these scripts wrongly as the examples below will show (and should probably be documented if they're not already). Whether the "wrong" rendering makes the label unreadable or just slightly off will depend heavily on the script involved, and in my experience, it's the worst in Right-to-Left scripts.

This is what MapLibre displays for the United States label, in Lao:

image

However, it is supposed to look like this:
image

The issue is that the squiggly line above the second-to-last character is supposed to be directly over it, not off to one side. This issue is pervasive across all ligature/combiner scripts. While a Lao reader might still understand the script above, it would definitely look wrong, and this problem in other scripts results in mangled text that's absolutely unreadable to native speakers.

This is something which has always been broken in mapbox-gl and is a high priority for maplibre-gl to get right, and it's documented in maplibre/maplibre#193.

Examples of rendering the Lao capital of Vientiane, which should appear in local script as:

ວຽງຈັນ

Google Maps:
image

OSM Americana (based on MapLibre):
image

OSM Carto:
image

However, looking at these examples, it seems that osm-carto isn't actually any different from what MapLibre is producing. Both MapLibre and osm-carto have the same issue with the diacritic being shifted to the right slightly.

Let's take a look at Kathmandu, Nepal, which is supposed to appear as:

काठमाडौँ

Google Maps:
image

OSM Americana
image

OSM Carto:
image

Mandalay, Burma, perhaps?

Should be:

မန္တလေး

Google Maps:
image

OSM Americana:
image

OSM Carto:
image

Burmese is where we really start to see the problem, with that infinity-sign looking diacritic that's supposed to be under the second character coming in rather mangled. However, OSM-Carto also gets it slightly wrong by jamming the ligature up into the character when there's supposed to be space between them.

@LaoshuBaby
Copy link
Contributor

LaoshuBaby commented Jun 3, 2023

图片

It seems that this new vector map does not handle the display of Chinese very well. It displays the Traditional and Simplified varient at the same time, because a DWG member with an Asian cultural background combined the two varients in name:zh, and usually The user will choose zh-CN or zh-TW two locales (for example, my browser uses zh-CN, so you can see other UI translations are displayed in my locale), but the label text is not correct from zh-CN match to name:zh-Hans.


图片

Even if I forcibly specify locale=zh-TW, the name:zh value mixed with various variants will still be preferred as the display label text

@SomeoneElseOSM
Copy link

Re "... because a DWG member ... combined the two variants in name:zh" that sounds like a tagging issue that's best discussed elsewhere - perhaps somewhere at https://community.openstreetmap.org/ ?

@LaoshuBaby
Copy link
Contributor

LaoshuBaby commented Jun 4, 2023

Re "... because a DWG member ... combined the two variants in name:zh" that sounds like a tagging issue

Yes, use variants combined value is a tagging issue, we can discuss in other places. Whether this kind of edit is reasonable will not be discussed on this github issue @SomeoneElseOSM

My reply is about vector map's label text(name:zh) can't match browser's locale settings(zh-CN/zh-TW), There should be a fallback from name:zh-Hans/t->name:zh, I don’t know how the new vector map handles the ICU problem, is about technical.

style: "https://api.maptiler.com/maps/openstreetmap/style.json?key=" + OSM.MAPTILER_KEY
},
onAdd: function (map) {
var supportedLanguages = "am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh".split(",");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name:ja_kana and name:ja_rm are both documented on the wiki as being deprecated, although they are still in use to some extent. Is it a good idea to support these nonstandard language codes as part of the website?

style: "https://api.maptiler.com/maps/openstreetmap/style.json?key=" + OSM.MAPTILER_KEY
},
onAdd: function (map) {
var supportedLanguages = "am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh".split(",");
Copy link
Contributor

@1ec5 1ec5 Jun 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This list doesn’t include the languages added to OpenMapTiles openmaptiles/openmaptiles#1477, including zh-Hans and zh-Hant. I assume that’s because the MapTiler-hosted tiles are on OpenMapTiles v3.14, whereas this change is still waiting to be released. Is there a way to automate this list so that the website maintainers won’t have to remember to update it based on OpenMapTiles’ release schedule and MapTiler’s upgrade schedule?

@tomhughes
Copy link
Member

To be clear what languages are used in OSM tags are not directly relevant here - this is a mapping from web site locales to locales used by this layer.

How the layer maps from OSM tags to it's supported languages is a matter for the layer and the tools used to generate it's tiles from the OSM data.

@1ec5
Copy link
Contributor

1ec5 commented Jun 4, 2023

Understood, but those OpenMapTiles properties are directly tied to the deprecated OSM keys, so it would essentially be a way to enable OSM users to enter nonstandard codes in OSM preferences to see those tags’ value. (As far as I can tell, your language preference can include language codes that the website has no localization for.)

I guess it’s fine if these nonstandard codes end up being an obscure, undocumented edge case, but if they do get used, then that would be problematic for other components that rely on this same preference for localization, such as iD.

@Triomphe346
Copy link

How the layer maps from OSM tags to it's supported languages is a matter for the layer and the tools used to generate it's tiles from the OSM data.

I see on their own demo that they could correctly handle different Chinese variants (attach sample is Simplified Chinese only demo with 'zh-CN' browser setting).

Demo of MapTiler in Simplified Chinese

It seems that they do not explicitly document this. If it is possible to support zh-Hans/zh-Hant variants disregarding the listed supported language code? (may need mapping zh-CN browser setting to zh-Hans in MapTiler & zh-HK/zh-TW browser setting to zh-Hant in MapTiler manually) @tomhughes

@@ -11,6 +15,40 @@ L.extend(L.LatLngBounds.prototype, {
}
});

if (OSM.MAPTILER_KEY) {
L.OpenMapTiles = L.MaplibreGL.extend({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arabic and Hebrew would be broken unless this code also installs the right-to-left text plugin. (Note that it only supports these two languages, not other languages in the same scripts or other right-to-left scripts.)

@1ec5
Copy link
Contributor

1ec5 commented Oct 2, 2023

In my opinion any style that uses Wikidata names even if OSM name tags are provided should not be included (note, if name:LANG tag is not present then using that language label from Wikidata is not covered by that).

That’s the case we’re talking about here, if I’m not mistaken.

(also, note that Wikidata imported data from Wikipedia which in turn encouraged sourcing geographic data from Google Maps which is separate part of a quagmire)

Let’s not turn this PR into a detailed rundown of Wikidata’s faults, as though the style brings in more Wikidata content than it actually does. Besides, Google Maps uses Wikidata for labels, not the other way around.

@zdila
Copy link
Author

zdila commented Apr 29, 2024

In the recent release of OpenMapTiles 3.15, the Wikidata translations have been switched off: https://maptiler.com/news/2024/04/openstreetmap-data-prepared-for-advanced-cartography/

Are there any other blockers for accepting this PR?

@tordans
Copy link
Contributor

tordans commented Apr 30, 2024

Can we test the current state somewhere?
https://osm.openmaptiles.org/ is down.
Maybe Tom can deploy this branch on one of the testing servers?

@zdila
Copy link
Author

zdila commented Apr 30, 2024

@tordans https://osm.openmaptiles.org/ - I've just started it again.

@jmaspons
Copy link

Can the language selector be added as a combobox at the map layers panel? Would be more discoverable and selectable for users without osm account.

@tomhughes
Copy link
Member

Can the language selector be added as a combobox at the map layers panel? Would be more discoverable and selectable for users without osm account.

There are already separate issues for adding a language selector for non-logged in users.

@LaoshuBaby
Copy link
Contributor

the Wikidata translations have been switched off

Is it possible to disable automatic transliteration of non-Latin names?

@matkoniecz
Copy link
Contributor

I think it is fine? In the same way as it would be fine to host nondefault layer showing names in Korean/Polish/Japanese?

@jachym
Copy link

jachym commented Oct 10, 2024

Any resolution regarding this PR? Can we have this PR merged and closed, I believed, we addressed the concerns and issues of the community.

@ZeLonewolf
Copy link

New layers are approved by the Operations Working Group, which has not happened yet.

It was last discussed at their May meeting. I do not see that it has been on an OWG meeting agenda since.

@ZeLonewolf
Copy link

This PR was discussed at the 10/31 OWG meeting according to the public minutes.

@AntonKhorev
Copy link
Collaborator

Commits such as "fix lint issues" 8887671 shouldn't exist, see CONTRIBUTING.md#pull-requests

@pnorman
Copy link
Contributor

pnorman commented Dec 9, 2024

This is awaiting a response from MapTiler

@AntonKhorev
Copy link
Collaborator

Still has "fix lint issues".

@zdila zdila marked this pull request as draft February 4, 2025 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.