Extend contributing.md with a bird's eye view of server and services

[tordans]

Problem

It would be great to have a section in CONTRIBUTING.md that gives a bird's eye view about servers and services of the OSM website.

This came up for me recently when…

• we talked about if there is a staging system to test out Bid for UserMute feature osmfoundation/ewg_bidding#8
• I commented on What should we do about note comments? #3831 (comment) and was wondering if there is a queuing system

I am sure is information on this elsewhere, but I suggest to collect and link it as a section "Overview of deployment, server and services" at the end of CONTRIBUTING.md

If we collect some links and info in comments here, I can create a draft for this section.

A few questions I think this section could answer:

• What is the production server architecture of the website (# of application server, deployment process)
• Is there a queuing system / which and some relevant specs
• Is there a system in place to use Action Cable? AFAIK we don't but could we?
• Relevant notes on the database server
• Relevant notes on the staging / testing server
• AFAIK there is not dev > staging > production deployment but the staging instances are used as needed, right?

---

Site note: I first thought this would be something to put into an ARCHITECTURE.md https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html which became a kind of standard for some projects. However, it looks like this doc would be more a of a code architecture overview which we IMO do not need due to rails standards. This issue is more about the system architecture and related processes.

Description

No response

Screenshots

No response

[pnorman]

A lot of this seems out of scope for the website. When developing a feature it shouldn't matter how its deployed unless you hit resource constraints, which is fairly rare.

We could use better documentation on the deployment in chef in the relevant cookbooks, but it would be written to help users get oriented in the chef code, not this code.

[tomhughes]

I tend to agree with @pnorman that this is not the place for details of the OpenStreetMap deployment of this code.

[tordans]

I disagree that the questions I raised are irrelevant for development of features. I even provided two examples where they came up recently.

I am sure most info can be found somewhere. For example I saw a list of hardware once. And the cookbooks are somewhere as well. I could research all this myself, but maybe someone who knows where to look can share the links here?

[tomhughes]

Well your position seems to be that development should be constrained by the current operational environment rather than that the operational environment should be adjusted to support what the code requires.

Of course the reality probably lies somewhere between the two but we wouldn't want people to walk away without talking to us just because the current setup doesn't support what they want to do.

There is also of course the question of other users of this code base - while openstreetmap.org is the primary user it's certainly not the only one.

[gravitystorm]

• What is the production server architecture of the website (# of application server, deployment process)

I don't actually know this. Although I'm a maintainer it's not really relevant to the codebase or the work we do on it. Last time I was involved in OWG matters there were 6 machines running rails (and some? with cgimap) a primary database and I think 2 read-only database replicas. But that was years ago and I think it has changed substantially since. But I haven't needed to know at any point, since it's not relevant to pretty much any of the development on the codebase, and as Tom pointed out it's different for different deployments too.

• Is there a queuing system / which and some relevant specs

We use the standard ActiveJob abstraction layer, with delayed_job_active_record as the implementation. I don't know any stats on how many jobs per day are processed, I expect it's a substantial number.

• Is there a system in place to use Action Cable? AFAIK we don't but could we?

At the moment we aren't using ActionCable for anything, but I suspect it will be used at some point in the future, likely as an implementation detail for Turbo.

• Relevant notes on the database server

It's PostgreSQL, but I don't know anything else about the database server. Other than it's huge.

• Relevant notes on the staging / testing server

We use Github Actions for testing. We don't have any test servers i.e. servers running automated tests or for QA teams.

We don't have staging servers, in the typical use of the term (i.e. as the final step before production). OSMF has a development server, and a small number of branches and/or configurations have been deployed on that, but it's very rarely used (compared to the amount of PRs we handle - I'd say once every 2-3 years a branch is published there) and I've never used it.

If you are curious you can see the list of deployments on the OSMF dev server at https://apis.dev.openstreetmap.org/

I'm happy to answer any other questions!

[tomhughes]

As @gravitystorm says we do have the ability to deploy a branch for testing on the dev server though it's mostly only used for large changes we want to be able to give them more testing or invite comment on UI changes etc.

You're welcome to ask for a branch to be added there though, or open a PR against chef to add one to https://github.com/openstreetmap/chef/blob/8cb84a3535943f8e24c9ff187dd90ceb83316ae9/roles/dev.rb#L110.

[mmd-osm]

When you're developing features on Rails only without much external dependencies, you can do a lot of testing locally. Things get much more involved once you start integrating with other components.

Speaking of other components, we have a few external applications with direct read and/or write access to some APIDB tables. The chef repo would be a good starting point to learn more. Any changes to the layout of these db tables should be carefully reviewed with externals apps in mind.

Last time I was involved in OWG matters there were 6 machines running rails

We're down to 3 machines, web backend servers are no more. (I'm only counting spike-0[678], since spike-0[123] look rather idle these days)

[tordans]

Thanks for all the input here. Let's close this for now. If someone is interested in knowing more, we have a reference here … and can update the Docs if needed.