Open API data for Nottingham Public Transport

[jamesmacwhite]

Hi @simonprickett

A very Interesting project you created with the scraping of the NCT website to format it into a JSON API.

I too have been on a quest to understand where the live timing data for Nottingham Public Transport comes from. A potential project I wanted to look at was displaying the latest times for public transport at certain times on digital signage within Nottingham College (my employer) at our campuses, but the lack of reliable/official API is somewhat of a barrier!

A long time ago there was an open API for the NET Tram under robinhood.arcticapi.com, this went NXDOMAIN a few years ago, credit to @mrrothe, who did some DNS analysis of the mobile app and came across it and created a real-time display as a PoC, although the benefit of this, was that the API had a specification and had production usage: https://github.com/mrrothe/Realtime-Tram-Display. Suprisingly, it was completely open with no authentication at all, it was however just for NET Tram data, but seemed to be the real data matching the stop displays. I even used it for a time, to check when Trams were coming before leaving the office, back when Nottingham College had a Beeston campus on HIgh Road.

For NCT, they announced in 2019 around supporting Open Data initivates with various data sources available, but not the real-time info. I did potentially come across a GOV.UK open data API: https://www.travelinedata.org.uk/traveline-open-data/nextbuses-api/, it is unclear if NCT real-time data is present, but I was potentially going to enquire as it appears to free for a certain amount of requests in a 6 month period, the usage rates seem generous too. In theory it covers operators across Great Britian, so Trentbarton and others too, but I don't know if real-time is there, as the API access is behind authentication, which you have to fill out a form on to access.

I don't know if you've considered loading the mobile apps on a mobile device which is routed through a proxy where you can MITM the traffic to see if there's any interesting API related calls that might provide further clues on how this data is being sourced? I was going to have a look potentially. Given the various bus stops in Nottingham now display multiple operators under a single stop, I suspect that the Robin Hood Network must have some form involvment that covers all major operators, but the open API availability is unlikely.

I could never find anyone in the know who could provide information around official API information or how it works.

From general press releases from Transport Nottingham, there's small clues on how the infrastructure behind it, might work, but just very small bits: https://www.transportnottingham.com/projects/upgrading-real-time/

[jamesmacwhite]

Some very low investigation, but interestingly that domain for the NET Tram that was NXDOMAIN, NCT, has their own: nctx.arcticapi.com, judging by the requests made by the app. I need to potentially MITM the traffic to see full requests, which is tricky, given newer Android app level changes.

https://nctx.arcticapi.com/operator for example, returns data.

Interestingly arcticapi.com clearly seems to be tied to the wider network.

[jamesmacwhite]

BIngo!

https://nctx.arcticapi.com/network/stops/3390FO07/visits

There's your API data hiding away! Following the original work of @mrrothe, the API structure is very similar if not the same.

The API is open with no authentication like the previous NET Tram one. Whether or not it is meant to be consumed by external applications, I guess is the other question. In theory I'd imagine if it was an issue, there would be authentication on it? Who knows!

[jamesmacwhite]

Plenty of API data, here's the live disruptions data: https://nctx.arcticapi.com/network/disruptions, which is possibly very useful for a use case I've been looking at.

[simonprickett]

Hi there - that NCT API is provided by a company called Passenger Group and isn't per se public (I've been trying to shame NCT into making it so). Someone from Passenger contacted me when I first started Tweeting about this project and they did kindly provide a back door into it for me to look at but I can't use it for real things... hope NCT really "gets" open data sometime.

There's definitely something behind the real time displays at the bus stops that also includes trentbarton and Stagecoach services, not sure what.

I've also had a go at the tram people a few times on Twitter... their app doesn't even show real time departures, yet magically the signs at the stops can. They really need to have a proper API.

[simonprickett]

Here's a summary of what Passenger told me about the arcticapi URLs:

You asked about an API. We don't have an official public API, and we don't tend to tell people about this, but we were impressed so I wanted to let your know that our json endpoint for stop departures currently does not require authentication and so is open for you to use for the moment

I can't promise we won't change it, and we're likely to be adding authentication here at some point.

[simonprickett]

BTW if you did want to use my API feel free - it's nowhere near its resource limits on Cloudflare so it won't cause me any issues.

[jamesmacwhite]

Interesting information! At the moment the API looks pretty open and if very much like the NET Tram one, I have no idea what domain that is under now, although it was interesting it was called "robinhood" and not "nettram" or similar. I will potentially inspect the mobile app and see if it's querying another arcticapi endpoint,.

For example you can get the same data behind the service disruptions page on the NCT website, which is likely the most useful for our project/uses. Getting stop data is cool, but we've got a lot of students from different areas in the city, so listing them all, not easy or relevant! I do however appreicate the effort you've put in on this project. It was a fun read of your blog.

I'd be cool if they added authentication, providing they would allow access through API key/OAuth. Rate limiting, fair use, maximum requests in a timeframe, would be happy to play ball under guidelines providing it. I can guess if too many external services start calling this API, they might move on doing that.

[jamesmacwhite]

Just for further info, I came across http://nottingham.tramlive.co.uk/ for tram times, as it uses AJAX it is easy to see the HTTP requests being made for the stop data e.g. http://api.tramlive.co.uk/api/N17 which is Forest Recreation Ground, looking at the API data it does not appear to be real-time. I suspect this data is coming from the NextBuses API which is being provided by the this open dataset: https://www.travelinedata.org.uk/traveline-open-data/traveline-national-dataset/.

I believe National Rail Enquiries provides real-time data through something called Darwin, which is openly accessible, providing you register an account with personal/company information and be approved.

It is frustrating though, because so far I'm up to three different entities in terms of accessing different datasets for different services.

1. Passenger Group which now appears to be just for NCT.
2. Traveline for the NextBuses API which may or may not have real-time data but possibly covers all major operators in the city.
3. National Rail Enquiries for real-time train information at different stations which in this case would be for Nottingham Station.

Using the mobile apps for clues/possible APIs is actually a bit more of challenge given use of TLS and MITM on app traffic is rather difficult given Google's more recent changes to Android, I mean obviously I understand why as normally a MITM on SSL/TLS is bad news, although possible running an Android Emulator may be an alternative, given the need to mess with a lot of root/high level stuff to achieve it.

Haven't looked at Trentbarton, I vaguely remember there being a Hugo app for live times, which still seems to be working and displays other transport providers, other than Trentbarton so that must be also be provided by another party that's not just itself.

Yay for reverse engineering with no information what so ever!

[simonprickett]

Sadly it is a possibility that due to the council's financial situation live bus displays may go away (and I guess the system behind them too). See https://twitter.com/nct_buses/status/1746584770492891171

[jamesmacwhite]

I heard about this. I understand Nottingham's financial situation is pretty dire with regard to the council's financial hole, but it seems a backwards proposal, they even said alternatives within that proposal was to print timetables again, so that's not exactly saving money or sustainable either.

[simonprickett]

Yeah they're raising car parking costs on street and in city council car parks, while also talking about closing the two big bus based park and ride locations to save money.

[simonprickett]

@jamesmacwhite FYI my service is broken at the moment due to changes in the underlying HTML on NCT's website.

I'm addressing it here:

#9

[jamesmacwhite]

Thank you, I've ended up sticking with just travel disruption data from NCT, rather than live times, as there's far too many journey variables to display accordingly that's relevant for the 10 campuses at Nottingham College.

I have found various different APIs and services from different providers during my research, I even had a contact at Trentbarton kindly provide how the Nottingham real-time data for operators works and where they all feed into.

Happy to drop you an email with the details if you find it useful.

[simonprickett]

FYI @jamesmacwhite you might want to have a read of this thread. https://x.com/acsnottingham/status/1757081554260471950

[jamesmacwhite]

@simonprickett Interesting stuff! Inspecting the mobile app with HTTPS interception the endpoints of interest are:

https://nctx.arcticapi.com/ticketing/topups - Lists all ticket products

With an example ticket product (Single Help for Households):

https://nctx.arcticapi.com/ticketing/topups/736e399f-a651-4c25-aa17-4c95c2ac1470

Or another example (City Zone):

https://nctx.arcticapi.com/ticketing/topups/1e41e45d-43dc-4005-afe1-d0cccf2c4be1

It's great that real-time fare data is something that is implemented in the BODS specification.

[jamesmacwhite]

@simonprickett

In response, officers managed to change the proposals in "small ways", including finding money to retain Linkbus services, including Easylink, keeping real-time information at bus stops as well as saving the Queens Drive and Racecourse Park and Ride sites from closure.

https://www.bbc.co.uk/news/uk-england-nottinghamshire-68288969

Looks like Nottingham City Council will be keeping the real-time stops alive!