Update the OpenAPI spec to just follow the readme.io convention

[nikhilshinday]

Hey @sigpwned! thanks for doing this. While browsing the documentation for pinecone, I realized that the official OpenAPI spec is loaded onto the page at runtime. There are two sections:

and if you click into either, you'll see a request that looks something like:

where reference/<slug> refers to the endpoint that's being selected in the sub-menu.

When you look at the JSON body, it has an openapi definition embedded in it!

and the paths in that definition:

When you click on any of the endpoints in the second sub-menu, the OpenAPI definition changes (probably because the authorization scoping is at the project-level rather than the index-level).

Effectively, all you have to do to keep this up-to-date is run:

index_openapi_definition = requests.get('https://docs.pinecone.io/reference/query?json=on').json()['oasDefinition']
controller_openapi_definition = requests.get('https://docs.pinecone.io/reference/create_collection?json=on').json()['oasDefinition']

[nikhilshinday]

Created a PR, feel free to merge it, or don't!

#3

[sigpwned]

Ha! How about that! I'll dig in ASAP. Thank you for flagging!

[nikhilshinday]

of course!

[nikhilshinday]

also looking at my late-night PR - didn't mean to tread on you with the pyproject.yml author, was just mindlessly punching through the configuration list while using poetry init. It was more meant as a reference for what I was talking about above!

[sigpwned]

Of course! I didn't read anything at all into it. Thank you for this (excellent) suggestion! I will dig in today.

[walsha2]

@nikhilshinday @sigpwned just a heads up, there might be a reason the OpenAPI spec is not public just yet. I am seeing a ton of inconsistencies between the spec and the actual API responses. Some examples:

• Items are not defined as nullable when they should be
• Deprecated items not even returned in the response, yet listed in the spec as non-nullable
• Certain schemas, as defined in the spec, are completely wrong and do not match the actual response

Example

Here is the OpenAPI spec for indexMeta response as returned from the describe_index operation endpoint.

JSON Spec

"indexMeta": { "type": "object", "properties": { "database": { "type": "object", "properties": { "name": { "type": "string" }, "dimension": { "type": "string" }, "index_type": { "type": "string", "deprecated": true }, "metric": { "type": "string" }, "pods": { "type": "integer" }, "replicas": { "type": "integer" }, "shards": { "type": "integer" }, "pod_type": { "type": "string" }, "index_config": { "oneOf": [ { "$ref": "#/components/schemas/ApproximatedConfig" } ] }, "metadata_config": { "type": "object" }, "status": { "type": "object", "properties": { "ready": { "type": "boolean" }, "state": { "type": "string", "enum": [ "Initializing", "ScalingUp", "ScalingDown", "Terminating", "Ready", "InitializationFailed" ] } } } } } } },

Now compare that with the response you get when querying a newly created index:

curl --request GET \
     --url https://controller.us-west1-gcp-free.pinecone.io/databases/xyz \
     --header 'Api-Key: *****-*****' \
     --header 'accept: application/json'

{
  "database": {
    "name": "xyz",
    "metric": "cosine",
    "dimension": 1536,
    "replicas": 1,
    "shards": 1,
    "pods": 1
  },
  "status": {
    "waiting": [],
    "crashed": [],
    "host": "xyz-e795e37.svc.us-west1-gcp-free.pinecone.io",
    "port": 433,
    "state": "Ready",
    "ready": true
  }
}

• pod_type is not returned, yet is not listed as nullable
• index_config is not returned, yet is not listed as nullable
• metadata_config is not returned, yet is not listed as nullable
• dimension is listed as a string, yet the response is an integer 🤯
• status is listed as a nested property of database, yet it is clearly not 🤯
  • This one gets worse. host, which is super important, is not even in the spec
  • Neither is port, crashed, or waiting. The entire status object spec is wrong

There are several other examples, but this one was particularly egregious and wasted a ton of time.

Takeaway

All of this has basically made it impossible for me to generate any sort of reliable client via the OpenAPI spec, or even by relying on the docs. In short, the docs are out of sync with the actual responses being returned from the Pinecone API. Its actually quite bad.

I have resorted to simply hand coding a client by inspecting the actual responses from the API.

[walsha2]

@sigpwned FWIW, I took your hand coded yaml spec in this repo, converted it to json via the online swagger editor, and used that to generate a client. Testing it now.

In general, it seems to work much better, although I still see some inconsistencies that I just hand modified in the JSON spec. You captured some of the issues I described above - namely that status related inconsistency.

[sigpwned]

This is great work! If you see any specific issues in this spec, please let me know and/or open a PR. Obviously, I'd love to keep this as up-to-date as possible!

And @nikhilshinday, it looks like it's best not to use the "official" OpenAPI spec for now, since it's inaccurate. But thank you for finding it, and I'll keep an eye on it! If and when an official release happens, I'll do one last release here and point folks at the official version.