Add a command to generate missing mappings #254

chesterbot01 · 2024-07-30T20:35:37Z

Added a command /generate_mappings trigger the generation of taxonomy mappings.
See the ticket linked to the PR for details of how it works.

To tophat, set ENV["OPENAI_API_KEY"] to a valid OpenAI API Token, and then run bin/generate_missing_mappings. We will see data/integrations/google/2021-09-21/mappings/from_shopify.yml is updated and tmp/mapping_update_message.txt is created.

bin/generate_missing_mappings

chesterbot01 · 2024-07-30T22:13:55Z

@elsom25 I am not able to push large files (over 5M) like embedding cache to the remote in this repo (but in other repos I did not encounter such error),

Enumerating objects: 22, done.
Counting objects: 100% (22/22), done.
Delta compression using up to 11 threads
Compressing objects: 100% (13/13), done.
error: RPC failed; HTTP 400 curl 22 The requested URL returned error: 400
send-pack: unexpected disconnect while reading sideband packet
Writing objects: 100% (14/14), 1.56 MiB | 4.05 MiB/s, done.
Total 14 (delta 5), reused 0 (delta 0), pack-reused 0
fatal: the remote end hung up unexpectedly

do you know how to avoid such error?

Gemfile

bin/generate_missing_mappings

elsom25 · 2024-07-31T14:51:47Z

@chesterbot01 let's split thiss into two:

First, add the command to run locally. then we can test/vet quality
Once we're confident, we can hook it up to GH Actions

bin/generate_missing_mappings

chesterbot01 · 2024-08-06T22:09:13Z

re #254 (review)
Done.

elsom25 · 2024-08-07T12:17:47Z

app/commands/generate_missing_mappings_command.rb

+  # TODO: move to makefile; ruby should not need to think abuot service setup
+  def ensure_qdrant_server_running
+    qdrant_port = URI.parse(params[:qdrant_api_base]).port
+    return if system("lsof -i:#{qdrant_port}", out: "/dev/null")
+
+    command = "podman run -p #{qdrant_port}:#{qdrant_port} qdrant/qdrant"
+    pid = Process.spawn(command, out: "/dev/null", err: "/dev/null")
+    Process.detach(pid)
+    logger.info("Started Qdrant server in the background with PID #{pid}.")
+  end


Can we make this part of the Makefile instead? Or what challenges do you forsee with that?

Sounds good, done.

elsom25 · 2024-08-07T12:19:14Z

data/integrations/README.md

@@ -38,6 +38,13 @@ dist/{locale}/integrations/
 ### Mappings
 The primary concern of integrations are mappings. Mappings are a set of rules that help us convert to and from the Shopify taxonomy and the taxonomy of that integration.

+#### Use tooling to generate taxonomy mappings
+Automate your taxonomy mapping with this tool. Make sure `podman` is installed using `homebrew`, then run `make generate_mappings OPENAI_API_BASE=<base URI>  OPENAI_API_KEY=<API acceess token>` to create mappings. It uses AI to assist in generating accurate mappings and reducing manual effort.


Should we setup .env.development.local files to autoload things like this? 🤔

Introduced dotenv to override local environment variables and updated README and .gitignore to reflect the change.

elsom25 · 2024-08-07T12:19:47Z

data/integrations/README.md

+#### Use tooling to generate taxonomy mappings
+Automate your taxonomy mapping with this tool. Install dependencies using `make setup-mapping-generation-tool`, then run `bin/generate_missing_mappings` to create mappings. It uses AI to assist in generating accurate mappings and reducing manual effort.


My guess is this was a rebase slip-in: I'm assuming 🔥 ?

Thanks! Removed.

elsom25

Main things on my mind still, in importance:

qdrant run via makefile vs in ruby-script — or at least a good reason to keep it as-is
Any common practices/facilities around our prompt we should include with this change? Thinking about promptfoo and it's kin
Let's add https://github.com/bkeepers/dotenv? We can keep .env in as a template (but empty values), with both .env.development.local and .env.test.local git-ignored. README should signal about that at minimum, and maybe even our Makefile/dev.yml warn/assist here?
A wrapper in dev for generate_mappings like the other main commands.

No longer blocking

…Command style

…ids list

chesterbot01 · 2024-08-07T20:43:56Z

re

qdrant run via makefile vs in ruby-script — or at least a good reason to keep it as-is

moved the logic of ensure_qdrant_server_running to makefile

Any common practices/facilities around our prompt we should include with this change? Thinking about promptfoo and it's kin

The current prompt is the best we've tested during hackdays. We can set up promptfoo in the tools repo, request an expert-labeled golden dataset to run the benchmark, and select the best prompt to override the current one if necessary.

Let's add https://github.com/bkeepers/dotenv?

done.

A wrapper in dev for generate_mappings like the other main commands.

done.

elsom25 · 2024-08-08T13:23:50Z

Gemfile

@@ -22,6 +22,7 @@ gem "tty-option", "~> 0.3", require: false
 # generate taxonomy mappings
 gem "qdrant-ruby", require: "qdrant"
 gem "ruby-openai"
+gem 'dotenv', groups: [:development, :test]


This should be in the group below pls @chesterbot01

elsom25 · 2024-08-08T13:24:40Z

data/integrations/README.md

@@ -39,7 +39,15 @@ dist/{locale}/integrations/
 The primary concern of integrations are mappings. Mappings are a set of rules that help us convert to and from the Shopify taxonomy and the taxonomy of that integration.

 #### Use tooling to generate taxonomy mappings
-Automate your taxonomy mapping with this tool. Make sure `podman` is installed using `homebrew`, then run `make generate_mappings OPENAI_API_BASE=<base URI>  OPENAI_API_KEY=<API acceess token>` to create mappings. It uses AI to assist in generating accurate mappings and reducing manual effort.
+Automate your taxonomy mapping with the CLI-based tooling. Make sure `podman` is installed using `homebrew`, set up local environment variables OPENAI_API_BASE and OPENAI_API_KEY in `.env.development.local`:


We shuold have a .env files that contains something like:

OPENAI_API_BASE= OPENAI_API_KEY=

And then our instructions should be to cp/rename this file and fill it in

chesterbot01 force-pushed the add-generate-mappings-command branch 4 times, most recently from a231d89 to 79a174f Compare July 30, 2024 22:06

elsom25 reviewed Jul 30, 2024

View reviewed changes