orchestration

[erwanlenagard]

Hi,

Great tool ! Thank you very much for sharing it !
I use pepp on a daily basis and I run the same template of queries for numerous recordings. Currently, I need to store 1 config file / recording and then launch script one by one. It would be great to use a config file including several recording id that is called to execute the same script.

I've a similar issue with filters. For instance, i need to generate regularly 3 reports focused on France / UK and US. Countries may change according to the client needs. I use constants to restrict my queries, but I still need to store 3 different config files. Do you have any possibility to make more efficient tasks orchestration ?

Erwan / The Metrics Factory.

[haganbt]

Hi Erwan,

Glad your finding the tool useful. If I understand correctly, you could just setup multiple index values. See https://github.com/haganbt/pepp#index-credentials. This enables you to set multiple recordings and reference the appropriate one from each task. There is an example of where the index keys sits in /config/demo.json.

That key has been in for a while, but if you do pull the latest changes, there has been a major refactoring (v2.0) to support the new upcoming /task api and as a result there are two changes you need to make to your old recipes to make them work. See here and instructions on pulling the latest changes are here.

Let me know if that meets you needs.

Ben.

[haganbt]

@erwanlenagard - thinking about this further, do you need to generate 3 Tableau reports along with the output data?

[erwanlenagard]

Thanks. I already know this features but it doesn't fit my needs :
1 : you duplicate queries in the same config file, which is not very clean especially for maintenance
2 : number of recordings I need to extract in bulk change often
3 : recordings id may also change often

From my marketer perspective (I don't master all the stakes of developping with node.js), it would be more efficient to have attached to my config file, a csv or txt including all the recording ids in need to extract in bulk. When I execute the script, it reads line by line the .txt passing recording id to query pylon api.

I don't generate tableau templates directly from pepp. I parse all csv with Talend and load in mysql to add complementary informations such as the date of extraction, name of the country, Facebook population, etc..

If you want we can schedule a call next week to show you how we use pepp and what would be our key features for a next version of pepp.

[haganbt]

OK I see, that makes sense. Yes a call would be great. I will sync with Rakan and set something up for next week.

[erwanlenagard]

great ! :)