PRQL at Three Years-Dev Call on Sunday, 5Jan

[richb-hanover]

The first commit for PRQL was 1 January 2022. The current implementation has remained remarkably true to the original vision. Here's the README from the initial commit: https://github.com/PRQL/prql/blob/8ddc6e0dcdbccf88465650571c25c7fb4ba13f28/README.md

It has now been three years since that first commit. Perhaps it's a good time to reflect on our progress.

I propose we pick an upcoming Sunday to take 60-90 minutes to review our accomplishments, talk about work that is in the pipeline, and discuss "what's on the horizon". We could find at a convenient hour, perhaps the same time as the regular PRQL Fortnightly Dev Call.

My questions:

1. Would this be useful?
2. Sunday, 5 January 2025 is the next regularly-scheduled Dev Call at 10:30am PST, 12:30pm 1:30pm EST, 6:30pm CET. Would that date and time work? Should we look for a better date/time?
3. What topics would be relevant?

I look forward to talking soon. Thanks!

@max-sixty @aljazerzen @snth @vanillajonathan @eitsupi @PrettyWood @kgutwin @anybody-else

[aljazerzen]

That seems fitting. The development has slowed down a bit, so it might be time to rethink the roadmap and set the endgoal for PRQL 1.0.

The main question we talked about with @max-sixty is whether PRQL should focus on being a query language preprocessor or a full standalone language. The difference between the two is subtle, so it would be hard to explain here.

The regular dev call slot is ok with me.

[snth]

Thank you @richb-hanover for this initiative. I think it's a great idea to take stock.

The timeslot is not great for me but I will try to make it this week for this discussion.

I have quite a few reflections and will try to write some up ahead of time.

[richb-hanover]

@max-sixty @aljazerzen @snth @vanillajonathan @eitsupi @PrettyWood @kgutwin

I proposed this coming Sunday because it happened to be the next regularly-scheduled meeting time.

Some people have responded, some have not yet. I would prefer to have broad representation - if the timing presents problems, perhaps we could suggest times that could work. I'll start with good times for me:

• Any day of the week is possible with a few day's notice
• Daylight hours in EST (GMT -5hours) are preferable. The regular Dev Call is scheduled at 1:30pm local time which works fine for me

Thanks

[kgutwin]

I would love to attend; that day and time is somewhat challenging for me. I could make that time slot or something surrounding it any day Jan 6-11, or I could join any time between 1:30pm and 4pm EST on Jan 5.

[max-sixty]

I'll be on! (Have been in Europe, returning tomorrow!)

[richb-hanover]

It sounds as if Sunday, 5 January 2025 at 1:30pm EST, 10:30am PST, 6:30pm CET would work for everyone who has responded. Now I have to write up my thoughts, too... THANKS!

[snth]

Sorry, I only saw this now. Did this happen?

I'll be around at the usual Dev Call time in 45 minutes.

[max-sixty]

Yes I'll be on in 10 min!

[richb-hanover]

@max-sixty @aljazerzen @snth @vanillajonathan @eitsupi @PrettyWood @kgutwin

As preparation for tomorrow's Dev Call, permit me to propose this agenda to guide our discussion. We can use the first few minutes for "agenda bashing" to modify as needed.

• Agenda bashing - 5 minutes
  • Discuss items that should be included, removed, or modified
  • Appoint facilitator, note-taker, other administrative stuff
• The Best Things about PRQL - 15 minutes
  • Each participant takes 2 minutes to discuss their reasons for liking PRQL
• Enumerate Big Issues - 10 minutes
  • @aljazerzen and @max-sixty should discuss what's in the pipeline
  • Identify other issues but don't discuss in detail
  • Could be language design, troubleshooting, development tools, managerial concerns
  • Pick the three most important
• Discuss each of the top-three Big Issues - 45 minutes
  • Spend 15 minutes outlining them, highlighting problems, areas for further work, possible solutions
• What would a "Roadmap for PRQL 1.0" look like? - 15 minutes
  • Helps decide what we plan to do, and what we won't do

Administrative items:

• The meeting will be long: the items above already come to 1h 30m
• We should appoint someone to be the facilitator
• Ditto for a note-taker to summarize the meeting
• Discord Channel is at: https://discord.gg/2RUBzz3R3J
• Do we advertise this meeting ahead of time on Discord?

[max-sixty]

Appreciate the agenda! I think this is good!

I think we should try and keep it to 30-60 min (we aim for 30, we hit 60), so maybe we cut some of the time in big issues.

The primary thing we need to evaluate is the gap between what we have now and something that a critical mass of people actually use. Jotting down a few issues:

• It's very reliant on @aljazerzen at the moment, and to a lesser extent me. (I won't spend time saying thanks to each person, and want to freely discuss any disagreements without me sounding critical — so I'll say quickly: @aljazerzen has carried a lot of PRQL on his back, is the person most responsible for its success so far)
• There's a big tradeoff between "A new first-principles language for data transformations" and "a nicer way to express SQL". Historically @aljazerzen has built toward the new world, I've advocated for the more incremental world
• Some of the compiler is really complicated, to the extent that I struggle to understand parts. Those parts are also where the bugs are.
• It's harder than I expected to be able to provide great semantics without knowing what's a column vs. a table vs. a function. We never made progress on the "DB connection" part, which could provide prqlc with information about the schema. SQL binds very late, but that's part of the reason it's so difficult to use

There's still a great "market" there, as demonstrated by the developer excitement. OTOH we have lost momentum, and to the extent that @aljazerzen is more focused on other things, we've lost about half of our developer time and 80% of our compiler knowledge. So it's a challenge.

I would probably advocate towards initially compromising on some semantic purity, and attempting to be the best language for writing SQL. That would mean de-emphasizing things like loop, or different null handling. I'm not sure that makes it sufficiently simple, though; to be discussed.

If we can start a flywheel where we get users, which encourages developers, which encourages integrations, which gets users, then we can grow to more ambitious ends. That's been the successful feature of some of the other projects I maintain, such as xarray or to a lesser extent, insta.

Look forward to speaking in a moment.