Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added LLM evaluator to example projects #1589

Merged
merged 6 commits into from
Jan 8, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions docs/guides/example-projects/batch-llm-evaluator.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
title: "Batch LLM Evaluator"
sidebarTitle: "Batch LLM Evaluator"
description: "This example project evaluates multiple LLM models using the Vercel AI SDK and streams updates to the frontend using Trigger.dev Realtime."
---

import RealtimeLearnMore from "/snippets/realtime-learn-more.mdx";

## Overview

This demo is a full stack example that uses the following:

- A [Next.js](https://nextjs.org/) app with [Prisma](https://www.prisma.io/) for the database.
- Trigger.dev [Realtime](https://trigger.dev/launchweek/0/realtime) to stream updates to the frontend.
- Work with multiple LLM models using the Vercel [AI SDK](https://sdk.vercel.ai/docs/introduction). (OpenAI, Anthropic, XAI)
- Distribute tasks across multiple tasks using the new [`batch.triggerByTaskAndWait`](https://trigger.dev/docs/triggering#batch-triggerbytaskandwait) method.

## GitHub repo

<Card
title="View the Batch LLM Evaluator repo"
icon="GitHub"
href="https://github.com/triggerdotdev/examples/tree/main/batch-llm-evaluator"
>
Click here to view the full code for this project in our examples repository on GitHub. You can
fork it and use it as a starting point for your own project.
</Card>

## Video

<video
controls
className="w-full aspect-video"
src="https://content.trigger.dev/batch-llm-evaluator.mp4"
></video>

## Relevant code

- View the Trigger.dev task code in the [src/trigger/batch.ts](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/trigger/batch.ts) file.
- The `evaluateModels` task uses the `batch.triggerByTaskAndWait` method to distribute the task to the different LLM models.
- It then passes the results through to a `summarizeEvals` task that calculates some dummy "tags" for each LLM response.
- We use a [useRealtimeRunsWithTag](https://trigger.dev/docs/frontend/react-hooks/realtime#userealtimerunswithtag) hook to subscribe to the different evaluation tasks runs in the [src/components/llm-evaluator.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/llm-evaluator.tsx) file.
- We then pass the relevant run down into three different components for the different models:
- The `AnthropicEval` component: [src/components/evals/Anthropic.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/evals/Anthropic.tsx)
- The `XAIEval` component: [src/components/evals/XAI.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/evals/XAI.tsx)
- The `OpenAIEval` component: [src/components/evals/OpenAI.tsx](https://github.com/triggerdotdev/examples/blob/main/batch-llm-evaluator/src/components/evals/OpenAI.tsx)
- Each of these components then uses [useRealtimeRunWithStreams](https://trigger.dev/docs/frontend/react-hooks/realtime#userealtimerunwithstreams) to subscribe to the different LLM responses.

<RealtimeLearnMore />
4 changes: 4 additions & 0 deletions docs/guides/example-projects/realtime-fal-ai.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ sidebarTitle: "Realtime image gen with Fal.ai"
description: "This example project generates an image from a prompt using Fal.ai and shows the progress of the task on the frontend using Trigger.dev Realtime."
---

import RealtimeLearnMore from "/snippets/realtime-learn-more.mdx";

## Overview

This full stack Next.js project showcases the following:
Expand Down Expand Up @@ -40,3 +42,5 @@ This video walks through the process of creating this task in a Next.js project.
allowfullscreen
/>
</div>

<RealtimeLearnMore />
11 changes: 11 additions & 0 deletions docs/guides/examples/fal-ai-realtime.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,17 @@ sidebarTitle: "Fal.ai with Realtime"
description: "This example task generates an image from a prompt using Fal.ai and shows the progress of the task on the frontend using Trigger.dev Realtime."
---

## GitHub repo

<Card
title="View the project on GitHub"
icon="GitHub"
href="https://github.com/triggerdotdev/examples/tree/main/realtime-fal-ai-image-generation"
>
Click here to view the full code for this project in our examples repository on GitHub. You can
fork it and use it as a starting point for your own project.
</Card>

## Walkthrough

This video walks through the process of creating this task in a Next.js project.
Expand Down
11 changes: 11 additions & 0 deletions docs/guides/frameworks/nextjs-webhooks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,17 @@ import VercelDocsCards from "/snippets/vercel-docs-cards.mdx";
- [A Next.js project, set up with Trigger.dev](/guides/frameworks/nextjs)
- [cURL](https://curl.se/) installed on your local machine. This will be used to send a POST request to your webhook handler.

## GitHub repo

<Card
title="View the project on GitHub"
icon="GitHub"
href="https://github.com/triggerdotdev/examples/tree/main/nextjs-webhooks/my-app"
>
Click here to view the full code for this project in our examples repository on GitHub. You can
fork it and use it as a starting point for your own project.
</Card>

## Adding the webhook handler

The webhook handler in this guide will be an API route.
Expand Down
11 changes: 11 additions & 0 deletions docs/guides/frameworks/remix-webhooks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,17 @@ description: "Learn how to trigger a task from a webhook in a Remix app."
- [A Remix project, set up with Trigger.dev](/guides/frameworks/remix)
- [cURL](https://curl.se/) installed on your local machine. This will be used to send a POST request to your webhook handler.

## GitHub repo

<Card
title="View the project on GitHub"
icon="GitHub"
href="https://github.com/triggerdotdev/examples/tree/main/remix-webhooks"
>
Click here to view the full code for this project in our examples repository on GitHub. You can
fork it and use it as a starting point for your own project.
</Card>

## Adding the webhook handler

The webhook handler in this guide will be an API route. Create a new file `app/routes/api.webhook-handler.ts` or `app/routes/api.webhook-handler.js`.
Expand Down
16 changes: 11 additions & 5 deletions docs/guides/frameworks/supabase-edge-functions-basic.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,6 @@ import NextjsTroubleshootingButtonSyntax from "/snippets/nextjs-button-syntax.md
import WorkerFailedToStartWhenRunningDevCommand from "/snippets/worker-failed-to-start.mdx";
import SupabaseDocsCards from "/snippets/supabase-docs-cards.mdx";

<Info>
The project created in this guide can be found in this [GitHub
repo](https://github.com/triggerdotdev/example-projects/tree/main/supabase).
</Info>

## Overview

Supabase edge functions allow you to trigger tasks either when an event is sent from a third party (e.g. when a new Stripe payment is processed, when a new user signs up to a service, etc), or when there are any changes or updates to your Supabase database.
Expand All @@ -36,6 +31,17 @@ This guide shows you how to set up and deploy a simple Supabase edge function ex
- [Create a Trigger.dev account](https://cloud.trigger.dev)
- [Create a new Trigger.dev project](/guides/dashboard/creating-a-project)

## GitHub repo

<Card
title="View the project on GitHub"
icon="GitHub"
href="https://github.com/triggerdotdev/examples/tree/main/supabase-edge-functions"
>
Click here to view the full code for this project in our examples repository on GitHub. You can
fork it and use it as a starting point for your own project.
</Card>

## Initial setup

<Steps>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,14 @@ import NextjsTroubleshootingButtonSyntax from "/snippets/nextjs-button-syntax.md
import WorkerFailedToStartWhenRunningDevCommand from "/snippets/worker-failed-to-start.mdx";
import SupabaseDocsCards from "/snippets/supabase-docs-cards.mdx";

<Info>
The project created in this guide can be found in this [GitHub
repo](https://github.com/triggerdotdev/example-projects/tree/main/supabase).
</Info>

## Overview

Supabase and Trigger.dev can be used together to create powerful workflows triggered by real-time changes in your database tables:

- A Supabase Database Webhook triggers an Edge Function when a row including a video URL is inserted into a table
- The Edge Function triggers a Trigger.dev task, passing the `video_url` column data from the new table row as the payload
- The Trigger.dev task then:

- Uses [FFmpeg](https://www.ffmpeg.org/) to extract the audio track from a video URL
- Uses [Deepgram](https://deepgram.com) to transcribe the extracted audio
- Updates the original table row using the `record.id` in Supabase with the new transcription using `update`
Expand All @@ -38,6 +34,17 @@ Supabase and Trigger.dev can be used together to create powerful workflows trigg
- [Create a new Trigger.dev project](/guides/dashboard/creating-a-project)
- [Create a new Deepgram account](https://deepgram.com/) and get your API key from the dashboard

## GitHub repo

<Card
title="View the project on GitHub"
icon="GitHub"
href="https://github.com/triggerdotdev/examples/tree/main/supabase-edge-functions"
>
Click here to view the full code for this project in our examples repository on GitHub. You can
fork it and use it as a starting point for your own project.
</Card>

## Initial setup

<Steps>
Expand Down
1 change: 1 addition & 0 deletions docs/guides/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Example projects are full projects with example repos you can fork and use. Thes
| Example project | Description | Framework | GitHub Repo |
| :--------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------- | :-------- | :---------------------------------------------------------------------------------------------------- |
| [Realtime Fal.ai image generation](/guides/example-projects/realtime-fal-ai) | Generate an image from a prompt using Fal.ai and show the progress of the task on the frontend using Realtime. | Next.js | [View the repo](https://github.com/triggerdotdev/examples/tree/main/realtime-fal-ai-image-generation) |
| [Batch LLM Evaluator](/guides/example-projects/batch-llm-evaluator) | Evaluate multiple LLM models and stream the results to the frontend. | Next.js | [View the repo](https://github.com/triggerdotdev/examples/tree/main/batch-llm-evaluator) |

## Example tasks

Expand Down
5 changes: 4 additions & 1 deletion docs/mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,10 @@
},
{
"group": "Example projects",
"pages": ["guides/example-projects/realtime-fal-ai"]
"pages": [
"guides/example-projects/realtime-fal-ai",
"guides/example-projects/batch-llm-evaluator"
]
},
{
"group": "Example tasks",
Expand Down
7 changes: 7 additions & 0 deletions docs/snippets/realtime-learn-more.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
## Learn more about Trigger.dev Realtime

To learn more, take a look at the following resources:

- [Realtime docs](https://trigger.dev/docs/realtime) - learn more about Realtime.
- [Batch Trigger docs](https://trigger.dev/docs/triggering) - learn more about Batch Triggers.
- [React hooks](https://trigger.dev/docs/frontend/react-hooks) - learn more about React hooks.
Loading