UnboundLocalError: local variable 'page' referenced before assignment

[amitsou]

Hello, I tried to set up the repo and execute the code, but I get the following error while trying to execute python main.py

Any ideas on how to fix it? Did you get any similar error?

Here's the terminal's output:

Scraping job pages in batches: 0%| | 0/2 [00:00<?, ?it/s]Error processing link https://www.upwork.com/freelance-jobs/apply/span-class-highlight-span-amp-Web-span-class-highlight-Development-span-Job_~021882918537097014024/?referrer_url_path=/nx/search/jobs: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
Error processing link https://www.upwork.com/freelance-jobs/apply/span-class-highlight-span-span-class-highlight-Agent-span-span-class-highlight-Development-span-for-Specific-Task-Automation_~021882917552830862542/?referrer_url_path=/nx/search/jobs: Page.content: Target page, context or browser has been closed
Error processing link https://www.upwork.com/freelance-jobs/apply/Create-span-class-highlight-span-Chatbot-for-website_~021882914047574399754/?referrer_url_path=/nx/search/jobs: Page.content: Target page, context or browser has been closed
Error processing link https://www.upwork.com/freelance-jobs/apply/Full-Stack-Promoter-Engineer-Mentorship_~021882912957754141448/?referrer_url_path=/nx/search/jobs: Page.goto: Target page, context or browser has been closed
Call log:

• navigating to "https://www.upwork.com/freelance-jobs/apply/Full-Stack-Promoter-Engineer-Mentorship_~021882912957754141448/?referrer_url_path=/nx/search/jobs", waiting until "load"

Error processing link https://www.upwork.com/freelance-jobs/apply/Personal-Executive-Assistant-with-Expertise-Notion-Google-Suite-and-span-class-highlight-span_~021882906097672208859/?referrer_url_path=/nx/search/jobs: Page.content: Target page, context or browser has been closed
Scraping job pages in batches: 50%|█████████████████████████████████ | 1/2 [00:07<00:07, 7.59s/it]Error processing link https://www.upwork.com/freelance-jobs/apply/Classifieds-Platform-Feature-Outline_~021882899721921811662/?referrer_url_path=/nx/search/jobs: Browser.new_context: Target page, context or browser has been closed
Error processing link https://www.upwork.com/freelance-jobs/apply/Chatbot-mongodb-database_~021882899586710033614/?referrer_url_path=/nx/search/jobs: Browser.new_context: Target page, context or browser has been closed
Scraping job pages in batches: 50%|█████████████████████████████████ | 1/2 [00:07<00:07, 7.68s/it]
Error processing link https://www.upwork.com/freelance-jobs/apply/Digital-Marketer-span-class-highlight-span-Tools_~021882896784014600968/?referrer_url_path=/nx/search/jobs: Browser.new_context: Target page, context or browser has been closed
Error processing link https://www.upwork.com/freelance-jobs/apply/Setup-Outbound-span-class-highlight-span-Voice-span-class-highlight-Agent-span-with-Twilio-and-HubSpot-Integration_~021882893385426262794/?referrer_url_path=/nx/search/jobs: Browser.new_context: Target page, context or browser has been closed
Error processing link https://www.upwork.com/freelance-jobs/apply/span-class-highlight-span-span-class-highlight-Agent-span-App-span-class-highlight-Development-span-for-Elderly-Support_~021882892301045054108/?referrer_url_path=/nx/search/jobs: Browser.new_context: Target page, context or browser has been closed
Traceback (most recent call last):
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/main.py", line 18, in
asyncio.run(automation.run(job_title=job_title))
File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/src/graph.py", line 97, in run
state = await self.graph.ainvoke({"job_title": job_title}, config)
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/venv/lib/python3.10/site-packages/langgraph/pregel/init.py", line 2014, in ainvoke
async for chunk in self.astream(
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/venv/lib/python3.10/site-packages/langgraph/pregel/init.py", line 1899, in astream
async for _ in runner.atick(
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/venv/lib/python3.10/site-packages/langgraph/pregel/runner.py", line 444, in atick
await arun_with_retry(
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/venv/lib/python3.10/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
return await task.proc.ainvoke(task.input, config)
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/venv/lib/python3.10/site-packages/langgraph/utils/runnable.py", line 499, in ainvoke
input = await step.ainvoke(input, config, **kwargs)
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/venv/lib/python3.10/site-packages/langgraph/utils/runnable.py", line 289, in ainvoke
ret = await self.afunc(*args, **kwargs)
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/src/nodes.py", line 43, in scrape_upwork_jobs
job_listings_df = await self.upwork_scraper.scrape_upwork_data(job_title, self.number_of_jobs)
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/src/scraper.py", line 56, in scrape_upwork_data
batch_results = await asyncio.gather(
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/src/scraper.py", line 51, in scrape_job_with_semaphore
return await self.scrape_job_details(browser, link)
File "/mnt/e/Code_base_Playground/Upwork-AI-jobs-applier/src/scraper.py", line 128, in scrape_job_details
await page.close() # Ensure the page is closed
UnboundLocalError: local variable 'page' referenced before assignment

[kaymen99]

Hi, thanks for reaching out and sharing the details of the issue you're experiencing.

The first error message:

The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

indicates that the required API key for using OpenAI models hasn't been set up. To resolve this, please ensure you add the API key for any language model (LLM) you want to use in your .env file.

Currently, this project uses gpt-4-mini and gemini-1.5-flash for demonstration purposes, but the system supports multiple LLMs. You can check out the relevant code in utils.py for reference on how the LLMs are configured.

[amitsou]

Hi, I set up the keys but the code still does not work. It's the exact same error. Thanks though

[kaymen99]

I wasn’t able to reproduce the error on my end, but based on your description, it seems like the page might not be fully loaded when the code reaches the finally block (in scrape_job_details function).

I’ve made some changes to the scraper class to address this. Please try running the scraper independently to see if the issue persists:

python scrape_upwork_jobs.py

[amitsou]

Could you please write a very detailed readme file on how you set up the project (maybe I'm missing something)? The project as an idea is a genius.