2024-07-24 17:42:16 [nodemon] starting `ts-node src/services/queue-worker.ts`
2024-07-24 17:42:19 Web scraper queue created
2024-07-24 17:42:19 Connected to Redis Session Store!
2024-07-24 17:42:39 taking job 6b90b07a-06b5-493b-a774-cad5c2eb0861
2024-07-24 17:42:40 Failed to fetch robots.txt from https://www.organiclink.io/robots.txt
2024-07-24 17:42:15 ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-9.6.0.tgz
2024-07-24 17:42:18 LOGTAIL_KEY is not provided - your events will not be logged. Using MockLogtail as a fallback. see logtail.ts for more.
2024-07-24 17:42:19 Authentication is disabled. Supabase client will not be initialized.
2024-07-24 17:42:19 POSTHOG_API_KEY is not provided - your events will not be logged. Using MockPostHog as a fallback. See posthog.ts for more.
2024-07-24 17:42:40 [Playwright] Error fetching url: https://www.organiclink.io -> AxiosError: Request failed with status code 404
2024-07-24 17:42:40 Attempted to access Supabase client when it's not configured.
2024-07-24 17:42:40 Error logging proxy:
2024-07-24 17:42:40  Error: Supabase client is not configured.
2024-07-24 17:42:40     at Proxy.<anonymous> (/app/src/services/supabase.ts:45:17)
2024-07-24 17:42:40     at logScrape (/app/src/services/logging/scrape_log.ts:24:52)
2024-07-24 17:42:40     at scrapWithPlaywright (/app/src/scraper/WebScraper/scrapers/playwright.ts:107:20)
2024-07-24 17:42:40     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-07-24 17:42:40     at async attemptScraping (/app/src/scraper/WebScraper/single_url.ts:188:28)
2024-07-24 17:42:40     at async scrapSingleUrl (/app/src/scraper/WebScraper/single_url.ts:299:23)
2024-07-24 17:42:40     at async /app/src/scraper/WebScraper/index.ts:67:26
2024-07-24 17:42:40     at async Promise.all (index 0)
2024-07-24 17:42:40     at async WebScraperDataProvider.convertUrlsToDocuments (/app/src/scraper/WebScraper/index.ts:64:7)
2024-07-24 17:42:40     at async Promise.all (index 0)
2024-07-24 17:42:40     at async WebScraperDataProvider.processLinks (/app/src/scraper/WebScraper/index.ts:268:36)
2024-07-24 17:42:40     at async WebScraperDataProvider.handleCrawlMode (/app/src/scraper/WebScraper/index.ts:202:19)
2024-07-24 17:42:40     at async runWebScraper (/app/src/main/runWebScraper.ts:73:19)
2024-07-24 17:42:40     at async startWebScraperPipeline (/app/src/main/runWebScraper.ts:20:11)
2024-07-24 17:42:40     at async Queue.processJob (/app/src/services/queue-worker.ts:30:40)
2024-07-24 17:42:40 Attempted to access Supabase client when it's not configured.
2024-07-24 17:42:40 Error logging proxy:
2024-07-24 17:42:40  Error: Supabase client is not configured.
2024-07-24 17:42:40     at Proxy.<anonymous> (/app/src/services/supabase.ts:45:17)
2024-07-24 17:42:40     at logScrape (/app/src/services/logging/scrape_log.ts:24:52)
2024-07-24 17:42:40     at scrapWithFetch (/app/src/scraper/WebScraper/scrapers/fetch.ts:75:20)
2024-07-24 17:42:40     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-07-24 17:42:40     at async attemptScraping (/app/src/scraper/WebScraper/single_url.ts:207:26)
2024-07-24 17:42:40     at async scrapSingleUrl (/app/src/scraper/WebScraper/single_url.ts:299:23)
2024-07-24 17:42:40     at async /app/src/scraper/WebScraper/index.ts:67:26
2024-07-24 17:42:40     at async Promise.all (index 0)
2024-07-24 17:42:40     at async WebScraperDataProvider.convertUrlsToDocuments (/app/src/scraper/WebScraper/index.ts:64:7)
2024-07-24 17:42:40     at async Promise.all (index 0)
2024-07-24 17:42:40     at async WebScraperDataProvider.processLinks (/app/src/scraper/WebScraper/index.ts:268:36)
2024-07-24 17:42:40     at async WebScraperDataProvider.handleCrawlMode (/app/src/scraper/WebScraper/index.ts:202:19)
2024-07-24 17:42:40     at async runWebScraper (/app/src/main/runWebScraper.ts:73:19)
2024-07-24 17:42:40     at async startWebScraperPipeline (/app/src/main/runWebScraper.ts:20:11)
2024-07-24 17:42:40     at async Queue.processJob (/app/src/services/queue-worker.ts:30:40)
2024-07-24 17:42:40 WARNING - You're bypassing authentication
2024-07-24 17:42:40 Error sending webhook for team ID: undefined Failed to parse URL from 
2024-07-24 17:42:40 Falling back to fetch
2024-07-24 17:42:40 job done 6b90b07a-06b5-493b-a774-cad5c2eb0861