Repository navigation
Genserver time out crash in long-running pipeline #230
Description
I wrote a crawler that crawls media galleries to download them and download files uing HTTPoison in a custom pipeline. This seems to work fine for small files as they download quickly but if I try to crawl a really big file like a video, I will almost always get this crash:
12:49:51.241 [error] GenServer #PID<0.471.0> terminating
** (stop) exited in: GenServer.call(Crawly.DataStorage, {:store, KemonoCrawler, %{...}, 5000)
** (EXIT) exited in: GenServer.call(#PID<0.468.0>, :stats, 5000)
** (EXIT) time out
(elixir 1.14.0) lib/gen_server.ex:1038: GenServer.call/3
(elixir 1.14.0) lib/enum.ex:975: Enum."-each/2-lists^foreach/1-0-"/2
(crawly 0.14.0) lib/crawly/worker.ex:173: Crawly.Worker.process_parsed_item/1
(crawly 0.14.0) lib/crawly/worker.ex:50: Crawly.Worker.handle_info/2
(stdlib 4.1) gen_server.erl:1123: :gen_server.try_dispatch/4
(stdlib 4.1) gen_server.erl:1200: :gen_server.handle_msg/6
(stdlib 4.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: :work
State: %Crawly.Worker{backoff: 10000, spider_name: KemonoCrawler, crawl_id: "e8e7bb02-465f-11ed-85df-3a9adacac61c"}
Is there a setting I can tweak to allow my pipeline to run for much longer? I didn't see anything easily tweakable in the config.
Reacted by Robert Shecter
Activity
Ziinc commentedon Nov 26, 2022
I would suggest firing off a Task instead or moving such heavy data processing to a separate process tree. Pipelines was designed for lightweight data processing.