What's the equivalent of Python's Celery project for Java?

I am trying to find an equivalent of Celery project for Java environment, I have looked at Spring Batch, but are there any better alternatives for distributed task queues. Thanks.

Celery chunks inside chain

I would like to use chunks inside celery chain command. chain = task1.s(arg1) | task2.chunks(?,CHUNK_SIZE) | task3.chunks(?, CHUNK_SIZE) Basicly what I would like to do is run task1, chunk it's...

Measuring Celery task execution time

I have converted a standalone batch job to use celery for dispatching the work to be done. I'm using RabbitMQ. Everything is running on a single machine and no other processes are using the...

Python based asynchronous workflow modules : What is difference between celery workflow and luigi workflow?

I am using django as a web framework. I need a workflow engine that can do synchronous as well as asynchronous(batch tasks) chain of tasks. I found celery and luigi as batch processing workflow....

Celery execute task with a batch of messages

I want to send messages to celery and when it reaches let's say 100 messages i want celery to execute them in batches. This is a common scenario if I want to commit in batches to a database. For...

How to measure Celery task duration

Is there an easy way to measure the time it takes for a Celery task to be processed and successfully finished? There is a similar question but the first answer is ugly and unfortunately the...

What is the best design pattern for batch insertion using the Django REST Framework?

Background I have a Django app that allows record insertion via the Django REST Framework. Records will be periodically batch-inserted row-by-row by client applications that interrogate...

Check status of Celery worker

I have a project that uses Celery. I am periodically running into a scenario where my requests are making it to Celery but the tasks aren't being handed off to the workers, but rather the server...

how get django current user's name both in shell and in request

I have to update a "modified by" field in models with current user's name. It's very easy to get username in views with request like this: request.user.username However, when an model's instance...

How to implement a async grpc python server?

I need to call a celery task for each GRPC request, and return the result. In default GRPC implementation, each request is processed in a separate thread from a threadpool. In my case, the server...

Service to periodically load data to BigQuery

I noticed that the billing of our project on Google BigQuery is dominated by the cost of the resource StreamingBytes (90% of the total cost). This is due to the fact that we stream one record at a...

Python: Adding subscribers to mailchimp with bulk API V3

Im trying to write a function to create subscribers in mailchimp, but I get an error, and I can't understand what is wrong. Im using python-mailchimp (And yes, I have mye correct list-id and not...

How to share global variables between Celery's Periodic function and a Python's function

I am using Celery's periodic function to do batch updates of my database. The data to be inserted is stored in a global list by some other function of the same script. The problem is, whenever the...

Get million record from django with queryset is slow

I want to iterate all the objects of a table(Post) I am using below code: posts = Post.objects.all() for post in posts: process_post(post) process_post is a celery task which will run in...

How should I parallelize a mix of cpu- and network-intensive tasks (in Celery)

I have a job that scans a network file system (can be remote), pulls many files, runs a computation on them and pushes the results (per file) into a DB. I am in process of moving this to Celery so...

Celery - Remote chain randomly hangs

In a django project, as part of rather heavy import pipeline, i'm having to call three tasks that are part of a different codebase/application. Both applications point to the same AMQP broker and...

Horizontal scale of celery workers results in same processing time

We are experiencing an unexpected behaviour with celery when we scale horizontally with celery (increasing the number of our instances). The current processing time for our tasks it takes is just...

Integrating Celery to Flask application running in windows virtual environment

I have been creating a flask application to process GNSS data receive by the user. In the flask application data processing done in backend. Since GNSS data processing takes a long time currently...

Error in Celery: "ModuleNotFoundError: No module named 'tasks'"

When I run the batch file to enable Celery: python.exe -m celery -A tasks worker.celery -P eventlet -f celery.logs -l info An error occurs: >Traceback (most recent call last): File...

Celery group.apply_async().join() never returns

Consider the following script tasks.py: from celery import Celery from celery import group app = Celery() app.conf.update( broker_url='pyamqp://[email protected]//', ...

given a Redis Broker and a db-backend, where do Celery Task requests first get written to?

Is there anyway to interrogate either redis, celery or postgres, to track a task request before its completion? config: BROKER_URL = "redis://localhost:6379" CELERY_RESULT_BACKEND =...

How to release a batch of celery backend resources without stopping/pausing producer script?

I was going through the celery documentations and I ran across this Warning Backends use resources to store and transmit results. To ensure that resources are released, you must eventually call...

Redis memory usage continues to climb when using task.forget()

I have a mysql database which stores thousands of stock OHLC data for 2 years. Data is read from MySQL in the form of pandas dataframes and then submitted to celery in large batch jobs which...

Best Practice for Batch Processing with RabbitMQ

I'm looking for the best way to preform ETL using Python. I'm having a channel in RabbitMQ which send events (can be even every second). I want to process every 1000 of them. The main problem is...

Celery: enqueuing multiple (100-1000) tasks at the same time via send_task?

We quite often have the need to enqueue many messages (we chunk them into groups of 1000) using Celery (backed by RabbitMQ). Does anyone have a way to do this? We're basically trying to "batch" a...

Celery: Getting unexpected run_command() takes 1 positional argument but 318 were given error

I am attempting to run an async task and am getting an unexpected error: run_command() takes 1 positional argument but 318 were given. I have a list of commands that I want to run from a celery...

Can Celery pass a Status Update to a non-Blocking Caller?

I am using Celery to asynchronously perform a group of operations. There are a lot of these operations and each may take a long time, so rather than send the results back in the return value of...

Celery what happen to running tasks when using app.control.purge()?

Currently i have a celery batch running with django like so: Celery.py: from __future__ import absolute_import, unicode_literals import os import celery from celery import Celery from...

How to use Google Cloud Functions / Tasks / PubSub for Batch Processing?

We are currently using Rabbit MQ with Celery on some VMs for this: We have a batch of tasks we want to process in parallel (e.g. process some files concurrently or run some machine learning...

Celery: How to batch produce tasks?

I have a large loop to produce tasks: for i in range(1000): receiver.apply_async(args=(i), kwargs={}, exchange=topic_exchange, routing_key=topic_key) And I found there is a module...