Django multiprocessing and database connections

Background: I'm working a project which uses Django with a Postgres database. We're also using mod_wsgi in case that matters, since some of my web searches have made mention of it. On web form...

Django Celery: Execute only one instance of a long-running process

I have a long-running process that must run every five minutes, but more than one instance of the processes should never run at the same time. The process should not normally run past five min,...

celery task_success with sender filter

I am trying to get the sender filter working e.g. @celery.task def run_timer(crawl_start_time): return crawl_start_time @task_success.connect def run_timer_success_handler(sender, result,...

Using HTML templates to send emails in python

I'm trying to write a separate mail service, which is decoupled with our Flask application. I'm looking for a way to send welcome emails when users first log into our Flask application. I'm using...

What's the difference between Docker Compose vs. Dockerfile

I have been reading up and learning about Docker, and am trying to correctly choose the Django setup to use. So far there is either: Docker Compose or Dockerfile I understand that Dockerfiles are...

Mock side effect only X number of times

I have a celery retry task that I would like to test that it retries until successful. Using mock's side_effect, I can fail it for a set number of executions and then passing None, clear the side...

Check status of Celery worker

I have a project that uses Celery. I am periodically running into a scenario where my requests are making it to Celery but the tasks aren't being handed off to the workers, but rather the server...

Connect new celery periodic task in django

It's not a question but help to those who will find that the declaration of periodic tasks described in celery 4.0.1 documentation is hard to integrate in...

on building docker image level=error msg="Can't close tar writer: io: read/write on closed pipe"

time="2017-10-27T07:39:20Z" level=error msg="Can't add file /var/app/current/app/content_classifier/forest.pickle to tar: io: read/write on closed pipe" time="2017-10-27T07:39:20Z" level=error...

how can i pass argument to celery task?

this is my celery task def task_a(arg1, arg2, arg3, arg4) and i call the task like following arg1 is list and arg2,3,4 is integer task_a.apply_async(arg1, arg2, arg3, arg4) and i got an error...

Python run_in_executor and forget?

How can I set a blocking function to be run in a executor, in a way that the result doesn't matter, so the main thread shouldn't wait or be slowed by it. To be honest I'm not sure if this is even...

Using multiple backends with Celery python

I am looking for some help on using Celery with multiple backends. Currently, with the knowledge I have, I can use either RabbitMQ or Redis as the backends but I could not find much information on...

Get Image Field Absolute Path in Django Rest Framework

I've a periodic celery task which needs to store representation of a object in a specific json field. Here is the simplified model structure. Parent <-- ChildWrapper <-- Child Image So basically...

How to save list of objects using bulk_update method in Django?

I have a loop iterating over objects and adding them to the list for note in notes: old_note = Note( user_id=note['id']), ...

How to update Cookiecutter Django config for existing project?

I have been building a project on top of cookiecutter django (https://github.com/pydanny/cookiecutter-django), with Celery turned off in the cookiecutter generation settings. However, now I'm...

How can I rearrange celery flower default columns?

I am using the following versions: flower==0.9.3 celery==4.3.0 Which gives me the following display for a tasks page with several columns: Each time I enter this page I end up rearranging the...

django send email with celery task

I want to send an email to a user with order.id and "manu_date" of a specific order automatically if the deadline of the order is today. I have already configured celery with rabbitmq and it's...

Apple M1: install psycopg2 package Symbol not found: _PQbackendPID

pg_config venv ❯ which pg_config /Applications/Postgres.app/Contents/Versions/latest/bin/pg_config pip3 install -r requirements.txt requirements.txt venv ❯ cat requirements.txt -i...

Unable to install pip azure-cognitiveservices-speech within a docker container

Original Issue I'm unable to install azure-cognitiveservices-speech within a docker container using pip. My container is running fine and is able to install all other packages (django,...

Airflow on_failure_callback

Hello hope you all are doing fine i would like to ask one question recently i have been trying airlfow and playing with it here is the situation everything works fine i have two...

How to get rid of cryptography build error?

I am trying to build a dockerfile but the problem is when it trying to build specifically cryptography is not building. MY Dockerfile FROM python:3.7-alpine ENV PYTHONUNBUFFERED 1 RUN apk update...

Prevent celery task from starting until a different task with similar arguments is done

Let's say I have a celery task which takes two arguments: X(a,b) I need to implement custom concurrency logic with the following two rules: Instances of X can run concurrently if they have...

How can I connect Flower to the broker RabbitMQ in docker-compose

How I can connect Flower to RabbitMQ. My celery running tasks do not appear on Flower. Although they are running fine. On the Flower UI, when I click on "broker", I get this error message on the...

Rabbitmq on docker: Application mnesia exited with reason: stopped

I'm trying to launch Rabbitmq with docker-compose alongside DRF and Celery. Here's my docker-compose file. Everything else works fine, except for rabbitmq: version: '3.7' services: drf: ...

NameError: name '_mysql' is not defined -- On airflow start in MacOSX

There are numbers of articles on the titled question but none of them worked for me. The detailed error is as follows: Traceback (most recent call last): File...

Gitlab CI/CD: New runner has not been connected yet

I am very new to Gitlab CI/CD and I have read its documentation very carefully about creating a new CI/CD process using .gitlab-ci.yml file. As I have found out in order to have Continuous...

Celery No hostname was supplied. Reverting to default 'localhost'

I have this in my /var/log/celery/w1.log I'm following the steps for Celery here. I have this in my celery.py from __future__ import absolute_import, unicode_literals import os from celery...

python celery AttributeError: 'int' object has no attribute 'upper' error

My tasks.py from celery import Celery app = Celery('tasks',broker='xxxxxxxxxxx') app.conf.timezone = 'UTC' app.conf.beat_schedule = { 'add-every-10-seconds': { 'task':...

Airflow CeleryExecutor - 'int' object has no attribute 'startswith' in Celery

Airflow 2.0 is queuing but not launching tasks in my dev environment. DAG and Pool settings are valid, but all tasks in each dag are queued when I trigger them, and are never running. When typing...

headless chrome on docker M1 error - unable to discover open window in chrome

I'm currently trying to run headless chrome with selenium on m1 mac host / amd64 ubuntu container. Because arm ubuntu does not support google-chrome-stable package, I decided to use amd64 ubuntu...