Celery hangs on postgres db population
We are using celery to manage a task queue within several docker containers. We are using Redis as both our broker and results backend. When we trigger the async task that populates our Postgres database, celery hangs, but only sometimes. We have not been able to observe what pattern this follows, but the task halts execution and hangs on the step in which we create several database tables in a Postgres database owned by the application.
We are using the following versions of the relevant softwares:
Postgres: 9.6.6
Psycopg: 2.7.5
Celery: 4.2.1
Ubuntu: 14.04.3
Redis: 3.2
When we cleared the hung active task, reserved tasks were not moved into active. We were unable to revoke these tasks as well and had to resort to restarting the docker container.
Here's what we tried first:
We attempted restarting the worker, but this did not move the tasks out of reserved.
We attempted to revoke the reserved tasks, and while celery confirmed that they were revoked, they still showed up in the list of reserved tasks for that worker.
When we tried to start a new task, this did not effect the status, either.
We found one relevant medium post.
In an attempt to follow their diagnostic steps, we were unable to use strace for our purpose, it threw the following error:
strace: attach: ptrace(PTRACE_SEIZE, 45): Operation not permitted
And have tried numerous methods to attempt diagnosis. We understand that there were database lock issues in a previous version of Postgres, but it appears we are using the version in which this was solved.
Any help or ideas would be appreciated. We're application engineers in over our heads with sparse dev-ops help available. Thanks!
postgresql docker ubuntu redis celery
asked Nov 16 '18 at 1:38
Eli ByersEli Byers
788
add a comment |
We are using celery to manage a task queue within several docker containers. We are using Redis as both our broker and results backend. When we trigger the async task that populates our Postgres database, celery hangs, but only sometimes. We have not been able to observe what pattern this follows, but the task halts execution and hangs on the step in which we create several database tables in a Postgres database owned by the application.
We are using the following versions of the relevant softwares:
Postgres: 9.6.6
Psycopg: 2.7.5
Celery: 4.2.1
Ubuntu: 14.04.3
Redis: 3.2
When we cleared the hung active task, reserved tasks were not moved into active. We were unable to revoke these tasks as well and had to resort to restarting the docker container.
Here's what we tried first:
We attempted restarting the worker, but this did not move the tasks out of reserved.
We attempted to revoke the reserved tasks, and while celery confirmed that they were revoked, they still showed up in the list of reserved tasks for that worker.
When we tried to start a new task, this did not effect the status, either.
We found one relevant medium post.
In an attempt to follow their diagnostic steps, we were unable to use strace for our purpose, it threw the following error:
strace: attach: ptrace(PTRACE_SEIZE, 45): Operation not permitted
And have tried numerous methods to attempt diagnosis. We understand that there were database lock issues in a previous version of Postgres, but it appears we are using the version in which this was solved.
Any help or ideas would be appreciated. We're application engineers in over our heads with sparse dev-ops help available. Thanks!
postgresql docker ubuntu redis celery
asked Nov 16 '18 at 1:38
Eli ByersEli Byers
788
Trystraceas root. Connect to the database and examinepg_locksandpg_stat_activity.
– Laurenz Albe
Nov 16 '18 at 4:47
add a comment |
We are using celery to manage a task queue within several docker containers. We are using Redis as both our broker and results backend. When we trigger the async task that populates our Postgres database, celery hangs, but only sometimes. We have not been able to observe what pattern this follows, but the task halts execution and hangs on the step in which we create several database tables in a Postgres database owned by the application.
We are using the following versions of the relevant softwares:
Postgres: 9.6.6
Psycopg: 2.7.5
Celery: 4.2.1
Ubuntu: 14.04.3
Redis: 3.2
When we cleared the hung active task, reserved tasks were not moved into active. We were unable to revoke these tasks as well and had to resort to restarting the docker container.
Here's what we tried first:
We attempted restarting the worker, but this did not move the tasks out of reserved.
We attempted to revoke the reserved tasks, and while celery confirmed that they were revoked, they still showed up in the list of reserved tasks for that worker.
When we tried to start a new task, this did not effect the status, either.
We found one relevant medium post.
In an attempt to follow their diagnostic steps, we were unable to use strace for our purpose, it threw the following error:
strace: attach: ptrace(PTRACE_SEIZE, 45): Operation not permitted
And have tried numerous methods to attempt diagnosis. We understand that there were database lock issues in a previous version of Postgres, but it appears we are using the version in which this was solved.
Any help or ideas would be appreciated. We're application engineers in over our heads with sparse dev-ops help available. Thanks!
postgresql docker ubuntu redis celery
asked Nov 16 '18 at 1:38
Eli ByersEli Byers
788
We are using celery to manage a task queue within several docker containers. We are using Redis as both our broker and results backend. When we trigger the async task that populates our Postgres database, celery hangs, but only sometimes. We have not been able to observe what pattern this follows, but the task halts execution and hangs on the step in which we create several database tables in a Postgres database owned by the application.
We are using the following versions of the relevant softwares:
Postgres: 9.6.6
Psycopg: 2.7.5
Celery: 4.2.1
Ubuntu: 14.04.3
Redis: 3.2
When we cleared the hung active task, reserved tasks were not moved into active. We were unable to revoke these tasks as well and had to resort to restarting the docker container.
Here's what we tried first:
We attempted restarting the worker, but this did not move the tasks out of reserved.
We attempted to revoke the reserved tasks, and while celery confirmed that they were revoked, they still showed up in the list of reserved tasks for that worker.
When we tried to start a new task, this did not effect the status, either.
We found one relevant medium post.
In an attempt to follow their diagnostic steps, we were unable to use strace for our purpose, it threw the following error:
strace: attach: ptrace(PTRACE_SEIZE, 45): Operation not permitted
And have tried numerous methods to attempt diagnosis. We understand that there were database lock issues in a previous version of Postgres, but it appears we are using the version in which this was solved.
Any help or ideas would be appreciated. We're application engineers in over our heads with sparse dev-ops help available. Thanks!
postgresql docker ubuntu redis celery
postgresql docker ubuntu redis celery
asked Nov 16 '18 at 1:38
Eli ByersEli Byers
788
asked Nov 16 '18 at 1:38
Eli ByersEli Byers
788
asked Nov 16 '18 at 1:38
Eli ByersEli Byers
788
asked Nov 16 '18 at 1:38
Eli ByersEli Byers
788
asked Nov 16 '18 at 1:38
Eli ByersEli Byers
788
788
Trystraceas root. Connect to the database and examinepg_locksandpg_stat_activity.
– Laurenz Albe
Nov 16 '18 at 4:47
add a comment |
Trystraceas root. Connect to the database and examinepg_locksandpg_stat_activity.
– Laurenz Albe
Nov 16 '18 at 4:47
Try
strace as root. Connect to the database and examine pg_locks and pg_stat_activity.– Laurenz Albe
Nov 16 '18 at 4:47
Try
strace as root. Connect to the database and examine pg_locks and pg_stat_activity.– Laurenz Albe
Nov 16 '18 at 4:47
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330254%2fcelery-hangs-on-postgres-db-population%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330254%2fcelery-hangs-on-postgres-db-population%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Try
straceas root. Connect to the database and examinepg_locksandpg_stat_activity.– Laurenz Albe
Nov 16 '18 at 4:47