python multiprocessing pickle attribute look up
0
i am trying to do multiprocessing as shown:
from multiprocessing import Pool
def main():
df1 = function1()
df2 = function2(df1)
df3 = list(df2.groupby('key')
p = Pool(2)
p.map(model, df3)
The above works perfectly. However, when I add in this:
from multiprocessing import Pool
def main():
df1 = function1()
df2 = function2(df1)
df3 = list(df2.groupby('key')
df4 = [x+ (df1 [x[0]][0],)+ (df1 [company[0]][1],) for x in df3 ]
p = Pool(2)
p.map(model, df4)
Now i get this error:
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
I dont understand how that extra line makes it fail? There is no external function call.
python multiprocessing pickle
edited Nov 15 '18 at 7:10
georgexsh
10.8k11338
asked Nov 15 '18 at 3:30
skim8201skim8201
373
add a comment |
0
i am trying to do multiprocessing as shown:
from multiprocessing import Pool
def main():
df1 = function1()
df2 = function2(df1)
df3 = list(df2.groupby('key')
p = Pool(2)
p.map(model, df3)
The above works perfectly. However, when I add in this:
from multiprocessing import Pool
def main():
df1 = function1()
df2 = function2(df1)
df3 = list(df2.groupby('key')
df4 = [x+ (df1 [x[0]][0],)+ (df1 [company[0]][1],) for x in df3 ]
p = Pool(2)
p.map(model, df4)
Now i get this error:
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
I dont understand how that extra line makes it fail? There is no external function call.
python multiprocessing pickle
edited Nov 15 '18 at 7:10
georgexsh
10.8k11338
asked Nov 15 '18 at 3:30
skim8201skim8201
373
you need to provide a Minimal, Complete, and Verifiable example.
– georgexsh
Nov 15 '18 at 7:10
add a comment |
0
0
0
i am trying to do multiprocessing as shown:
from multiprocessing import Pool
def main():
df1 = function1()
df2 = function2(df1)
df3 = list(df2.groupby('key')
p = Pool(2)
p.map(model, df3)
The above works perfectly. However, when I add in this:
from multiprocessing import Pool
def main():
df1 = function1()
df2 = function2(df1)
df3 = list(df2.groupby('key')
df4 = [x+ (df1 [x[0]][0],)+ (df1 [company[0]][1],) for x in df3 ]
p = Pool(2)
p.map(model, df4)
Now i get this error:
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
I dont understand how that extra line makes it fail? There is no external function call.
python multiprocessing pickle
edited Nov 15 '18 at 7:10
georgexsh
10.8k11338
asked Nov 15 '18 at 3:30
skim8201skim8201
373
i am trying to do multiprocessing as shown:
from multiprocessing import Pool
def main():
df1 = function1()
df2 = function2(df1)
df3 = list(df2.groupby('key')
p = Pool(2)
p.map(model, df3)
The above works perfectly. However, when I add in this:
from multiprocessing import Pool
def main():
df1 = function1()
df2 = function2(df1)
df3 = list(df2.groupby('key')
df4 = [x+ (df1 [x[0]][0],)+ (df1 [company[0]][1],) for x in df3 ]
p = Pool(2)
p.map(model, df4)
Now i get this error:
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
I dont understand how that extra line makes it fail? There is no external function call.
python multiprocessing pickle
python multiprocessing pickle
edited Nov 15 '18 at 7:10
georgexsh
10.8k11338
asked Nov 15 '18 at 3:30
skim8201skim8201
373
edited Nov 15 '18 at 7:10
georgexsh
10.8k11338
asked Nov 15 '18 at 3:30
skim8201skim8201
373
edited Nov 15 '18 at 7:10
georgexsh
10.8k11338
edited Nov 15 '18 at 7:10
georgexsh
10.8k11338
edited Nov 15 '18 at 7:10
georgexsh
10.8k11338
10.8k11338
asked Nov 15 '18 at 3:30
skim8201skim8201
373
asked Nov 15 '18 at 3:30
skim8201skim8201
373
asked Nov 15 '18 at 3:30
skim8201skim8201
373
373
you need to provide a Minimal, Complete, and Verifiable example.
– georgexsh
Nov 15 '18 at 7:10
add a comment |
you need to provide a Minimal, Complete, and Verifiable example.
– georgexsh
Nov 15 '18 at 7:10
you need to provide a Minimal, Complete, and Verifiable example.
– georgexsh
Nov 15 '18 at 7:10
you need to provide a Minimal, Complete, and Verifiable example.
– georgexsh
Nov 15 '18 at 7:10
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53312018%2fpython-multiprocessing-pickle-attribute-look-up%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53312018%2fpython-multiprocessing-pickle-attribute-look-up%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
you need to provide a Minimal, Complete, and Verifiable example.
– georgexsh
Nov 15 '18 at 7:10