Ndtyjky

Suppose I have a list of objects and for each of these objects I want to create a dictionary and put all the dictionaries I generated in a list.

What I am doing is the following:

def f(x):

    some function f that returns a dictionary given x



list_of_dict = 



xlist = [x1, x2, ..., xN]



for x in xlist:

    list_of_dict.append(f(x))

I am wondering whether there is a more efficient (faster) way to create a list of dictionaries than the one I am proposing.

Thank you.

Since you are dealing with HTTP requests (which is not obvious from your question), the rest of the answer is irrelevant: communications will dominate the computations by a sheer margin. I'll leave the answer here, anyway.

The original approach seems to be the slowest:

In [20]: %%timeit 

    ...: list_of_dict =  

    ...: for x in xlist: 

    ...:   list_of_dict.append(f(x)) 

    ...:                                                                        

13.5 µs ± 39.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Mapping is the best way to go:

In [21]: %timeit list(map(f,xlist))                                             

8.45 µs ± 17 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

List comprehension is somewhere in the middle:

In [22]: %timeit [f(x) for x in xlist]                                          

10.2 µs ± 22.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Since you mentioned in the comments that you want faster execution overall, maybe async the requests?

Check out this example adapted from this blog post: http://skipperkongen.dk/2016/09/09/easy-parallel-http-requests-with-python-and-asyncio/

import asyncio

import requests



async def main():

    xlist = [...]

    list_of_dict = 

    loop = asyncio.get_event_loop()

    futures = [

        loop.run_in_executor(

            None, 

            requests.get, 

            i

        )

        for i in xlist

    ]

    for response in await asyncio.gather(*futures):

        respons_dict = your_parser(response) # Your parsing to dict from before

        list_of_dict.append(response_dict)

    return list_of_dict



loop = asyncio.get_event_loop()

loop.run_until_complete(main())

I think this can help you with implement parallelism. Just note that you should be sure you aren't flooding somebody with requests that they wouldn't appreciate.