Ndtyjky

Sorry if this has been answered before, but I'm having trouble with the solution.

I have a 2D DataFrame with column names, where the elements contain both non-null and null values. I'd like to 'flatten' the 2D DataFrame to a 1D Series, where I preserve only the non-null data as a list in the series cell with the corresponding header.

ie: the following:

Going from (type pandas.Dataframe):

| asset | name | id |

---------------------

| a     | john | 001|

| a     | NaN  | 002|

| NaN   | dave | 003|

To (type pandas.Series):

| asset | name         | id              |

------------------------------------------

| [a]   | [john, dave] | [001, 002, 003] |

Thank you!

EDIT: Why I would need this:

I am starting with a large DataFrame that has multiple duplicated attributes with timestamped 'rows'. At any given timestamp, the information in the rows could be added to, to deleted. I have used df.where() to return a dataframe of the unique values, and am attempting to flatten it down to one attribute collection of 'ids' per row.

In practice, the example table is from a a single GroupBy object.

Instantiate a new series using a dict comprehension (this should be faster than an apply based solution).

pd.Series({c : df[c].dropna().unique().tolist() for c in df.columns})



asset             [a]

name     [john, dave]

id          [1, 2, 3]

dtype: object

If you want a single rowed DataFrame instead, use

pd.Series(

    {c : df[c].dropna().unique().tolist() for c in df.columns}

).to_frame().T



  asset          name         id

0   [a]  [john, dave]  [1, 2, 3]