How to convert multiple columns to single column?

If there is always only one non missing value per rows use forward filling missing values (like DataFrame.fillna with method='ffill') and then select last column by position with DataFrame.iloc, also for one column DataFrame add Series.to_frame:

df = df.ffill(axis=1).iloc[:, -1].to_frame('new')
print (df)
      new
0     cat
1     dog
2   horse
3  donkey
4     pig

If possible more non missing values per rows use DataFrame.stack with join per first level:

print (df)
    p1   p2     p3      p4    p5
0  cat  NaN    NaN     NaN  lion
1  NaN  dog    NaN     NaN   NaN
2  NaN  NaN  horse     NaN   NaN
3  NaN  NaN    NaN  donkey   NaN
4  NaN  NaN    NaN     NaN   pig

df2 = df.stack().groupby(level=0).apply(', '.join).to_frame('new')
print (df2)
         new
0  cat, lion
1        dog
2      horse
3     donkey
4        pig

Or lambda function:

df2 = df.apply(lambda x: x.dropna().str.cat(sep=', '), axis=1).to_frame('new')
print (df2)
         new
0  cat, lion
1        dog
2      horse
3     donkey
4        pig