How to implement PReLU activation in Tensorflow?

I think it's much easier to implement with tf.maximum

My implementation is as following

import tensorflow as tf

def PReLU(_x, name=None):
  if name is None:
    name = "alpha"
  _alpha = tf.get_variable(name,
                           shape=_x.get_shape(),
                           initializer=tf.constant_initializer(0.0),
                           dtype=_x.dtype)

  return tf.maximum(_alpha*_x, _x)

Although the solution with tf.maximum is very efficient, it can't represent a concave function. Here's a solution that can:

def prelu(_x, scope=None):
    """parametric ReLU activation"""
    with tf.variable_scope(name_or_scope=scope, default_name="prelu"):
        _alpha = tf.get_variable("prelu", shape=_x.get_shape()[-1],
                                 dtype=_x.dtype, initializer=tf.constant_initializer(0.1))
        return tf.maximum(0.0, _x) + _alpha * tf.minimum(0.0, _x)

The implementation of PReLU seems straight-forward based on the PreLU implementations (see: Keras, TFLearn and TensorLayer) of the higher level libraries. My code is as follows:

def parametric_relu(_x):
  alphas = tf.get_variable('alpha', _x.get_shape()[-1],
                       initializer=tf.constant_initializer(0.0),
                        dtype=tf.float32)
  pos = tf.nn.relu(_x)
  neg = alphas * (_x - abs(_x)) * 0.5

  return pos + neg

Tags:

Tensorflow