英文:
Activity regularizer in Keras: before or after activation?
问题
Keras文档(https://keras.io/api/layers/regularizers/)中说，activity regularizer "对该层的输出应用惩罚"，但它没有明确指定"输出"是指仅仅稠密操作的输出，还是整个层，包括激活函数。
对于我的问题，我需要在激活函数之后应用activity regularizer。如果Keras以另一种方式实现了它，我该如何修复它？
英文:
Suppose I have:
output = Dense(units=12, activation='sigmoid', activity_regularizer=L1(1e-2))(input)
Keras documentation says activity regularizer "apply a penalty on the layer's output", but it does not specify whether "output" means the output of the dense operation only, or that of the entire layer including activation.
For my problem I need the activity regularizer to apply after activation. In case Keras implements it the other way around, how can I fix it?
答案1
得分: 2
Keras在整个层（包括激活函数）之后应用活动正则化。
如果您滚动到Dense层的call方法的末尾，您将看到如果已定义，激活函数在返回输出之前被应用。
活动正则化是在Layer基类的call函数之后应用的。请参见此处
英文:
Keras applies the activity regularization after the entire layer including activation.
If you scroll to the end of the Dense layer call method you will see that, if defined, the activation is applied on the output before returning it.
The activity regularization is applied after this call function in the Layer base class. See here
答案2
得分: 0
正如thmslmr已经提到的那样：活动正则化是应用在该层的输出上，因此是在激活函数之后应用的。
如果您想在应用softmax激活之前应用正则化，您可以将激活函数移到一个单独的层中：
model.add(layers.Dense(units=12, activity_regularizer=L1(1e-2))
model.add(layers.Activation('softmax'))
英文:
As already mentioned by thmslmr: the activity regularizer is applied on the output of the layer, therefore after the activation function is applied.
If you want to apply regularization before the softmax activation, you can move the activation function into a separate layer :
model.add(layers.Dense(units=12, activity_regularizer=L1(1e-2) )
model.add(layers.Activation('softmax'))
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论