在Keras模型中保存元数据/信息。

huangapple go评论112阅读模式
英文:

Saving meta data/information in Keras model

问题

在Keras模型中保存元数据/元信息是否可能?我的目标是保存输入预处理参数、训练/测试集使用情况、类别标签映射等等,以便在再次加载模型时使用。
我查阅了Keras文档但没有找到相关信息。我在GitHub上找到了类似的问题,但两年前已关闭,没有解决方案。
目前,我将所有这些信息保存在单独的文件中,并在加载模型时使用该文件。
尽管可能与问题不相关,但我正在使用tf.keras的功能型模型,并将我的模型保存为h5文件使用model.save()

英文:

Is it possible to save meta data/meta information in Keras model? My goal is to save input pre-processing parameters, train/test set used, class label maps etc. which I can use while loading model again.<br> I went through Keras documentation and did not find anything. I found similar issue on GitHub but it was closed two years back without any resolution.<br>
Currently I am saving all these information in separate file, and using this file while loading the model.
<br> Although probably not relevant but I am using tf.keras functional model and saving my model as h5 file using model.save().

答案1

得分: 8

以下是已翻译的代码部分:

from tensorflow.python.keras.saving import hdf5_format
import h5py


# 保存模型
with h5py.File(model_path, mode='w') as f:
    hdf5_format.save_model_to_hdf5(my_keras_model, f)
    f.attrs['param1'] = param1
    f.attrs['param2'] = param2

# 加载模型
with h5py.File(model_path, mode='r') as f:
    param1 = f.attrs['param1']
    param2 = f.attrs['param2']
    my_keras_model = hdf5_format.load_model_from_hdf5(f)
英文:

This is working for me:

from tensorflow.python.keras.saving import hdf5_format
import h5py


# Save model
with h5py.File(model_path, mode=&#39;w&#39;) as f:
    hdf5_format.save_model_to_hdf5(my_keras_model, f)
    f.attrs[&#39;param1&#39;] = param1
    f.attrs[&#39;param2&#39;] = param2

# Load model
with h5py.File(model_path, mode=&#39;r&#39;) as f:
    param1 = f.attrs[&#39;param1&#39;]
    param2 = f.attrs[&#39;param2&#39;]
    my_keras_model = hdf5_format.load_model_from_hdf5(f)

答案2

得分: 1

我认为最接近满足您需求的方法(至少部分满足)是保存一个MetaGraph

您可以使用tf.saved_model方法来实现这一点(至少在TensorFlow 2.0中可以)。

您的原始模型也可以在Keras中进行训练,不一定要使用纯TensorFlow来使用tf.saved_model

您可以在这里了解更多关于tf.saved_model的信息:https://www.tensorflow.org/guide/saved_model

英文:

I think the closest think you could implement in order to satisfy your needs(at least part of them) is to save a MetaGraph.

You can achieve that by using tf.saved_model method (at least in TensorFlow 2.0).

Your original model can also be trained in Keras, not necessarily in pure tensorflow in order to use tf.saved_model.

You can read more about tf.saved_model here: https://www.tensorflow.org/guide/saved_model

答案3

得分: 1

这很恶劣,但对我有效。

我有一个名为'thr'的阈值参数,用于预处理。我将它构建到输入名称中...

input_img = layers.Input((None, None, 1), name="input-thr{0:f}".format(thr))

在另一个程序中,当我读取模型并使用它时,我会扫描输入名称以获取该值...

try:
    thr = float(re.match('input-thr(.*)', model.layers[0].name).group(1))
except:
    thr = args.thr # 默认值   

也许这并不像看起来那么讨厌,因为输入名称描述了模型对该输入所期望的预处理过程。

如果Keras模型有一个公共的元数据字典,我们可以将这样的东西存放在其中会更好。

附言:我已从我的代码中删除了这部分。

只需几行代码就可以将所有训练参数保存到一个单独的文件中。一旦设置好这个功能,就可以轻松保存所有参数,而不仅仅是你当前需要的参数。如果你真的很担心将这些数据同步到训练好的模型中,还可以保存模型的名称和创建时间。

英文:

This is vile, but it works for me.

I have a threshold parameter 'thr' that is used in the preprocessing. I build it into the input name...

input_img = layers.Input((None, None, 1), name=&quot;input-thr{0:f}&quot;.format(thr))

In another program, when I read the model and use it, I scan the input name for the value...

try:
    thr = float(re.match(&#39;input-thr(.*)&#39;, model.layers[0].name).group(1))
except:
    thr = args.thr # default value   

Perhaps this is not as nasty as it seems, for the input name describes the pre-processing the model expects for that input.

It would be nicer if the Keras model had a public metadata dictionary where we could stash stuff like this.

Postscript: I have removed this from my code.

It is only a few lines to save all the training parameters to a separate file. Once you set this up, it is easy to saved all the arguments and not the ones for your immediate needs. If you are really paranoid about syncing this data to the trained model, save the model name and creation time too.

huangapple
  • 本文由 发表于 2020年1月6日 19:07:54
  • 转载请务必保留本文链接:https://go.coder-hub.com/59611000.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定