Deep Learning Recipe

Posted on Views: Disqus:

High bias and Variance

High Bias

  • Underfitting the training set
  • Training set performance similar to development set performance

High variance

  • Overfitting the training set
  • Training set performance superior than development set performance

Examples

Assume that the Bayes error is nearly 0%

  1. Training set error 1%; development set error 11%
    • High variance
  2. Training set error 15%; development set error 16%
    • High bias
  3. Training set error 0.5%; development set error 0.5%
    • Low variance & Low bias
  4. Training set error 15%; development set error 30%
    • High variance & High bias
    • e.g. fit 的这条线 一部分是linear 的 一部分过分curvy

General process

graph LR
A[High Bias?] -->|Y| B(Bigger network)
A --> |Y| C(Train longer)
A --> |Y| D(Use more advanced optimzation algoorithm)
E[High Variance?] -->|Y| F(Regularlization)
E -->|Y| G(Use more data)

In conclusion

以前其实是有那么个东西 叫做 bias-variance trade-off 的指的是variance 和 bias 之间的 inverse relation。 Low variance 和 low bias 很难共存。但随着大数据时代的来袭，我们有更多的渠道去收集更多的数据(比如爬虫) 导致修复其中一个问题并不会导致另外一个问题变得更糟糕。