使用PyTorch计算用于分类和回归的95%可信区间的正确方法是什么？ [英] What is the proper way to compute 95% confidence intervals with PyTorch for classification and regression?
本文介绍了使用PyTorch计算用于分类和回归的95%可信区间的正确方法是什么？的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我想使用PyTorch报告我的数据的90、95、99等可信区间。但置信度间隔似乎太重要了，不能让我的实现未经测试或受到批评，所以我希望得到反馈-至少应该由一些专家进行检查。此外，我已经注意到，当我的值为负值时，我得到了NaN值，这让我认为我的代码只适用于分类(至少)，但我也会进行回归。我还感到惊讶的是，直接使用NumPy代码实际上给了我可微的张量……这是我意想不到的。
那么这是正确的吗？：
import numpy as np
import scipy
import torch
from torch import Tensor
P_CI = {0.90: 1.64,
0.95: 1.96,
0.98: 2.33,
0.99: 2.58,
}
def mean_confidence_interval_rfs(data, confidence=0.95):
"""
https://stackoverflow.com/a/15034143/1601580
"""
a = 1.0 * np.array(data)
n = len(a)
m, se = np.mean(a), scipy.stats.sem(a)
h = se * scipy.stats.t.ppf((1 + confidence) / 2., n - 1)
return m, h
def mean_confidence_interval(data, confidence=0.95):
a = 1.0 * np.array(data)
n = len(a)
m, se = np.mean(a), scipy.stats.sem(a)
h = se * scipy.stats.t.ppf((1 + confidence) / 2., n - 1)
return m, m - h, m + h
def ci(a, p=0.95):
import numpy as np, scipy.stats as st
st.t.interval(p, len(a) - 1, loc=np.mean(a), scale=st.sem(a))
# def ci(a, p=0.95):
# import statsmodels.stats.api as sms
#
# sms.DescrStatsW(a).tconfint_mean()
def compute_confidence_interval_classification(data: Tensor,
by_pass_30_data_points: bool = False,
p_confidence: float = 0.95
) -> Tensor:
"""
Computes CI interval
[B] -> [1]
According to [1] CI the confidence interval for classification error can be calculated as follows:
error +/- const * sqrt( (error * (1 - error)) / n)
The values for const are provided from statistics, and common values used are:
1.64 (90%)
1.96 (95%)
2.33 (98%)
2.58 (99%)
Assumptions:
Use of these confidence intervals makes some assumptions that you need to ensure you can meet. They are:
Observations in the validation data set were drawn from the domain independently (e.g. they are independent and
identically distributed).
At least 30 observations were used to evaluate the model.
This is based on some statistics of sampling theory that takes calculating the error of a classifier as a binomial
distribution, that we have sufficient observations to approximate a normal distribution for the binomial
distribution, and that via the central limit theorem that the more observations we classify, the closer we will get
to the true, but unknown, model skill.
Ref:
- computed according to: https://machinelearningmastery.com/report-classifier-performance-confidence-intervals/
todo:
- how does it change for other types of losses
"""
B: int = data.size(0)
# assert data >= 0
assert B >= 30 and (not by_pass_30_data_points), f' Not enough data for CI calc to be valid and approximate a'
f'normal, you have: {B=} but needed 30.'
const: float = P_CI[p_confidence]
error: Tensor = data.mean()
val = torch.sqrt((error * (1 - error)) / B)
print(val)
ci_interval: float = const * val
return ci_interval
def compute_confidence_interval_regression():
"""
todo
:return:
"""
raise NotImplementedError
# - tests
def ci_test():
x: Tensor = abs(torch.randn(35))
ci_pytorch = compute_confidence_interval_classification(x)
ci_rfs = mean_confidence_interval(x)
print(f'{x.var()=}')
print(f'{ci_pytorch=}')
print(f'{ci_rfs=}')
x: Tensor = abs(torch.randn(35, requires_grad=True))
ci_pytorch = compute_confidence_interval_classification(x)
ci_rfs = mean_confidence_interval(x)
print(f'{x.var()=}')
print(f'{ci_pytorch=}')
print(f'{ci_rfs=}')
x: Tensor = torch.randn(35) - 10
ci_pytorch = compute_confidence_interval_classification(x)
ci_rfs = mean_confidence_interval(x)
print(f'{x.var()=}')
print(f'{ci_pytorch=}')
print(f'{ci_rfs=}')
if __name__ == '__main__':
ci_test()
print('Done, success! a')
输出：
tensor(0.0758)
x.var()=tensor(0.3983)
ci_pytorch=tensor(0.1486)
ci_rfs=(tensor(0.8259), tensor(0.5654), tensor(1.0864))
tensor(0.0796, grad_fn=<SqrtBackward>)
x.var()=tensor(0.4391, grad_fn=<VarBackward>)
ci_pytorch=tensor(0.1559, grad_fn=<MulBackward0>)
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1483, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"
", file, 'exec'), glob, loc)
File "/Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/metrics/metrics.py", line 154, in <module>
ci_test()
File "/Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/metrics/metrics.py", line 144, in ci_test
ci_pytorch = compute_confidence_interval_classification(x, by_pass_30_data_points)
如何修复上面的回归代码，例如任意大小的负值？
令人有点惊讶的是，现在还没有一个实现，特别是没有正式的PyTorch实现，因为CI应该是多么重要……也许是一种深度学习的坏习惯？不幸的是，很少在报纸上看到它。引用：
- 交叉发布：https://discuss.pytorch.org/t/what-is-the-proper-way-to-compute-95-confidence-intervals-with-pytorch-for-classification-and-regression/139398
- Compute a confidence interval from sample data
- https://machinelearningmastery.com/report-classifier-performance-confidence-intervals/
- https://medium.com/hal24k-techblog/how-to-generate-neural-network-confidence-intervals-with-keras-e4c0b78ebbdf
- https://discuss.pytorch.org/t/variance-or-confidence-interval-for-outputs/91043
- https://github.com/WangYueFt/rfs/blob/master/eval/meta_eval.py
- Calculate the accuracy every epoch in PyTorch
- Construct 95% confidence interval for regression model
- https://www.quora.com/unanswered/How-do-you-calculate-a-95-confidence-interval-for-a-simple-regression
- https://www.quora.com/unanswered/What-is-the-95-confidence-interval-in-regression
- https://www.quora.com/unanswered/What-is-the-proper-way-to-compute-95-confidence-intervals-with-PyTorch-for-classification-and-regression
- https://stats.stackexchange.com/questions/556265/what-is-the-proper-way-to-report-confidence-intervals-or-std-variance-for-mac
推荐答案
tldr；
可信区间(Ci)计算：
- 真实平均值在给定区间内的概率(通常写为
mu_n +- ci
假设：
- 传统的可信区间语句仅适用于关于我们要估计的值(参数、随机数量等)的语句平均值
- 您有足够的样本使分析成立(例如，平均值$MU_n=1/n sum_i x_i$，其中推荐
n>=30)
如果这些假设成立(**即您通过样本平均值与+值匹配真实平均值**)，则使用下面我提供的名为torch_compute_confidence_interval的代码进行回归、分类，任何您想要的。
首先，asfaik可信区间(Ci)是深度学习(DL)中的一个开放研究问题，因此可能存在更复杂的答案。但我将提供一个我计划使用的实用答案(并看到其他人在以数字语言报告结果时使用)。
要计算可信区间，我们必须先了解一点Ci。它们是关于随机调查/数据集样本的概率陈述，即您试图报告的平均值在报告的间隔内。所以当人们说：
mean_error +- CI for p=95%
这意味着如果您采样了95个数据集，您预计真正的平均值会在95个时间间隔内(但您不知道是哪一个，所以您不能说对于您计算的任何特定间隔，平均值都会在那里)。
这意味着您只能将其用于报告意味着。这是因为它背后的数学(这并不是很难)通过利用我们可以解析地计算样本平均值的概率来近似计算界限(或可信区间)的概率，因为根据中心极限定理CLT，近似是正态的。因此，计算的特定CI假设您要计算的量是样本平均值，并使用此正态近似计算+数。因此，通常建议您正在使用的特定数据集有n>=30个数据点，但事情仍然可以很好地解决，因为可以用t分布而不是正态分布(在STATS软件中表示为z)来计算ci。
根据这些假设，您只需执行以下操作：
def torch_compute_confidence_interval(data: Tensor,
confidence: float = 0.95
) -> Tensor:
"""
Computes the confidence interval for a given survey of a data set.
"""
n = len(data)
mean: Tensor = data.mean()
# se: Tensor = scipy.stats.sem(data) # compute standard error
# se, mean: Tensor = torch.std_mean(data, unbiased=True) # compute standard error
se: Tensor = data.std(unbiased=True) / (n**0.5)
t_p: float = float(scipy.stats.t.ppf((1 + confidence) / 2., n - 1))
ci = t_p * se
return mean, ci
我测试了它，并将其与专门用于分类的东西进行了比较，它们的值一致，最高可达1e-2，因此代码可以工作。输出：
Connected to pydev debugger (build 213.5744.248)
x_bernoulli.std()=tensor(0.5040)
ci_95=0.1881992999915952
ci_95_cls=tensor(0.1850)
ci_95_anything=tensor(0.1882)
x_bernoulli.std()=tensor(0.5085, grad_fn=<StdBackward>)
ci_95_torch=tensor(0.1867, grad_fn=<MulBackward0>)
x.std()=tensor(0.9263)
ci_95=0.3458867459004733
ci_95_torch=tensor(0.3459)
x.std()=tensor(1.0181, grad_fn=<StdBackward>)
ci_95_torch=tensor(0.3802, grad_fn=<MulBackward0>)
有关更多详细信息，请参阅我的UltraUtils库，其中我对文档中的数学内容进行了评论：https://github.com/brando90/ultimate-utils/blob/e81a8c3c4425b33e00b3ade172705f20b626b2b1/ultimate-utils-proj-src/uutils/torch_uu/metrics/confidence_intervals.py#L1
关于数字图书馆的评论
如果您报告的是特定模型的错误，例如神经网络，就像这样，您或多或少会报告该特定神经网络和权重的真实平均误差在这些范围内。但正如我所说，这是一个开放的研究领域，因此必须有更花哨的东西可用，例如，假设某些层实际上是随机的，等等。这篇关于使用PyTorch计算用于分类和回归的95%可信区间的正确方法是什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
