英文:
Kolmogorov-Smirnov Test and Regression to derive the goodness of fit for negative binomial and Poisson distribution in R
问题
I have a vector dataset dt and only a few of 500 observations is given below.
[1] 0 4 8 13 15 1 0 5 4 6 0 0 0 1 2 3 0 0 0 0 0 3 0 10 15 0 23 0 6 0 10 2 15 0 3 0 2 0 [39] 3 0 42 4 6 12 3 2 0 0 0 2 3 9 6 0 3 2 1 6 0 0 0 0 0 1 0 0 2 0 0 0 0 7 0 0 0 1
I want to check if the distribution of data follows a negative binomial distribution or Poisson distribution better using the KS test and regression. There are only a few R packages available for the KS test of which not all give a good summary and some are no longer valid.
What I have tried?
Package iZID
install.packages("iZID")
library("iZID")
dis.kstest(dt, nsim = 100, bootstrap = TRUE, distri = 'nb')$pvalue
dis.kstest(dt, nsim = 100, bootstrap = TRUE, distri = 'Poisson')$pvalue
Returns
> dis.kstest(dt, nsim = 100, bootstrap = TRUE, distri = 'nb')$pvalue
[1] 0
> dis.kstest(dt, nsim = 100, bootstrap = TRUE, distri = 'Poisson')$pvalue
[1] 0
Question
First, With the above results it’s hard to recommend which distribution is a better fit. Second, I have also used ks.test() in dgof package but it needs initialization of lambda value for Poisson which is unknown and also Negative binomial should have a probability value initialized which is unknown. Third, I receive an error suggesting that goodfit() in vcd package is not available hence couldn't use this tool.
-
Can I use a ks-test package other than
dgof,iZID, andvcd? -
Can I use regression to compare the AIC of two distributions using
glmin R? In that case, how to build the model? What is X and Y?
Data in dput format
dt <- c(0L, 4L, 8L, 13L, 15L, 1L, 0L, 5L, 4L, 6L, 0L, 0L, 0L, 1L, 2L,
3L, 0L, 0L, 0L, 0L, 0L, 3L, 0L, 10L, 15L, 0L, 23L, 0L, 6L, 0L,
10L, 2L, 15L, 0L, 3L, 0L, 2L, 0L, 3L, 0L, 42L, 4L, 6L, 12L, 3L,
2L, 0L, 0L, 0L, 2L, 3L, 9L, 6L, 0L, 3L, 2L, 1L, 6L, 0L, 0L, 0L,
0L, 0L, 1L, 0L, 0L, 2L, 0L, 0L, 0L, 0L, 7L, 0L, 0L, 0L, 1L)
英文:
I have a vector dataset dt and only a few of 500 observations is given below.
[1] 0 4 8 13 15 1 0 5 4 6 0 0 0 1 2 3 0 0 0 0 0 3 0 10 15 0 23 0 6 0 10 2 15 0 3 0 2 0
[39] 3 0 42 4 6 12 3 2 0 0 0 2 3 9 6 0 3 2 1 6 0 0 0 0 0 1 0 0 2 0 0 0 0 7 0 0 0 1
I want to check if the distribution of data follows a negative binomial distribution or Poisson distribution better using the KS test and regression. There are only a few R packages available for the KS test of which not all give a good summary and some are no longer valid.
What I have tried?
Package iZID
install.packages("iZID")
library("iZID")
dis.kstest(dt,nsim=100,bootstrap=TRUE,distri='nb')$pvalue
dis.kstest(dt,nsim=100,bootstrap=TRUE,distri='Poisson')$pvalue
Returns
> dis.kstest(dt,nsim=100,bootstrap=TRUE,distri='nb')$pvalue
[1] 0
> dis.kstest(dt,nsim=100,bootstrap=TRUE,distri='Poisson')$pvalue
[1] 0
Question
First, With the above results it’s hard to recommend which distribution is a better fit. Second, I have also used ks.test() in dgof package but it needs initialization of lambda value for Poisson which is unknown and also Negative binomial should have a probability value initialized which is unknown. Third, I receive an error suggesting that goodfit() in vcd package is not available hence couldn't use this tool.
-
Can I use a ks-test package other than
dgof,iZID, andvcd? -
Can I use regression to compare the AIC of two distributions using
glmin R? In that case, how to build the model? What is X and Y?
Data in dput format
dt <-
c(0L, 4L, 8L, 13L, 15L, 1L, 0L, 5L, 4L, 6L, 0L, 0L, 0L, 1L, 2L,
3L, 0L, 0L, 0L, 0L, 0L, 3L, 0L, 10L, 15L, 0L, 23L, 0L, 6L, 0L,
10L, 2L, 15L, 0L, 3L, 0L, 2L, 0L, 3L, 0L, 42L, 4L, 6L, 12L, 3L,
2L, 0L, 0L, 0L, 2L, 3L, 9L, 6L, 0L, 3L, 2L, 1L, 6L, 0L, 0L, 0L,
0L, 0L, 1L, 0L, 0L, 2L, 0L, 0L, 0L, 0L, 7L, 0L, 0L, 0L, 1L)
答案1
得分: 1
First, to answer your question about KS test packages, yes, you can try the ks package in R.
It provides the ks.test() function that you could use to perform a KS test on your dataset.
For example, to perform a KS test on your dataset using a Poisson distribution, you can use the following code (of course I am assuming that you already have installed the package, before running the below code):
install.packages("ks", dependencies = T)
library(ks)
ks.test(dt, "ppois", lambda=mean(dt))
Here,
"ppois" is the Poisson distribution,
"lambda" is an estimate for the rate parameter of the Poisson distribution and we have set it to the mean of your data
The above code returns:
Asymptotic one-sample Kolmogorov-Smirnov test
data: dt
D = 0.44349, p-value = 2.078e-13
alternative hypothesis: two-sided
Which means:
The p-value is very small (2.078e-13), thus, suggesting a strong evidence
against the null hypothesis.
Therefore, you can reject the null hypothesis that the data follow the specified theoretical distribution at the significance level of 0.05, and hence, conclude that the data do not follow the theoretical distribution.
Similarly, you could use the qnbinom distribution for the negative binomial distribution.
Now, to answer your second question about comparing the AIC of two distributions using regression, what you could use is glm() native function in R.
All you need to do is, first, create an output variable Y that represents the count data, and then an explanatory (input) variable X that refers to the index of each observation in your dataset.
Now you can fit two models; one, with the Poisson Distributions and, second, with the Negative Binomial Distributions, respectively, and compare their AIC values to determine which distribution provides a better fit.
library(ks)
library(MASS)
Y <- dt # create response var
X <- 1:length(Y) # create independent (input) var
modelPD <- glm(Y ~ X, family = "poisson") # fit the Poisson Distribution
modelNBD <- glm.nb(Y ~ X) # fit the Negative Binomial Distribution from MASS packages
# compare two AICs
AIC(modelPD)
AIC(modelNBD)
Which returns:
688.5675 and 334.3564
And as we know, lower the AIC value, the better the model fits the data.
Therefore, we can conclude that the Negative Binomial Distribution provides a better fit to the data compared to the Poisson Distribution.
Let me know if this helps...
英文:
First, to answer your question about KS test packages, yes, you can try the ks package in R.
It provides the ks.test() function that you could use to perform a KS test on your dataset.
For example, to perform a KS test on your dataset using a Poisson distribution, you can use the following code (of course I am assuming that you already have installed the package, before running the below code):
install.packages("ks", dependencies = T)
library(ks)
ks.test(dt, "ppois", lambda=mean(dt))
Here,
"ppois" is the Poisson distribution,
"lambda" is an estimate for the rate parameter of the Poisson distribution and we have set it to the mean of your data
The above code returns:
Asymptotic one-sample Kolmogorov-Smirnov test
data: dt
D = 0.44349, p-value = 2.078e-13
alternative hypothesis: two-sided
Which means:
> The p-value is very small (2.078e-13), thus, suggesting a strong evidence
> against the null hypothesis.
> Therefore, you can reject the null hypothesis that the data follow the specified theoretical distribution at the significance level of 0.05, and hence, conclude that the data do not follow the theoretical distribution.
Similarly, you could use the qnbinom distribution for the negative binomial distribution.
Now, to answer your second question about comparing the AIC of two distributions using regression, what you could use is glm() native function in R.
All you need to do is, first, create an output variable Y that represents the count data, and then an explanatory (input) variable X that refers to the index of each observation in your dataset.
Now you can fit two models; one, with the Poisson Distributions and, second, with the Negative Binomial Distributions, respectively, and compare their AIC values to determine which distribution provides a better fit.
library(ks)
library(MASS)
Y <- dt # create response var
X <- 1:length(Y) # create independent (input) var
modelPD <- glm(Y ~ X, family = "poisson") # fit the Poisson Distribution
modelNBD <- glm.nb(Y ~ X) # fit the Negative Binomial Distribution from MASS packages
# compare two AICs
AIC(modelPD)
AIC(modelNBD)
Which returns:
688.5675 and 334.3564
And as we know, lower the AIC value, the better the model fits the data.
Therefore, we can conclude that the Negative Binomial Distribution provides a better fit to the data compared to the Poisson Distribution.
Let me know if this helps...
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。


评论