Learning objectives

In this lesson you will

  1. Understand the applied context in which eventglm is used
  2. Fit some basic models, interpreting the resulting estimates
  3. Compare the eventglm models and estimates to what you get from the survival package

Setup

Library the survival package and then the eventglm package. We will use some commands from survival to make sense of what is going on in the regression models.

library(survival)
library(eventglm)
## 
## Attaching package: 'eventglm'
## The following objects are masked from 'package:survival':
## 
##     colon, mgus2

What is the outcome?

Colon cancer treatment and time to death (standard survival)

Our first example concerns the colon dataset, which is included in the package:

?eventglm::colon

This is a randomized trial, so the main interest is in comparing the distributions of time to death between the three treatment arms. Let’s start with a survival curve.

sfit <- survfit(Surv(time, status) ~ rx, data = colon)
plot(sfit, col = c("black", "slateblue", "salmon"), 
     xlab = "days since registration", ylab = "survival")
legend("bottomleft", fill = c("black", "slateblue", "salmon"), 
       legend = names(sfit$strata))

There are many ways to summarize the distribution of time to event. We will focus on two options: the survival at a particular time, and the restricted mean survival up to a given time. Let’s compare the survival at 7 years, or about 2500 days since registration.

plot(sfit[1], conf.int = FALSE, xlab = "days since registration", ylab = "survival")
seg0 <- summary(sfit[1], times = sfit[1]$time[sfit[1]$time <= 2500])
rect(c(0, seg0$time), 0, c(seg0$time, 2500), c(seg0$surv), 
     border = NA, col = "grey80")
lines(sfit[1], conf.int = FALSE)
abline(v = 2500, lty = 2)
points(x = 2500, y = summary(sfit[1], times = 2500)$surv)

In the figure above, we plot only the survival curve in the observation group. The vertical dotted line is at the time of interest (tmax = 2500 days). The open point is at the estimated survival probability at time tmax, i.e., \(P(T > tmax)\) and the shaded area represents the restricted mean survival up to tmax, i.e., \(E\{\min(T, tmax)\} = \int_0^{tmax} P(T > u) \, du\). We can estimate these things using the survival package:

colon.sfit <- summary(sfit, times = 2500, rmean = 2500)
colon.sfit
## Call: survfit(formula = Surv(time, status) ~ rx, data = colon)
## 
##                 rx=Obs 
##         time       n.risk      n.event     survival      std.err lower 95% CI 
##     2.50e+03     5.00e+01     1.65e+02     4.55e-01     2.98e-02     4.00e-01 
## upper 95% CI 
##     5.18e-01 
## 
##                 rx=Lev 
##         time       n.risk      n.event     survival      std.err lower 95% CI 
##     2.50e+03     5.80e+01     1.57e+02     4.85e-01     2.92e-02     4.31e-01 
## upper 95% CI 
##     5.46e-01 
## 
##                 rx=Lev+5FU 
##         time       n.risk      n.event     survival      std.err lower 95% CI 
##     2.50e+03     6.50e+01     1.21e+02     5.88e-01     2.96e-02     5.33e-01 
## upper 95% CI 
##     6.49e-01
colon.sfit$table
##            records n.max n.start events   *rmean *se(rmean) median 0.95LCL
## rx=Obs         315   315     315    168 1666.948   49.85546   2083    1656
## rx=Lev         310   310     310    161 1661.239   51.26220   2152    1540
## rx=Lev+5FU     304   304     304    123 1862.262   49.57411     NA    2725
##            0.95UCL
## rx=Obs        2789
## rx=Lev          NA
## rx=Lev+5FU      NA

And we can now do inference on these quantities using the eventglm package. First, we fit a regression model for the cumulative incidence, or 1 - survival:

colon.cifit <- cumincglm(Surv(time, status) ~ rx, time = 2500, data = colon)
summary(colon.cifit)
## 
## Call:
## cumincglm(formula = Surv(time, status) ~ rx, time = 2500, data = colon)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -0.5875  -0.4902  -0.3467   0.4863   2.1103  
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)    
## (Intercept)  0.54345    0.02946  18.449  < 2e-16 ***
## rxLev       -0.02907    0.04173  -0.697  0.48596    
## rxLev+5FU   -0.13176    0.04186  -3.148  0.00165 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for quasi family taken to be 1)
## 
##     Null deviance: 253.10  on 928  degrees of freedom
## Residual deviance: 250.15  on 926  degrees of freedom
## AIC: NA
## 
## Number of Fisher Scoring iterations: 2
se.ci <- sqrt(diag(vcov(colon.cifit, type = "robust")))
b.ci <- coefficients(colon.cifit)
conf.ci <- confint(colon.cifit)

In the model above, which is fit using the function cumincglm, the model is for the cumulative incidence of death up to 2500 days: \[ P(T \leq 2500 | \mbox{rx}) = \beta_0 + \beta_1 I(rx = "Lev") + \beta_2 I(rx = "Lev+5FU"). \]

Exercise

What is the interpretation of the coefficients in the model? How do the estimates from the eventglm model compare to the Kaplan-Meier estimate?

Solution

The cumulative incidence of death up to 2500 days in the Observation arm is the intercept 0.54. The first coefficient is the expected difference in cumulative incidence of death up to 2500 days comparing the Levamisole alone treatment group to the Observation arm: an estimated -0.03 difference in the cumulative incidence of death at 2500 days, with 95% confidence interval -0.11, 0.05, while the Levamisole plus 5-FU group has a -0.13 difference in the cumulative incidence of death at 2500 days, with 95% confidence interval -0.21, -0.05.

Loosely speaking, the coefficients are risk differences, where the risk refers to the probability of dying within 2500 days. This roughly agrees with the Kaplan-Meier estimates from survfit above:

cbind(eventglm = b.ci, 
      survfit = c(1 - colon.sfit$surv[1], 
  (1 - colon.sfit$surv[2:3]) - 
    (1 - rep(colon.sfit$surv[1], 2))))
##                eventglm     survfit
## (Intercept)  0.54345139  0.54479221
## rxLev       -0.02907499 -0.02990601
## rxLev+5FU   -0.13175778 -0.13301654

Other estimands

Survival

Look at the help file for cumincglm. There is a long list of options for the function, but one important one is the survival argument which is FALSE by default. If we set this to TRUE:

colon.survfit <- cumincglm(Surv(time, status) ~ rx, time = 2500, 
                           survival = TRUE, data = colon)
summary(colon.survfit)
## 
## Call:
## cumincglm(formula = Surv(time, status) ~ rx, time = 2500, data = colon, 
##     survival = TRUE)
## 
## Deviance Residuals: 
##    Min      1Q  Median      3Q     Max  
##                                         
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)    
## (Intercept)  0.45655    0.02946  15.498  < 2e-16 ***
## rxLev        0.02907    0.04173   0.697  0.48596    
## rxLev+5FU    0.13176    0.04186   3.148  0.00165 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for quasi family taken to be 1)
## 
##     Null deviance: 253.10  on 928  degrees of freedom
## Residual deviance: 250.15  on 926  degrees of freedom
## AIC: NA
## 
## Number of Fisher Scoring iterations: 2
b2.ci <- coefficients(colon.survfit)
conf2.ci <- confint(colon.survfit)

we get a model for the probability of surviving beyond 2500 days: \[ P(T > 2500 | \mbox{rx}) = \beta_0 + \beta_1 I(rx = "Lev") + \beta_2 I(rx = "Lev+5FU"). \]

Stop and think

What is the interpretation of the coefficients in the model? How do they differ from the first model?

Restricted mean

There is another function called rmeanglm that fits regression models for the restricted mean. The syntax is very similar:

colon.rmeanfit <- rmeanglm(Surv(time, status) ~ rx, time = 2500, data = colon)
colon.rmeanfit
## 
## Call:  rmeanglm(formula = Surv(time, status) ~ rx, time = 2500, data = colon)
## 
## 
## Model for the identity restricted mean at time 2500 
## 
## Coefficients:
## (Intercept)        rxLev    rxLev+5FU  
##    1667.403       -6.074      194.954  
## 
## Degrees of Freedom: 928 Total (i.e. Null);  926 Residual
b.rm <- coefficients(colon.rmeanfit)
conf.rm <- confint(colon.rmeanfit)

This is a model for the restricted mean up to 2500 days (where \(x \wedge y = \min(x, y)\)): \[ E(T \wedge 2500 | \mbox{rx}) = \beta_0 + \beta_1 I(rx = "Lev") + \beta_2 I(rx = "Lev+5FU"). \]

We can compare these to the estimates from the Kaplan-Meier fits as before. Here the coefficients are interpreted as restricted mean differences.

cbind(eventglm = b.rm, 
      survfit = c(colon.sfit$table[1,5], 
  colon.sfit$table[2:3,5] - rep(colon.sfit$table[1,5], 2)))
##               eventglm     survfit
## (Intercept) 1667.40308 1666.948078
## rxLev         -6.07367   -5.708803
## rxLev+5FU    194.95446  195.313754

Stop and think

What is the point of having regression models for these quantities that we can easily get from survfit?

Solution

The regression models are much more flexible. Yes survfit can give you estimates in a small number of subgroups, but with eventglm it is trivial to do the following things:

  • Inference comparing groups on different scales, including the difference, ratio, odds …
  • Flexible modeling of continuous variables
  • Adjustment for confounding
  • Interaction terms
  • Prediction
  • Modeling of multiple time points
  • Modeling with competing risks and in more general multi-state models
  • Joint modeling of multiple outcomes

As we continue this tutorial we will explore these possibilities and learn more about how these types of models work.

LS0tDQp0aXRsZTogIk92ZXJ2aWV3IG9mIGVzdGltYW5kcyBhbmQgYmFzaWMgcmVncmVzc2lvbiBtb2RlbHMiDQpvdXRwdXQ6DQogIGh0bWxfZG9jdW1lbnQ6DQogICAgY29kZV9mb2xkaW5nOiBzaG93DQotLS0NCg0KIyMjIExlYXJuaW5nIG9iamVjdGl2ZXMgey5hbGVydCAuYWxlcnQtc3VjY2Vzc30NCg0KSW4gdGhpcyBsZXNzb24geW91IHdpbGwgDQoNCjEuIFVuZGVyc3RhbmQgdGhlIGFwcGxpZWQgY29udGV4dCBpbiB3aGljaCBgZXZlbnRnbG1gIGlzIHVzZWQNCjIuIEZpdCBzb21lIGJhc2ljIG1vZGVscywgaW50ZXJwcmV0aW5nIHRoZSByZXN1bHRpbmcgZXN0aW1hdGVzDQozLiBDb21wYXJlIHRoZSBgZXZlbnRnbG1gIG1vZGVscyBhbmQgZXN0aW1hdGVzIHRvIHdoYXQgeW91IGdldCBmcm9tIHRoZSBzdXJ2aXZhbCBwYWNrYWdlDQoNCg0KIyMgU2V0dXANCg0KTGlicmFyeSB0aGUgYHN1cnZpdmFsYCBwYWNrYWdlIGFuZCB0aGVuIHRoZSBgZXZlbnRnbG1gIHBhY2thZ2UuIFdlIHdpbGwgdXNlIHNvbWUgY29tbWFuZHMgZnJvbSBgc3Vydml2YWxgIHRvIG1ha2Ugc2Vuc2Ugb2Ygd2hhdCBpcyBnb2luZyBvbiBpbiB0aGUgcmVncmVzc2lvbiBtb2RlbHMuDQoNCg0KYGBge3J9DQpsaWJyYXJ5KHN1cnZpdmFsKQ0KbGlicmFyeShldmVudGdsbSkNCmBgYA0KDQoNCiMjIFdoYXQgaXMgdGhlIG91dGNvbWU/IA0KDQojIyMgQ29sb24gY2FuY2VyIHRyZWF0bWVudCBhbmQgdGltZSB0byBkZWF0aCAoc3RhbmRhcmQgc3Vydml2YWwpDQoNCk91ciBmaXJzdCBleGFtcGxlIGNvbmNlcm5zIHRoZSBgY29sb25gIGRhdGFzZXQsIHdoaWNoIGlzIGluY2x1ZGVkIGluIHRoZSBwYWNrYWdlOiANCg0KYGBge3IsIGV2YWwgPSBGQUxTRX0NCj9ldmVudGdsbTo6Y29sb24NCmBgYA0KDQpUaGlzIGlzIGEgcmFuZG9taXplZCB0cmlhbCwgc28gdGhlIG1haW4gaW50ZXJlc3QgaXMgaW4gY29tcGFyaW5nIHRoZSBkaXN0cmlidXRpb25zIG9mIHRpbWUgdG8gZGVhdGggYmV0d2VlbiB0aGUgdGhyZWUgdHJlYXRtZW50IGFybXMuIExldCdzIHN0YXJ0IHdpdGggYSBzdXJ2aXZhbCBjdXJ2ZS4gDQoNCmBgYHtyfQ0Kc2ZpdCA8LSBzdXJ2Zml0KFN1cnYodGltZSwgc3RhdHVzKSB+IHJ4LCBkYXRhID0gY29sb24pDQpwbG90KHNmaXQsIGNvbCA9IGMoImJsYWNrIiwgInNsYXRlYmx1ZSIsICJzYWxtb24iKSwgDQogICAgIHhsYWIgPSAiZGF5cyBzaW5jZSByZWdpc3RyYXRpb24iLCB5bGFiID0gInN1cnZpdmFsIikNCmxlZ2VuZCgiYm90dG9tbGVmdCIsIGZpbGwgPSBjKCJibGFjayIsICJzbGF0ZWJsdWUiLCAic2FsbW9uIiksIA0KICAgICAgIGxlZ2VuZCA9IG5hbWVzKHNmaXQkc3RyYXRhKSkNCmBgYA0KDQpUaGVyZSBhcmUgbWFueSB3YXlzIHRvIHN1bW1hcml6ZSB0aGUgZGlzdHJpYnV0aW9uIG9mIHRpbWUgdG8gZXZlbnQuIFdlIHdpbGwgZm9jdXMgb24gdHdvIG9wdGlvbnM6IHRoZSBzdXJ2aXZhbCBhdCBhIHBhcnRpY3VsYXIgdGltZSwgYW5kIHRoZSByZXN0cmljdGVkIG1lYW4gc3Vydml2YWwgdXAgdG8gYSBnaXZlbiB0aW1lLiBMZXQncyBjb21wYXJlIHRoZSBzdXJ2aXZhbCBhdCA3IHllYXJzLCBvciBhYm91dCAyNTAwIGRheXMgc2luY2UgcmVnaXN0cmF0aW9uLiANCg0KYGBge3J9DQpwbG90KHNmaXRbMV0sIGNvbmYuaW50ID0gRkFMU0UsIHhsYWIgPSAiZGF5cyBzaW5jZSByZWdpc3RyYXRpb24iLCB5bGFiID0gInN1cnZpdmFsIikNCnNlZzAgPC0gc3VtbWFyeShzZml0WzFdLCB0aW1lcyA9IHNmaXRbMV0kdGltZVtzZml0WzFdJHRpbWUgPD0gMjUwMF0pDQpyZWN0KGMoMCwgc2VnMCR0aW1lKSwgMCwgYyhzZWcwJHRpbWUsIDI1MDApLCBjKHNlZzAkc3VydiksIA0KICAgICBib3JkZXIgPSBOQSwgY29sID0gImdyZXk4MCIpDQpsaW5lcyhzZml0WzFdLCBjb25mLmludCA9IEZBTFNFKQ0KYWJsaW5lKHYgPSAyNTAwLCBsdHkgPSAyKQ0KcG9pbnRzKHggPSAyNTAwLCB5ID0gc3VtbWFyeShzZml0WzFdLCB0aW1lcyA9IDI1MDApJHN1cnYpDQpgYGANCg0KSW4gdGhlIGZpZ3VyZSBhYm92ZSwgd2UgcGxvdCBvbmx5IHRoZSBzdXJ2aXZhbCBjdXJ2ZSBpbiB0aGUgb2JzZXJ2YXRpb24gZ3JvdXAuIFRoZSB2ZXJ0aWNhbCBkb3R0ZWQgbGluZSBpcyBhdCB0aGUgdGltZSBvZiBpbnRlcmVzdCAodG1heCA9IDI1MDAgZGF5cykuIFRoZSBvcGVuIHBvaW50IGlzIGF0IHRoZSBlc3RpbWF0ZWQgc3Vydml2YWwgcHJvYmFiaWxpdHkgYXQgdGltZSB0bWF4LCBpLmUuLCAkUChUID4gdG1heCkkIGFuZCB0aGUgc2hhZGVkIGFyZWEgcmVwcmVzZW50cyB0aGUgcmVzdHJpY3RlZCBtZWFuIHN1cnZpdmFsIHVwIHRvIHRtYXgsIGkuZS4sICRFXHtcbWluKFQsIHRtYXgpXH0gPSBcaW50XzBee3RtYXh9IFAoVCA+IHUpIFwsIGR1JC4gV2UgY2FuIGVzdGltYXRlIHRoZXNlIHRoaW5ncyB1c2luZyB0aGUgYHN1cnZpdmFsYCBwYWNrYWdlOiANCg0KYGBge3J9DQpjb2xvbi5zZml0IDwtIHN1bW1hcnkoc2ZpdCwgdGltZXMgPSAyNTAwLCBybWVhbiA9IDI1MDApDQpjb2xvbi5zZml0DQpjb2xvbi5zZml0JHRhYmxlDQpgYGANCg0KQW5kIHdlIGNhbiBub3cgZG8gaW5mZXJlbmNlIG9uIHRoZXNlIHF1YW50aXRpZXMgdXNpbmcgdGhlIGBldmVudGdsbWAgcGFja2FnZS4gRmlyc3QsIHdlIGZpdCBhIHJlZ3Jlc3Npb24gbW9kZWwgZm9yIHRoZSBjdW11bGF0aXZlIGluY2lkZW5jZSwgb3IgMSAtIHN1cnZpdmFsOiAgDQoNCmBgYHtyfQ0KY29sb24uY2lmaXQgPC0gY3VtaW5jZ2xtKFN1cnYodGltZSwgc3RhdHVzKSB+IHJ4LCB0aW1lID0gMjUwMCwgZGF0YSA9IGNvbG9uKQ0Kc3VtbWFyeShjb2xvbi5jaWZpdCkNCnNlLmNpIDwtIHNxcnQoZGlhZyh2Y292KGNvbG9uLmNpZml0LCB0eXBlID0gInJvYnVzdCIpKSkNCmIuY2kgPC0gY29lZmZpY2llbnRzKGNvbG9uLmNpZml0KQ0KY29uZi5jaSA8LSBjb25maW50KGNvbG9uLmNpZml0KQ0KYGBgDQoNCg0KSW4gdGhlIG1vZGVsIGFib3ZlLCB3aGljaCBpcyBmaXQgdXNpbmcgdGhlIGZ1bmN0aW9uIGBjdW1pbmNnbG1gLCB0aGUgbW9kZWwgaXMgZm9yIHRoZSBjdW11bGF0aXZlIGluY2lkZW5jZSBvZiBkZWF0aCB1cCB0byAyNTAwIGRheXM6IA0KXFsNClAoVCBcbGVxIDI1MDAgfCBcbWJveHtyeH0pID0gXGJldGFfMCArIFxiZXRhXzEgSShyeCA9ICJMZXYiKSArIFxiZXRhXzIgSShyeCA9ICJMZXYrNUZVIikuDQpcXSANCg0KDQoNCiMjIyBFeGVyY2lzZSB7LmFsZXJ0IC5hbGVydC13YXJuaW5nfQ0KDQpXaGF0IGlzIHRoZSBpbnRlcnByZXRhdGlvbiBvZiB0aGUgY29lZmZpY2llbnRzIGluIHRoZSBtb2RlbD8gSG93IGRvIHRoZSBlc3RpbWF0ZXMgZnJvbSB0aGUgZXZlbnRnbG0gbW9kZWwgY29tcGFyZSB0byB0aGUgS2FwbGFuLU1laWVyIGVzdGltYXRlPyANCg0KPGRldGFpbHM+DQo8c3VtbWFyeT48c3Ryb25nPlNvbHV0aW9uPC9zdHJvbmc+PC9zdW1tYXJ5Pg0KVGhlIGN1bXVsYXRpdmUgaW5jaWRlbmNlIG9mIGRlYXRoIHVwIHRvIDI1MDAgZGF5cyBpbiB0aGUgT2JzZXJ2YXRpb24gYXJtIGlzIHRoZSBpbnRlcmNlcHQgYHIgcm91bmQoYi5jaVsxXSwgMilgLiBUaGUgZmlyc3QgY29lZmZpY2llbnQgaXMgdGhlIGV4cGVjdGVkIGRpZmZlcmVuY2UgaW4gY3VtdWxhdGl2ZSBpbmNpZGVuY2Ugb2YgZGVhdGggdXAgdG8gMjUwMCBkYXlzIGNvbXBhcmluZyB0aGUgTGV2YW1pc29sZSBhbG9uZSB0cmVhdG1lbnQgZ3JvdXAgdG8gdGhlIE9ic2VydmF0aW9uIGFybTogYW4gZXN0aW1hdGVkIGByIHJvdW5kKGIuY2lbMl0sIDIpYCBkaWZmZXJlbmNlIGluIHRoZSBjdW11bGF0aXZlIGluY2lkZW5jZSBvZiBkZWF0aCBhdCAyNTAwIGRheXMsIHdpdGggOTVcJSBjb25maWRlbmNlIGludGVydmFsIGByIHJvdW5kKGNvbmYuY2lbMixdLCAyKWAsIHdoaWxlIHRoZSBMZXZhbWlzb2xlIHBsdXMgNS1GVSBncm91cCBoYXMgYSBgciByb3VuZChiLmNpWzNdLCAyKWAgZGlmZmVyZW5jZSBpbiB0aGUgY3VtdWxhdGl2ZSBpbmNpZGVuY2Ugb2YgZGVhdGggYXQgMjUwMCBkYXlzLCB3aXRoIDk1XCUgY29uZmlkZW5jZSBpbnRlcnZhbCBgciByb3VuZChjb25mLmNpWzMsXSwgMilgLiANCg0KTG9vc2VseSBzcGVha2luZywgdGhlIGNvZWZmaWNpZW50cyBhcmUgcmlzayBkaWZmZXJlbmNlcywgd2hlcmUgdGhlIHJpc2sgcmVmZXJzIHRvIHRoZSBwcm9iYWJpbGl0eSBvZiBkeWluZyB3aXRoaW4gMjUwMCBkYXlzLiBUaGlzIHJvdWdobHkgYWdyZWVzIHdpdGggdGhlIEthcGxhbi1NZWllciBlc3RpbWF0ZXMgZnJvbSBzdXJ2Zml0IGFib3ZlOiANCg0KYGBge3J9DQpjYmluZChldmVudGdsbSA9IGIuY2ksIA0KICAgICAgc3VydmZpdCA9IGMoMSAtIGNvbG9uLnNmaXQkc3VydlsxXSwgDQogICgxIC0gY29sb24uc2ZpdCRzdXJ2WzI6M10pIC0gDQogICAgKDEgLSByZXAoY29sb24uc2ZpdCRzdXJ2WzFdLCAyKSkpKQ0KYGBgDQo8L2RldGFpbHM+DQoNCg0KIyMjIE90aGVyIGVzdGltYW5kcw0KDQojIyMjIFN1cnZpdmFsDQoNCkxvb2sgYXQgdGhlIGhlbHAgZmlsZSBmb3IgYGN1bWluY2dsbWAuIFRoZXJlIGlzIGEgbG9uZyBsaXN0IG9mIG9wdGlvbnMgZm9yIHRoZSBmdW5jdGlvbiwgYnV0IG9uZSBpbXBvcnRhbnQgb25lIGlzIHRoZSBgc3Vydml2YWxgIGFyZ3VtZW50IHdoaWNoIGlzIGBGQUxTRWAgYnkgZGVmYXVsdC4gSWYgd2Ugc2V0IHRoaXMgdG8gYFRSVUVgOg0KDQoNCmBgYHtyfQ0KY29sb24uc3VydmZpdCA8LSBjdW1pbmNnbG0oU3Vydih0aW1lLCBzdGF0dXMpIH4gcngsIHRpbWUgPSAyNTAwLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgIHN1cnZpdmFsID0gVFJVRSwgZGF0YSA9IGNvbG9uKQ0Kc3VtbWFyeShjb2xvbi5zdXJ2Zml0KQ0KYjIuY2kgPC0gY29lZmZpY2llbnRzKGNvbG9uLnN1cnZmaXQpDQpjb25mMi5jaSA8LSBjb25maW50KGNvbG9uLnN1cnZmaXQpDQpgYGANCg0Kd2UgZ2V0IGEgbW9kZWwgZm9yIHRoZSBwcm9iYWJpbGl0eSBvZiBzdXJ2aXZpbmcgYmV5b25kIDI1MDAgZGF5czogDQpcWw0KUChUID4gMjUwMCB8IFxtYm94e3J4fSkgPSBcYmV0YV8wICsgXGJldGFfMSBJKHJ4ID0gIkxldiIpICsgXGJldGFfMiBJKHJ4ID0gIkxldis1RlUiKS4NClxdIA0KDQoNCiMjIyMgU3RvcCBhbmQgdGhpbmsgey5hbGVydCAuYWxlcnQtaW5mb30NCg0KV2hhdCBpcyB0aGUgaW50ZXJwcmV0YXRpb24gb2YgdGhlIGNvZWZmaWNpZW50cyBpbiB0aGUgbW9kZWw/IEhvdyBkbyB0aGV5IGRpZmZlciBmcm9tIHRoZSBmaXJzdCBtb2RlbD8gDQoNCg0KIyMjIyBMaW5rIGZ1bmN0aW9ucw0KDQpPbmUgb2YgdGhlIGFyZ3VtZW50cyBmb3IgYGN1bWluY2dsbWAgaXMgYGxpbmtgLCB3aGljaCBpcyAiaWRlbnRpdHkiIGJ5IGRlZmF1bHQuIE90aGVyIGxpbmsgZnVuY3Rpb25zIGNhbiBiZSB1c2VkIGFuZCB0aGUgaW50ZXJwcmV0YXRpb24gb2YgdGhlIGNvZWZmaWNpZW50cyBkaWZmZXJzIGRlcGVuZGluZyBvbiB0aGUgbGluayBmdW5jdGlvbi4gVGhlIGdlbmVyYWwgZ2VuZXJhbGl6ZWQgbGluZWFyIG1vZGVsIGluIG91ciBleGFtcGxlIGlzDQpcWw0KZ1x7UChUIFxsZXEgMjUwMCB8IFxtYm94e3J4fSlcfSA9IFxiZXRhXzAgKyBcYmV0YV8xIEkocnggPSAiTGV2IikgKyBcYmV0YV8yIEkocnggPSAiTGV2KzVGVSIpDQpcXQ0Kd2hlcmUgJGckIGlzIHRoZSBsaW5rIGZ1bmN0aW9uLiBMZXQncyB1c2UgJGcoeCkgPSBcbG9nKHgpJCBhbmQgc2VlIHdoYXQgaGFwcGVucy4gDQoNCmBgYHtyfQ0KY29sb24ubG9nY2kgPC0gY3VtaW5jZ2xtKFN1cnYodGltZSwgc3RhdHVzKSB+IHJ4LCB0aW1lID0gMjUwMCwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgIGxpbmsgPSAibG9nIiwgZGF0YSA9IGNvbG9uKQ0KY29sb24ubG9nY2kNCmBgYA0KDQpJbiB0aGUgYWJvdmUgbW9kZWwsIHRoZSBpbnRlcmNlcHQgaXMgaW50ZXJwcmV0ZWQgYXMgdGhlIGxvZyBjdW11bGF0aXZlIGluY2lkZW5jZSBpbiB0aGUgb2JzZXJ2YXRpb24gYXJtLiBUaGUgY29lZmZpY2llbnQgZm9yIExldiBpcw0KXFsNClxsb2dce1AoVCBcbGVxIDI1MDAgfCBcbWJveHtyeD1MZXZ9XH0gLSBcbG9nXHtQKFQgXGxlcSAyNTAwIHwgXG1ib3h7cng9T2JzfVx9ID0gDQpcXQ0KXFsNClxsb2dce1xmcmFje1AoVCBcbGVxIDI1MDAgfCBcbWJveHtyeD1MZXZ9fXtQKFQgXGxlcSAyNTAwIHwgXG1ib3h7cng9T2JzfX1cfSwgDQpcXQ0KdGhlIGxvZyBfcmVsYXRpdmUgcmlza18gb3IgbG9nIF9yaXNrIHJhdGlvXy4gVGh1cyB0byBnZXQgdGhlIHJlbGF0aXZlIHJpc2ssIGFuIGludGVycHJldGFibGUgcXVhbnRpdHksIHdlIGNhbiBleHBvbmVudGlhdGUgdGhlIGNvZWZmaWNpZW50IGVzdGltYXRlcyBhbmQgY29uZmlkZW5jZSBpbnRlcnZhbCBsaW1pdHM6IA0KDQpgYGB7cn0NCmV4cChjb2VmKGNvbG9uLmxvZ2NpKSkNCmV4cChjb25maW50KGNvbG9uLmxvZ2NpKSkNCmBgYA0KDQpPdGhlciBsaW5rIGZ1bmN0aW9ucyBhcmUgcG9zc2libGUuIEZvciBleGFtcGxlLCAibG9naXQiIGlzIGEgY29udmVuaWVudCB3YXkgdG8gZXN0aW1hdGUgb2RkcyBhbmQgb2RkcyByYXRpb3MsIGFuZCBhICJjbG9nbG9nIiBsaW5rIGZvciB0aGUgY3VtdWxhdGl2ZSBpbmNpZGVuY2UgaXMgcmVsYXRlZCB0byBhIHByb3BvcnRpb25hbCBoYXphcmRzIG1vZGVsLiBXZSB3aWxsIHJldHVybiB0byB0aGlzIGluIHRoZSBuZXh0IHNlY3Rpb24uDQoNCg0KDQojIyMjIFJlc3RyaWN0ZWQgbWVhbg0KDQpUaGVyZSBpcyBhbm90aGVyIGZ1bmN0aW9uIGNhbGxlZCBgcm1lYW5nbG1gIHRoYXQgZml0cyByZWdyZXNzaW9uIG1vZGVscyBmb3IgdGhlIHJlc3RyaWN0ZWQgbWVhbi4gVGhlIHN5bnRheCBpcyB2ZXJ5IHNpbWlsYXI6IA0KDQpgYGB7cn0NCmNvbG9uLnJtZWFuZml0IDwtIHJtZWFuZ2xtKFN1cnYodGltZSwgc3RhdHVzKSB+IHJ4LCB0aW1lID0gMjUwMCwgZGF0YSA9IGNvbG9uKQ0KY29sb24ucm1lYW5maXQNCmIucm0gPC0gY29lZmZpY2llbnRzKGNvbG9uLnJtZWFuZml0KQ0KY29uZi5ybSA8LSBjb25maW50KGNvbG9uLnJtZWFuZml0KQ0KYGBgDQoNClRoaXMgaXMgYSBtb2RlbCBmb3IgdGhlIHJlc3RyaWN0ZWQgbWVhbiB1cCB0byAyNTAwIGRheXMgKHdoZXJlICR4IFx3ZWRnZSB5ID0gXG1pbih4LCB5KSQpOiANClxbDQpFKFQgXHdlZGdlIDI1MDAgfCBcbWJveHtyeH0pID0gXGJldGFfMCArIFxiZXRhXzEgSShyeCA9ICJMZXYiKSArIFxiZXRhXzIgSShyeCA9ICJMZXYrNUZVIikuDQpcXSANCg0KV2UgY2FuIGNvbXBhcmUgdGhlc2UgdG8gdGhlIGVzdGltYXRlcyBmcm9tIHRoZSBLYXBsYW4tTWVpZXIgZml0cyBhcyBiZWZvcmUuIEhlcmUgdGhlIGNvZWZmaWNpZW50cyBhcmUgaW50ZXJwcmV0ZWQgYXMgcmVzdHJpY3RlZCBtZWFuIGRpZmZlcmVuY2VzLiANCg0KYGBge3J9DQpjYmluZChldmVudGdsbSA9IGIucm0sIA0KICAgICAgc3VydmZpdCA9IGMoY29sb24uc2ZpdCR0YWJsZVsxLDVdLCANCiAgY29sb24uc2ZpdCR0YWJsZVsyOjMsNV0gLSByZXAoY29sb24uc2ZpdCR0YWJsZVsxLDVdLCAyKSkpDQpgYGANCg0KIyMjIyBTdG9wIGFuZCB0aGluayB7LmFsZXJ0IC5hbGVydC1pbmZvfQ0KDQpXaGF0IGlzIHRoZSBwb2ludCBvZiBoYXZpbmcgcmVncmVzc2lvbiBtb2RlbHMgZm9yIHRoZXNlIHF1YW50aXRpZXMgdGhhdCB3ZSBjYW4gZWFzaWx5IGdldCBmcm9tIGBzdXJ2Zml0YD8NCg0KPGRldGFpbHM+DQo8c3VtbWFyeT48c3Ryb25nPlNvbHV0aW9uPC9zdHJvbmc+PC9zdW1tYXJ5Pg0KVGhlIHJlZ3Jlc3Npb24gbW9kZWxzIGFyZSBtdWNoIG1vcmUgZmxleGlibGUuIFllcyBgc3VydmZpdGAgY2FuIGdpdmUgeW91IGVzdGltYXRlcyBpbiBhIHNtYWxsIG51bWJlciBvZiBzdWJncm91cHMsIGJ1dCB3aXRoIGBldmVudGdsbWAgaXQgaXMgdHJpdmlhbCB0byBkbyB0aGUgZm9sbG93aW5nIHRoaW5nczoNCg0KLSBJbmZlcmVuY2UgY29tcGFyaW5nIGdyb3VwcyBvbiBkaWZmZXJlbnQgc2NhbGVzLCBpbmNsdWRpbmcgdGhlIGRpZmZlcmVuY2UsIHJhdGlvLCBvZGRzIC4uLg0KLSBGbGV4aWJsZSBtb2RlbGluZyBvZiBjb250aW51b3VzIHZhcmlhYmxlcw0KLSBBZGp1c3RtZW50IGZvciBjb25mb3VuZGluZw0KLSBJbnRlcmFjdGlvbiB0ZXJtcw0KLSBQcmVkaWN0aW9uDQotIE1vZGVsaW5nIG9mIG11bHRpcGxlIHRpbWUgcG9pbnRzDQotIE1vZGVsaW5nIHdpdGggY29tcGV0aW5nIHJpc2tzIGFuZCBpbiBtb3JlIGdlbmVyYWwgbXVsdGktc3RhdGUgbW9kZWxzDQotIEpvaW50IG1vZGVsaW5nIG9mIG11bHRpcGxlIG91dGNvbWVzDQoNCkFzIHdlIGNvbnRpbnVlIHRoaXMgdHV0b3JpYWwgd2Ugd2lsbCBleHBsb3JlIHRoZXNlIHBvc3NpYmlsaXRpZXMgYW5kIGxlYXJuIG1vcmUgYWJvdXQgaG93IHRoZXNlIHR5cGVzIG9mIG1vZGVscyB3b3JrLiANCg0KPC9kZXRhaWxzPg0KDQoNCg0KDQo=