Last updated: 2017-06-20

Code version: ac13b67

rank 0 vs rank 1

this document is based on Matthew’s suggestion:

If you simulate rank 0, and run greedy with K=1:

  • how often does the K=1 converge to a point mass at 0?

  • how often does the K=1 converge to a lower bound that is worse than the lower bound for K=0?

Then you could do the same question for method =fdr (but with this penalized objective function).

shrink method without penalty term in objective function

we find that we might need fdr method and we should use the objective funciton with penalty.rank 0 vs rank 1

check the objective function with penalty

In this case, we check the objective funciton with penalty term, and it does monotonically increase. rank 0 vs rank 1: fdr with penalty

now we have two methods (fdr, shrink), and both of them have monotonically increasing corresponding objective function in each case.

  1. fit flash (with or without penalty)

  2. check if flash-obtained objective function exceeds the K=0 objective. (note: not the limit of the objective as g_f, g_l -> 0, but the actual objective at g_f=g_l=0). If not, set result to g_f=g_l = 0. If so, keep g_f, g_l as output by flash.

– Matthew’s advice.

  • we have done the (i), and we found that it is reasonable to do the (ii). This is because we find that \(diff(LB(fitted) - LB(rank=0))\) is negatively correlated with RMSE, which means larger obj value goes with smaller RMSE. So it makes sense we cut the obj value off at the rank 0 model.
    1. is easy to implement.

questions

  1. how to deal with \((\lambda_0 - 1) log \pi_0\) when \(\pi_0 = 0\)?
  2. now we only have two cases: shrink with \(\lambda_0 = 1\) and fdr with \(\lambda_0 = 10\). Do we need more choice of \(\lambda_0\)?

fdr method with penalty term in objective function

we can see the improvement that the proportion of P(estimated structure better than rank 0 structure) is much larger. rank 0 vs rank 1: fdr with penalty

Session information

sessionInfo()
R version 3.3.0 (2016-05-03)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.12.4 (unknown)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] workflowr_0.4.0 rmarkdown_1.3  

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.11      knitr_1.15.1      magrittr_1.5     
 [4] REBayes_0.73      MASS_7.3-45       munsell_0.4.3    
 [7] doParallel_1.0.10 pscl_1.4.9        colorspace_1.3-2 
[10] SQUAREM_2016.8-2  lattice_0.20-34   foreach_1.4.3    
[13] plyr_1.8.4        flashr_0.1.1      ashr_2.1-21      
[16] stringr_1.2.0     tools_3.3.0       parallel_3.3.0   
[19] grid_3.3.0        gtable_0.2.0      irlba_2.1.2      
[22] git2r_0.18.0      htmltools_0.3.5   iterators_1.0.8  
[25] lazyeval_0.2.0    assertthat_0.2.0  yaml_2.1.14      
[28] rprojroot_1.2     digest_0.6.12     tibble_1.2       
[31] Matrix_1.2-8      ggplot2_2.2.1     codetools_0.2-15 
[34] evaluate_0.10     stringi_1.1.2     scales_0.4.1     
[37] Rmosek_7.1.2      backports_1.0.5   truncnorm_1.0-7

This R Markdown site was created with workflowr