Harold J Kushner, Paul G Dupuis, Numerical methods for stochastic control problems in continuous time, Applications of mathematics 24, Springer-Verlag, 1992.
P E Kloeden, Eckhard Platen, Numerical Solutions of Stochastic Differential Equations, Applications of Mathematics 23, Stochastic Modelling and Applied probability, Springer, 1992.
F. B. Hanson, "Markov Chain Approximation", in C. T. Leondes, ed., Stochastic Digital Control System Techniques, Academic Press, 1996, ISBN 978-0120127764. http://homepages.math.uic.edu/~hanson/pub/CADS96/cads96mca.pdf