Finite difference formulas approximate the derivatives of a function given its values at a discrete set of grid points. Much of the theory for choosing grid points is concerned with discretization errors and rounding errors are ignored completely. However, when the grids become fine the rounding errors dominate. Finding finite difference formulas which optimize the total error leads to a combinatorial optimization problem with a large search space. In this talk, we will describe a random walk based strategy for tackling the combinatorial optimization problem.