So which method should you use? It depends on the application, and maybe on your choice of language. With regard to the latter, I simply concede that if you are using the VB.NET IDE, it is very easy to just use IsNumeric, and since it is fast and flexible, it may be the option of choice in that environment. But even if you are using the VB.NET IDE, you may want to consider using Double.TryParse.
In any case, keep in mind that for the given regular expression and incremental char comparison implementations, the definition of "IsNumeric" is limited to digits with one optional decimal. If you needed to expand this definition to include, e.g., allowing globalization, allowing signs, allowing scientific notation, etc., you could customize them to meet your definition of what a number is, but I would expect that as you get more flexible with the definition, these methods would start to lose out in raw performance (and simplicity).
Therefore, given the insignificant practical performance difference between the Incremental Char, Double.TryParse, and VisualBasic IsNumeric methods, I would choose either Double.TryParse or VB IsNumeric since their definitions of "IsNumeric" include many common variations not accounted for in the Regular Expression or Incremental Char. Clearly, using one of the Try-Catch methods is a bad idea due to the terrible performance for bad inputs; only use these if you truly do need to handle the exception.
For most applications, the best method will be TryParse because it offers the most flexibility in definition, the ability to return the converted value concurrently with the type check, and it is part of the System library, so no extra references or work is needed to use it, no matter what language you choose. If you find it too cumbersome to use directly (because it offers no overloads), you can always wrap it in a static IsNumeric method that only requires the input to check as a parameter.