Should I worry about matching my mousepad color to my mouse's LED?

No. Modern optical sensors track surface texture, not color, so the LED color has no effect on performance. Tracking issues at high DPI come from uneven surfaces or dust, not color contrast. A solid-color cloth pad with a tight weave will outperform any matching setup.