Should I worry about mousepad color?

No, you can ignore the color. Modern optical and laser sensors track fine on any non-mirror, non-clear-glass surface. The texture and finish matter far more than the shade. A glossy or perfectly uniform surface can cause issues, but a standard cloth pad in any color works perfectly.