Will upgrading to 16GB of RAM improve ZFS deduplication performance?

No, upgrading to 16GB will not meaningfully improve deduplication performance. Deduplication on ZFS needs the dedup table (DDT) to live entirely in RAM, and even modest datasets can require gigabytes for the DDT. With 8GB you’re already in a bad spot, and 16GB still isn’t enough — the table will spill to disk and kill performance. For a small home NAS, you usually need 64GB or more for dedup to work well. What 16GB will do is improve caching via the ARC.