The answer to this question depends on the context. If the gems are part of a traditional cultural practice, such as a religious ceremony or a cultural celebration, then wearing them could be seen as cultural appropriation. However, if the gems are simply being worn as a fashion statement, then it is unlikely to be seen as cultural appropriation.
