One suggestion is to have more practice speaking with an English native speaker so that it would be easier for you to understand this problem.
Join the waitlist for the complete English course: https://gonaturalenglish.com/prereg/
\u201cI don\u2019t care\u201d can be very offensive because it could mean that it\u2019s not important to me or it\u2019s not much of a big deal for me.\xa0
For example:
\u201cDo you want to paint the wall blue or brown?\u201d
\u201cI don\u2019t care.\u201d
or
\u201cI don\u2019t care. Whatever you like.\u201d
Your tone and your expression here is the most important tip in speaking English.
So if you want to be on the safe side, it\u2019s recommendable to use, \u201cI don\u2019t mind.\u201d