The usual notation for [x] is the greatest integer that is equal to or less than x.
For example, [2.5]=2, [-3.4]=-4, but -[3.4]=-3 because the [ ] function applies to a positive number. The greatest integer not exceeding 3.4 is 3. On the other hand, for [-3.4], the greatest integer not exceeding -3.4 is -4.
However, having said all that, all numbers inside the [] sign are integers, so the greatest integer function has no effect.
The best way to solve this problem is to first remove the greatest integer function [], for example for (A),
0, -3, -4, -5, 8
We can see that -3, -4, -5 are all less than zero, so it is not ordered correctly from least to greatest.
If you draw a number line, you will find that they fall in this order:
-5...-4...-3...0...8
So use the number line and figure out the remainder, and you will eventually find one answer that is ordered from least to greatest.