11. a. A binary operation * is defined on the set of R of real numbers by

𝑎 ∗ 𝑏 = 𝑎2 − 2𝑎𝑏 + 𝑏2, where 𝑎, 𝑏 ∈ 𝑅. Find
i.
R, such that 2 ∗ 5 = √𝑟
ii.
(𝑥+1) ∗ 𝑥
𝑥
, 𝑥 ≠ 0 𝑠𝑖𝑚𝑝𝑙𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑓𝑎𝑟 𝑎𝑠 𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑒

i. We know that 2 ∗ 5 = 2^2 - 2(2)(5) + 5^2 = -6. Therefore, we need to find an r such that -6 is a perfect square. The only real number that satisfies this is r = 36. Thus, R = {36}.

ii. We have (x+1) ∗ x / x = [(x+1)^2 - 2(x+1)x + x^2] / x = [x^2 + 2x + 1 - 2x^2 - 2x + x^2] / x = (1 - x^2) / x. This expression is not defined for x = 0, but for any other value of x it is simplified as (1 - x)(1 + x) / x. Thus, the operation is defined for all x ≠ 0 in R.

i. To find the set R such that 2 * 5 = √r, we substitute the values of a and b into the given equation:

2 * 5 = 2^2 - 2(2)(5) + 5^2
= 4 - 20 + 25
= 9

So, 2 * 5 = √9.

Taking the square root of both sides, we get:

√(2 * 5) = √√9
√9 = √√9
3 = √3

Therefore, R = {3}.

ii. To find (x+1) * x / x, where x ≠ 0, we substitute the given expression into the equation:

(x+1) * x / x = (x+1)^2 - 2(x+1)x + x^2 / x
= x^2 + 2x + 1 - 2x^2 - 2x + x^2 / x
= -x + 1 / x

Therefore, (x+1) * x / x = -x + 1 / x, where x ≠ 0.