원의 중심을 O라 하면
A P ⃗ = A O ⃗ + O P ⃗ \mathrm{\vec{AP}} = \vec{AO} + \vec{OP} AP = A O + O P 이므로
A P ⃗ ⋅ A Q ⃗ = ( A O ⃗ + O P ⃗ ) ⋅ A Q ⃗ \mathrm{\vec{AP}} \cdot \vec{AQ} = ( \vec{AO} + \vec{OP} ) \cdot \vec{AQ} AP ⋅ A Q = ( A O + O P ) ⋅ A Q = A O ⃗ ⋅ A Q ⃗ + O P ⃗ ⋅ A Q ⃗ = \mathrm{\vec{AO}} \cdot \vec{AQ} + \vec{OP} \cdot \vec{AQ} = AO ⋅ A Q + O P ⋅ A Q
여기서 A O ⃗ ⋅ A C ⃗ ≤ A O ⃗ ⋅ A Q ⃗ ≤ A O ⃗ ⋅ A B ⃗ \mathrm{\vec{AO}} \cdot \vec{AC} \leq \vec{AO} \cdot \vec{AQ} \leq \vec{AO} \cdot \vec{AB} AO ⋅ A C ≤ A O ⋅ A Q ≤ A O ⋅ A B
그림에서 B M ‾ = 1 3 \displaystyle \mathrm{\overline{BM}} = \frac{1}{\sqrt{3}} BM = 3 1 , O D ‾ = 3 + 1 3 \displaystyle \mathrm{\overline{OD}} = \frac{\sqrt{3} + 1}{\sqrt{3}} OD = 3 3 + 1 ,
A D ‾ = 3 − 1 \displaystyle \mathrm{\overline{AD}} = \sqrt{3} - 1 AD = 3 − 1 이므로
A O ⃗ = ( − 3 + 1 3 , 1 − 3 ) \displaystyle \mathrm{\vec{AO}} = \left( - \frac{\sqrt{3} + 1}{\sqrt{3}} , 1 - \sqrt{3} \right) AO = ( − 3 3 + 1 , 1 − 3 )
A B ⃗ = ( − 1 , − 3 ) \displaystyle \mathrm{\vec{AB}} = \left( - 1 , - \sqrt{3} \right) AB = ( − 1 , − 3 ) , A C ⃗ = ( 1 , − 3 ) \displaystyle \mathrm{\vec{AC}} = \left( 1 , - \sqrt{3} \right) AC = ( 1 , − 3 )
A O ⃗ ⋅ A C ⃗ = 2 − 4 3 3 \displaystyle \mathrm{\vec{AO}} \cdot \vec{AC} = 2 - \frac{4}{3} \sqrt{3} AO ⋅ A C = 2 − 3 4 3 , A O ⃗ ⋅ A B ⃗ = 4 − 2 3 3 \displaystyle \mathrm{\vec{AO}} \cdot \vec{AB} = 4 - \frac{2}{3} \sqrt{3} AO ⋅ A B = 4 − 3 2 3
또, O P ⃗ ⋅ A Q ⃗ = ∣ A Q ⃗ ∣ cos θ \mathrm{\vec{OP}} \cdot \vec{AQ} = \left| \vec{AQ} \right| \cos \theta OP ⋅ A Q = A Q cos θ 이므로 − 2 ≤ O P ⃗ ⋅ A Q ⃗ ≤ 2 - 2 \leq \mathrm{\vec{OP}} \cdot \vec{AQ} \leq 2 − 2 ≤ OP ⋅ A Q ≤ 2
따라서 최댓값은 ( 4 − 2 3 3 ) + 2 = 6 − 2 3 3 \displaystyle \left( 4 - \frac{2}{3} \sqrt{3} \right) + 2 = 6 - \frac{2}{3} \sqrt{3} ( 4 − 3 2 3 ) + 2 = 6 − 3 2 3
최솟값은 ( 2 − 4 3 3 ) − 2 = − 4 3 3 \displaystyle \left( 2 - \frac{4}{3} \sqrt{3} \right) - 2 = - \frac{4}{3} \sqrt{3} ( 2 − 3 4 3 ) − 2 = − 3 4 3
이므로 최댓값과 최솟값의 합은 a + b 3 = \displaystyle a + b \sqrt{3} = a + b 3 = 6 − 2 3 \displaystyle 6 - 2 \sqrt{3} 6 − 2 3
∴ \therefore ∴ a 2 + b 2 = 36 + 4 = 40 a ^{2} + b ^{2} = 36 + 4 = 40 a 2 + b 2 = 36 + 4 = 40