auto-saved βœ“

Deep Learning β€” Week 6 (Hinglish Notes)

Regularization: Bias–Variance, Train vs Test error, True error & model complexity, L2 regularization, Dataset augmentation, Parameter sharing/tying, Noise at input/output (label smoothing), Early stopping, Ensembles (bagging), Dropout, Loss landscape & implicit regularization  Β·  Prof. Mitesh Khapra ke slides (Module 6.1–6.11) + transcripts + PYQs + GA Week 6

Page kaise use karein Upar toolbar: Pen / Highlight / Line / Text / Eraser, colour aur size choose karo. Jo bhi likhoge woh browser me auto-save hota hai. Backup ke liye ⬇ Save file / ⬆ Load. πŸ™ˆ Practice dabao to answers blur; click karke dekho. Shortcuts: P pen Β· H highlight Β· L line Β· T text Β· E eraser Β· Esc read Β· Ctrl+Z / Ctrl+Y.

Contents

  1. Poori kahani ek nazar me
  2. Exam me kya aata hai
  3. Bias & Variance
  4. Train error vs Test error
  5. True error & model complexity
  6. L2 regularization
  7. Data augmentation
  8. Parameter sharing & tying
  9. Noise at inputs
  10. Noise at outputs (label smoothing)
  11. Early stopping
  12. Ensembles / Bagging
  13. Dropout
  14. Loss landscape & implicit regularization
  15. Formula sheet
  16. PYQs solved
  17. Graded Assignment (GA)
  18. My notes

1. Poori kahani ek nazar me

Week 4–5 me humne seekha loss ko minimize kaise karein (optimizers). Week 6 ka sawaal ulta hai: training loss ko zero kar diya, to kya model achha hai? Jawab: nahi!

Professor ka main argument Deep networks me billions of parameters, data sirf millions β†’ over-parameterized β†’ asaani se overfit (training data ratt lete hain). Isliye:
  1. Pehle samjho bias vs variance (simple vs complex model).
  2. Phir prove karo ki training error true error se kam (optimistic) dikhata hai, aur farak model complexity pe depend karta hai.
  3. To training me minimize karo: $\mathcal L_{train}(\theta)+\Omega(\theta)$, jahan $\Omega$ = complexity ka penalty β†’ yahi regularization hai.
  4. Phir 8 tareeke: L2, data augmentation, parameter sharing, input noise, output noise, early stopping, ensembles, dropout.

2. Exam me kya aata hai padh ke shuru karo

TopicPYQ me kahanType
"Kaun se regularization / overfitting kam karte hain?"Sep'24, Sep'25, ET Jan'26 (Γ—2)MSQ
Bias / variance numerically nikalna (models ki predictions se)Jan'25, Jan'26SA
Graphs: loss curves / error vs complexity / decision boundaries β†’ bias-varianceSep'24, T2'25, ET Jan'25MCQ/MSQ
L2: purpose, $w^*$ aur Ξ» nikalna, shrinkageJan'25, T2'25, Sep'25 + GAMCQ/SA
Dropout: test-time scaling numerical, training me kya hota haiSep'25, Jan'26, ET Jan'25MCQ/MSQ
Parameter tying (autoencoder) param countT2'25SA
Soft targets (label smoothing)Jan'25MCQ

3. Bias aur Variance Module 6.1 numerical aata hai

Setup: asli function $f(x)$ ek sinusoid hai (hume pata nahi). Do models try karte hain:

Training data se 500 points baar baar ($k$ times) alag sample karke alag alag model train karo, phir dekho:

Definitions $$\text{Bias}(\hat f(x)) = E[\hat f(x)] - f(x)$$ $$\text{Variance}(\hat f(x)) = E\big[(\hat f(x)-E[\hat f(x)])^2\big]$$ $E[\hat f(x)]$ = alag alag training samples pe trained models ka average prediction.
BiasVarianceNaam
Simple modelHighLowUnderfit
Complex modelLowHighOverfit
βœ” Numerical recipe (PYQ Jan'26) 3 models, $x=2$ pe predictions $1.1, 0.9, 1.0$; true $y=1$.
$E[\hat f]=\frac{1.1+0.9+1.0}{3}=1$ β†’ Bias $=1-1=\mathbf 0$.
Variance $=\frac{(0.1)^2+(-0.1)^2+0^2}{3}=\frac{0.02}{3}=\mathbf{0.0067}$.
Dhyan: variance me $1/k$ (population formula) use hota hai, $1/(k-1)$ nahi. Variance ke liye true $y$ ki zaroorat hi nahi.
⚠ Graph se pehchanna Straight-line decision boundary = high bias. Bahut wiggly boundary jo har point ke around mudti hai = low bias, high variance. Smooth curve = beech me. Bias ordering: line > smooth curve > wiggly.
πŸ“ Exam me aise poocha β€” abhi try karo
$x=2$ pe 3 models: 1.1, 0.9, 1.0; true $y=1$. Bias? Variance? Quiz-2 Jan'26
Bias 0; Variance $0.02/3=0.0067$.
True $y=[1,3,6,8]$; A $[1,4,7,8]$, B $[3,5,6,9]$, C $[2,3,5,7]$. Average variance? Quiz-2 Jan'25
Har point pe mean 2, 4, 6, 8; har point ka variance 2/3 β†’ 0.67.
Boundaries: A seedhi line, B bahut wiggly, C smooth curve. Bias order? Quiz-2 Sep'24
A > C > B.
$K_1\gg K_2$ layers, same width. Zyada complex kaun? GA Q1
$N_1$.

4. Train error vs Test error Module 6.2

Naya point $(x,y)$ jo training me nahi dekha, uska expected squared error:

Bias–variance decomposition $$E[(y-\hat f(x))^2] = \text{Bias}^2 + \text{Variance} + \sigma^2\ (\text{irreducible error})$$

Error vs complexity graph:

$$\text{train}_{err}=\frac1n\sum_{i=1}^{n}(y_i-\hat f(x_i))^2 \qquad \text{test}_{err}=\frac1m\sum_{i=n+1}^{n+m}(y_i-\hat f(x_i))^2$$

Complexity badhne pe train error over-optimistic ho jaata hai; validation error asli picture deta hai.

πŸ“ Exam me aise poocha β€” abhi try karo
$E[(y-\hat f)^2]=\text{Bias}^2+\text{Var}+\sigma^2$ ke hisaab se: train error zero ho sakta hai? Test error? GA Q2
Train error: haan (complex model). Test error: nahi ($\sigma^2$ irreducible).
Error vs complexity: U-shape curve kaunsa? Monotonic girta kaunsa? Far-left point? ET Jan'25
U = test, girta = train, far-left = high bias, low variance.
Loss curves: dono badh rahe (M2), dono high pe plateau (M4), val U-shape (M3), dono low (M1)? Quiz-2 T2'25
M2 β†’ LR too high, M4 β†’ high bias, M3 β†’ high variance, M1 β†’ low bias + low variance.
Train error 10%, test error 56% β€” kya karein? (Adam / ReLU / input noise / Maxout / L2) ET Jan'26
Overfit β†’ input noise, L2.

5. True error estimate & model complexity Module 6.2–6.3 derivation

Maano $y=f(x)+\varepsilon$, $\varepsilon\sim\mathcal N(0,\sigma^2)$. Hume chahiye true error $E[(\hat f(x)-f(x))^2]$, par $f$ pata nahi. To $\hat y-y$ (jo dikh raha hai) se relate karo:

$$E[(\hat y-y)^2]=E[(\hat f-f-\varepsilon)^2]=E[(\hat f-f)^2]-2E[\varepsilon(\hat f-f)]+E[\varepsilon^2]$$
Key equation $$\underbrace{E[(\hat f-f)^2]}_{\text{true error}}=\underbrace{\tfrac1m\textstyle\sum(\hat y_i-y_i)^2}_{\text{empirical error}}-\underbrace{\sigma^2}_{\text{small const}}+2\underbrace{E[\varepsilon(\hat f-f)]}_{\text{covariance}}$$

(Expectation empirically = average, jaise $k$ matches ke goals ka average.)

Case 1: Test dataCase 2: Training data
Test points $\hat f$ banane me use nahi hue β†’ $\varepsilon \perp (\hat f-f)$ β†’ covariance $=0$.
True error = test error + small constant. βœ” Isliye hamesha alag validation set use karo.
Training points se hi $\hat f$ bana β†’ $\varepsilon$ aur $\hat f-f$ independent nahi β†’ covariance $\ne0$.
Empirical train error true error se kam β€” sahi picture nahi deta.

Stein's Lemma se:

$$\frac{2\sigma^2}{n}\sum_{i=1}^n\frac{\partial\hat f(x_i)}{\partial y_i}$$

$\partial\hat f(x_i)/\partial y_i$ kab bada? Jab observation me chhota change β†’ prediction me bada change. Yeh complex model ki pehchaan hai (slide: ek point hilao, complex curve bahut hilti hai, simple line nahi).

Conclusion = regularization ka base $$\text{true error}=\text{train error}+\text{small const}+\Omega(\text{model complexity})$$ To train karte waqt minimize karo: $\ \min_\theta\ \mathcal L_{train}(\theta)+\Omega(\theta)=\tilde{\mathcal L}(\theta)$.
L1, L2, early stopping, input noise β€” sab isi form ke instances hain.
πŸ“ Exam me aise poocha β€” abhi try karo
S1: zero training loss β†’ unseen data pe guaranteed achha. S2: regularization se train loss badh sakta hai par generalization error ghat sakta hai. ET May'25
S1 false, S2 true. (Train error true error se optimistic hota hai!)

6. L2 regularization (weight decay) Module 6.4 most asked

L2 $$\tilde{\mathcal L}(w)=\mathcal L(w)+\frac{\alpha}{2}\|w\|^2 \qquad \nabla\tilde{\mathcal L}(w)=\nabla\mathcal L(w)+\alpha w$$ $$w_{t+1}=w_t-\eta\nabla\mathcal L(w_t)-\eta\alpha w_t$$ Code me bas ek chhota term add β€” har step weight ko thoda zero ki taraf kheencho ("weight decay").

Geometric meaning (eigen analysis)

$w^*$ = bina regularization ka optimum ($\nabla\mathcal L(w^*)=0$). Taylor 2nd order:

$$\mathcal L(w)=\mathcal L(w^*)+\tfrac12(w-w^*)^TH(w-w^*)\ \Rightarrow\ \nabla\mathcal L(w)=H(w-w^*)$$

Regularized optimum $\tilde w$: $H(\tilde w-w^*)+\alpha\tilde w=0$:

Regularized solution $$\tilde w=(H+\alpha I)^{-1}Hw^*=Q(\Lambda+\alpha I)^{-1}\Lambda Q^Tw^*$$ $H=Q\Lambda Q^T$ (symmetric PSD). Matlab: $w^*$ ko $Q^T$ se rotate β†’ har component $i$ ko $\dfrac{\lambda_i}{\lambda_i+\alpha}$ se scale β†’ $Q$ se wapas rotate.
βœ” SaarSaare weights shrink hote hain (norm hamesha $\|\tilde w\|<\|w^*\|$), lekin kam important directions (chhota eigenvalue) zyada shrink. Weight vector rotate bhi hota hai. Regularization minimum ko badal deta hai (jab tak $w^*\ne0$).
⚠ L1 vs L2 trap L2 = sum of squares $\lambda\sum w_i^2$, weights chhote karta hai (exactly zero nahi).
L1 = sum of absolute values $\lambda\sum|w_i|$, sparsity (kuch weights exactly 0).
Ξ» choose karna: jo validation loss minimum de (training loss nahi β€” woh to Ξ»=0 pe sabse kam hoga).
βœ” Quick numerical trick (PYQ T2'25) $L_D=w_1^2+w_2^2-2w_1-2w_2+2=(w_1-1)^2+(w_2-1)^2$ β†’ $w_D^*=(1,1)$.
L2 add karo $\frac\lambda2\|w\|^2$: $2w_1-2+\lambda w_1=0$ β†’ $w_1=\frac{2}{2+\lambda}$. To $w^*=(0.5,0.5)$ ke liye $\lambda=2$. Aur $\lambda>0$ pe $\|w^*\|$ hamesha $<\|w_D^*\|$.
πŸ“ Exam me aise poocha β€” abhi try karo
L2 ka primary purpose? Quiz-2 Jan'25
Weights ke squares ka sum loss me add karke overfitting rokna. (Absolute values / sparsity = L1.)
$L_D=w_1^2+w_2^2-2w_1-2w_2+2$, $+\frac\lambda2\|w\|^2$. $w_D^*$? $w^*=(0.5,0.5)$ ke liye Ξ»? Kabhi $\|w^*\|>\|w_D^*\|$? Quiz-2 T2'25
(1,1); Ξ» = 2; kabhi nahi (hamesha shrink).
Val loss Ξ» = 0.1 pe minimum, train loss Ξ» ghatne pe ghatta. Kya choose? Quiz-2 Sep'25
Ξ» = 0.1.
Training error already 0; L2 lagaya. Kya ho sakta hai? GA Q3
Train error badh sakta hai, test error ghat sakta hai, unimportant weights β†’ ~0.
$\mathcal L=0.5w^2+5b^2+1$: min value, grad sum, det H, sensitive param, regularization se min badlega? GA Q4–Q8
1; 0; 10; $b$ (eigenvalue 10); is function ke liye min $(0,0)$ hi rahega β†’ True.

7. Dataset augmentation Module 6.5

Idea: kuch transformations se label nahi badalta β€” "2" ko 20Β° ya 65Β° rotate karo, shift karo, blur karo, kuch pixels badlo β†’ abhi bhi "2". To in se naya training data bana lo (task ki knowledge use karke).

8. Parameter sharing & tying Module 6.6

Kam independent parameters β†’ kam complexity β†’ regularization.

βœ” PYQ: tied vs untied autoencoderInput $n$, hidden $0.3n$. Untied: $n\cdot0.3n+0.3n\cdot n=0.6n^2$. Tied: sirf $0.3n^2$. Ratio $n_1/n_2=\mathbf2$.
πŸ“ Exam me aise poocha β€” abhi try karo
Autoencoder, hidden = 0.3 Γ— input; untied vs tied ($W_2=W_1^T$) params ratio? Quiz-2 T2'25
$0.6n^2/0.3n^2=2$.

9. Input me noise Module 6.7

Input $\tilde x_i=x_i+\varepsilon_i$, $\varepsilon_i\sim\mathcal N(0,\sigma^2)$ (denoising autoencoder me dekha tha). Ise data augmentation bhi maan sakte ho. Simple linear network $\hat y=\sum w_ix_i$ ke liye:

$$\tilde y=\sum w_i\tilde x_i=\hat y+\sum w_i\varepsilon_i$$ $$E[(\tilde y-y)^2]=E[(\hat y-y)^2]+\underbrace{2E[(\hat y-y)\textstyle\sum w_i\varepsilon_i]}_{=0}+E[(\textstyle\sum w_i\varepsilon_i)^2]$$
Result $$E[(\tilde y-y)^2]=E[(\hat y-y)^2]+\sigma^2\sum_i w_i^2$$ Yaani Gaussian input noise = L2 regularization (weight decay), jahan $\alpha\leftrightarrow\sigma^2$. (Cross terms 0 kyunki $\varepsilon_i$ independent aur zero-mean.)
πŸ“ Exam me aise poocha β€” abhi try karo
Overfitting (train 10%, test 56%) β€” kya input noise help karega? ET Jan'26
Haan β€” input noise β‰ˆ L2 regularization.

10. Output me noise β€” label smoothing Module 6.8

Intuition: true labels pe pura bharosa mat karo, woh noisy ho sakte hain.

Hard target (one-hot)Soft target
$p=(0,0,1,0,\dots,0)$$p=\left(\tfrac{\varepsilon}{9},\tfrac{\varepsilon}{9},1-\varepsilon,\tfrac{\varepsilon}{9},\dots\right)$

10 classes: sahi class ko $1-\varepsilon$, baaki 9 me $\varepsilon$ barabar baant do ($\varepsilon/9$ each). Phir minimize $-\sum p_i\log q_i$. Model over-confident nahi hota.

⚠ PYQ definitionSoft target = "a target value that is continuous or probabilistic rather than discrete". (0/1 wala = hard target.)
πŸ“ Exam me aise poocha β€” abhi try karo
Output pe noise inject karne me "soft target" kya hai? Quiz-2 Jan'25
Continuous/probabilistic target (discrete 0/1 nahi).

11. Early stopping Module 6.9

AlgorithmValidation error track karo. Patience $p$: agar step $k$ pe pichhle $p$ steps me validation error improve nahi hua, to training roko aur step $k-p$ wala model return karo.
πŸ“ Exam me aise poocha β€” abhi try karo
Overfitting kaun rokta hai: Dropout, Early stopping, L2, MaxOut? ET Jan'26 MSQ
Dropout, Early stopping, L2.

12. Ensemble methods / Bagging Module 6.10

Alag models ke outputs combine karo (LR, SVM, Naive Bayes, ya same classifier with different hyperparameters / features / data samples).

Bagging: same classifier ke instances; dataset se sampling with replacement karke $T_1,\dots,T_k$ banao, $i$-th model ko $T_i$ pe train karo.

Bagging kab kaam karta hai? $k$ models, error $\varepsilon_i$, $E[\varepsilon_i^2]=V$, $E[\varepsilon_i\varepsilon_j]=C$: $$\text{mse}=E\Big[\Big(\tfrac1k\textstyle\sum\varepsilon_i\Big)^2\Big]=\frac1{k^2}\big(kV+k(k-1)C\big)=\frac{V}{k}+\frac{k-1}{k}C$$

13. Dropout Module 6.11 numerical aata hai

Problem: ensembles achhe hain lekin kai bade neural nets train karna mehenga, aur test time pe sabko combine karna real-time me infeasible. Dropout dono solve karta hai.

Dropout kya hai Training me har node ko probability $p$ se retain karo (typically hidden $p=0.5$, input/visible $p=0.8$); drop hua node apne saare in/out connections ke saath temporarily hata do β†’ thinned network.
$n$ nodes β†’ $2^n$ possible thinned networks. Trick: (1) saare networks weights share karte hain, (2) har training instance / mini-batch pe naya thinned network sample karo.
Test time $2^n$ networks aggregate karna impossible. Isliye poora network use karo, aur har node ka output (ya uske outgoing weights) ko uske training me "on" rehne ke fraction $p$ se scale karo: $w\to pw$.

Kyu kaam karta hai: masking noise β†’ hidden units co-adapt nahi kar paate (kisi doosre unit pe zyada depend nahi kar sakte, woh kabhi bhi drop ho sakta hai). Example: $h_i$ nose detect karke face pehchanta hai; $h_i$ drop hua to model ko redundantly nose encode karna ya doosre features (aankh) se face pehchanna seekhna padta hai β†’ robust.

βœ” PYQ Sep'25 type (test-time numerical) $x=[2,1]$; hidden ReLU: $h_{11}=2(1)+1(0.5)=2.5$, $h_{12}=2(-1)+1(1)=-1\to0$. Dropout 0.5 β†’ test pe scale: $[1.25,0]$. Output $\sigma(1\cdot1.25+1\cdot0)=\sigma(1.25)=\mathbf{0.777}$.
⚠ Common traps Dropout sparse weights nahi banata. Test time pe koi neuron drop nahi hota. Dropped set fixed nahi, har iteration random.
πŸ“ Exam me aise poocha β€” abhi try karo
$x=[2,1]$, ReLU hidden ($w$: 1, 0.5 / βˆ’1, 1), output weights 1, 1, sigmoid, dropout 0.5. Test output? Quiz-2 Sep'25
$h=[2.5,0]$ β†’ Γ—0.5 β†’ 1.25 β†’ $\sigma(1.25)=0.77$.
Dropout + mini-batch GD: sparsity? layer-wise alag $p$? Iteration $t$ pe drop hua neuron, $t+1$ pe bhi update nahi hoga? Quiz-2 Jan'26
Sirf "$p$ har layer ke liye alag ho sakta hai" sahi hai.
Dropout $p=0.5$: training me random/fixed set? Test pe drop ya scale? ET Jan'25
Training: har iteration random set. Test: koi drop nahi, 0.5 se scale.

14. Loss landscape, grouping, implicit regularization

Data $x_i$Architecture $f(\cdot)$Optimizer $\nabla$Cost penalty $\mathcal L(\cdot)$
Data augmentation, Noise injectionDropout, Skip connectionsSGD, Early stoppingL1, L2, Weight sharing, Pooling

Explicit: L1, L2, data augmentation, noise injection, skip connections.   Implicit: SGD, large/small initial LR, early stopping, weight sharing, pooling, dropout.

Random labelling experiment (Zhang et al.) Training labels ko random shuffle karo (input–output relation khatam). Chhota model (19 params) aise data ko fit nahi kar paata β€” training error high (expected). Lekin deep nets random labels pe bhi training error zero kar dete hain! Conclusion: regularizers tuned hon to generalization improve karte hain, lekin woh generalization ki fundamental wajah nahi lagte (sab regularizers hata do tab bhi nets achha perform karte hain). Over-parameterized models SGD/GD ke discrete steps se khud implicitly regularize hote hain.
πŸ“ Exam me aise poocha β€” abhi try karo
Regularization kaun: data augmentation, activation functions, L2, input/output noise, optimization techniques? Quiz-2 Sep'24 MSQ
Data augmentation, L2, noise. (Activations aur optimizers nahi.)
Overfitting kaun kam karta hai: data aug, dropout, L2, Adam instead of SGD? Quiz-2 Sep'25 MSQ
Pehle teen.

15. One-page formula sheet

Bias$E[\hat f(x)]-f(x)$
Variance$E[(\hat f(x)-E[\hat f(x)])^2]$  (1/k)
Decomposition$E[(y-\hat f)^2]=\text{Bias}^2+\text{Var}+\sigma^2$
True errortest err $-\sigma^2$ (+0)  |  train err $-\sigma^2+\frac{2\sigma^2}{n}\sum\frac{\partial\hat f(x_i)}{\partial y_i}$
L2 loss / update$\mathcal L+\frac\alpha2\|w\|^2$;   $w\leftarrow w-\eta\nabla\mathcal L-\eta\alpha w$
L2 solution$\tilde w=Q(\Lambda+\alpha I)^{-1}\Lambda Q^Tw^*$, scale $\frac{\lambda_i}{\lambda_i+\alpha}$
Input noise$E[(\tilde y-y)^2]=E[(\hat y-y)^2]+\sigma^2\sum w_i^2$
Label smoothingtrue class $1-\varepsilon$, others $\varepsilon/(K-1)$
Early stopping ≑ L2$(I-\eta\Lambda)^t=(\Lambda+\alpha I)^{-1}\alpha$
Bagging mse$\frac Vk+\frac{k-1}{k}C$
Dropout$2^n$ thinned nets; test: weights $\times p$

16. PYQs β€” fully solved

Yeh sab sawaal upar theory sections ke πŸ“ boxes me already aa chuke hain. Yahan poore options ke saath. Official answer keys. πŸ™ˆ Practice se blur karke khud try karo.

A. Kaun se regularizers hain? (MSQ)

Quiz-2 Sep'24MSQ Β· 3 marks
Which of the following are regularization technique(s)?
  1. Dataset augmentation
  2. Activation functions
  3. L2 regularization
  4. Adding noise to the inputs or outputs
  5. Optimization techniques
  6. All of these
Answer: (a), (c), (d). Activation functions (sigmoid, ReLU) aur optimizers (Adam, RMSProp) regularizers nahi.
Quiz-2 Sep'25MSQ Β· 3 marks
Which techniques can be used to reduce model overfitting?
  1. Data augmentation
  2. Dropout
  3. L2 regularization
  4. Using Adam instead of SGD
Answer: (a), (b), (c).
ET Jan'26 ANMSQ Β· 2 marks
Which techniques prevent the model from overfitting?
  1. Dropout
  2. Early stopping
  3. L2 regularization
  4. MaxOut
Answer: (a), (b), (c). MaxOut ek activation function hai.
ET Jan'26 ANMSQ Β· 2 marks
Training error 10%, test error 56%. Which methods can reduce the test error?
  1. Adam
  2. ReLU activation
  3. Injecting noise at input
  4. Maxout
  5. L2 regularization
Answer: (c), (e). Bada gap = overfitting (high variance) β†’ regularization chahiye.

B. Bias / Variance

Quiz-2 Sep'24MCQ Β· 3 marks
Decision boundaries: Model A = straight line, Model B = bahut wiggly (har point ke around), Model C = smooth moderate curve. Bias ka correct order?
  1. A > B > C
  2. B > C > A
  3. C > B > A
  4. A > C > B
Answer: (d) A > C > B. Simple = high bias.
Quiz-2 Jan'25SA Β· 3 marks important
Same architecture, 3 subsets pe trained models A, B, C. True $y=[1,3,6,8]$. Predictions: A $[1,4,7,8]$, B $[3,5,6,9]$, C $[2,3,5,7]$. Variance averaged over all test points?
Har point pe mean: $x_1:\frac{1+3+2}3=2$, $x_2:4$, $x_3:6$, $x_4:8$.
Variance per point: $x_1:\frac{1+1+0}{3}=\frac23$; $x_2:\frac{0+1+1}3=\frac23$; $x_3:\frac{1+0+1}3=\frac23$; $x_4:\frac{0+1+1}3=\frac23$.
Average $=\mathbf{0.67}$ (range 0.62–0.72). True $y$ variance me use nahi hota (woh bias ke liye hai).
Quiz-2 Jan'26Comprehension Q9–10
Model $y=w_0+w_1x$, 3 baar 5-point samples pe fit. $x=2$ pe $\hat f_1=1.1,\hat f_2=0.9,\hat f_3=1.0$, true $y=1$. (i) Bias? (ii) Variance (4 decimals)?
(i) $E[\hat f]=1$ β†’ Bias $=\mathbf0$.   (ii) $\frac{0.01+0.01+0}{3}=\mathbf{0.0067}$ (range 0.0064–0.0070).
Quiz-2 2025 T2MCQ Β· 3 marks tricky
4 loss curves (solid = train, dashed = val), 50 epochs, final model rakha:
$M_1$: dono low aur paas paas. $M_2$: dono badhte ja rahe. $M_3$: train girta, val U-shape (bada gap). $M_4$: dono girke high level pe plateau, chhota gap.
(A) LR too high (B) low bias high var (C) low bias low var (D) high bias low var
  1. M1β†’C, M2β†’A, M3β†’B, M4β†’D
  2. M1β†’D, M2β†’A, M3β†’C, M4β†’B
  3. M1β†’A, M2β†’B, M3β†’C, M4β†’D
  4. M1β†’D, M2β†’C, M3β†’B, M4β†’A
Answer: (a). Loss badh raha hai (train bhi) = LR too high, diverge. Dono high pe atke = underfit = high bias. ⚠ PDF me handwritten note M2/M4 ulta likha hai β€” key follow karo.
ET Jan'25MSQ Β· 3 marks
Error vs complexity plot: curve (1) U-shape, curve (2) monotonically girta (1 ke neeche), (3) = bahut low complexity pe dashed line.
  1. (1) test error
  2. (2) training error
  3. (3) high bias, low variance
  4. (1) training error
  5. (2) test error
  6. (3) low bias, high variance
Answer: (a), (b), (c). (3) = underfit.
ET May'25 ANMCQ Β· 2 marks
S1: Zero training loss wala model unseen data pe achha perform karega hi. S2: Regularization training loss badha sakta hai lekin generalization error kam.
  1. Both true
  2. S1 true, S2 false
  3. None of these
  4. S1 false, S2 true
Answer: (d).

C. L2

Quiz-2 Jan'25MCQ Β· 3 marks
Primary purpose of L2 regularization?
  1. Prevent overfitting by adding sum of absolute values of weights
  2. Encourage sparsity by setting some weights to zero
  3. Prevent overfitting by adding sum of squared values of weights to the loss
  4. Increase complexity by penalizing small weights
Answer: (c). (a), (b) = L1.
Quiz-2 2025 T2Comprehension Q240–242
$L_D(w)=w_1^2+w_2^2-2w_1-2w_2+2$, $L_R=\frac\lambda2\|w\|^2$, $L=L_D+L_R$.
(i) $w_D^*$? (ii) $w^*=(0.5,0.5)$ ho to Ξ»? (iii) Kya kisi Ξ»>0 ke liye $\|w^*\|>\|w_D^*\|$?
(i) $2w_i-2=0$ β†’ $\mathbf{(1,1)}$.
(ii) $2w_i-2+\lambda w_i=0$ β†’ $w_i=\frac{2}{2+\lambda}=0.5$ β†’ $\lambda=\mathbf2$.
(iii) No, $w^*=\frac{2}{2+\lambda}(1,1)$ hamesha origin ki taraf shrink.
Quiz-2 Sep'25MCQ Β· 3 marks
L2, λ ∈ {0.001, 0.01, 0.1, 1}. Training loss λ ghatne pe ghatta hai, validation loss λ = 0.1 pe minimum. Kya choose karein?
Answer: Ξ» = 0.1 β€” lowest validation loss, best bias–variance trade-off.

D. Dropout, label smoothing, tying

Quiz-2 Sep'25MCQ Β· 4 marks
2 inputs β†’ 2 ReLU hidden β†’ 1 sigmoid output. $w_{111}=1,w_{112}=0.5,w_{121}=-1,w_{122}=1,w_{211}=1,w_{212}=1$, no bias, dropout 0.5 on hidden. $x=[2,1]$. Expected output while testing?
  1. 0.77
  2. 1.25
  3. 2.5
  4. 0.92
Answer: (a) 0.77. $h=\text{ReLU}([2.5,-1])=[2.5,0]$ β†’ Γ—0.5 β†’ $[1.25,0]$ β†’ $\sigma(1.25)=0.777$. (Bina scaling: $\sigma(2.5)=0.92$ β€” trap option.)
Quiz-2 Jan'26MCQ Β· 2 marks
Feed-forward net me dropout + mini-batch GD. Correct statement?
  1. This will lead to sparsity in the trained weights
  2. The dropout probability $p$ can be different for each hidden layer
  3. Weights of neurons dropped at iteration $t$ will not get updated during iteration $t+1$
Answer: (b). (c) galat β€” har iteration naya mask.
ET Jan'25MSQ Β· 2 marks
Hidden layer pe dropout $p=0.5$. True?
  1. Training me random set drop; har iteration dynamically decide
  2. Inference me koi drop nahi; har activation 0.5 se scale
  3. Training me fixed set drop, pehle se decided
  4. Inference me random set drop
Answer: (a), (b).
Quiz-2 Jan'25MCQ Β· 3 marks
What is the soft target used when injecting noise at the output?
  1. Label that is either 0 or 1
  2. A target value that is continuous or probabilistic rather than discrete
  3. Random label for misclassified samples
  4. Label always equal to predicted output
Answer: (b).
Quiz-2 2025 T2SA Β· 2 marks
Autoencoder: hidden = 30% of input, no biases. Network-1 untied $W_1,W_2$; Network-2 tied $W_2=W_1^T$. $n_1/n_2$?
Answer: 2. $0.6n^2/0.3n^2$.
NoteWeek 6 PYQ PDF me ek CNN comprehension (Q237–239: 64Γ—64Γ—8 input, 16 kernels 5Γ—5, stride 1, pad 2 β†’ output 64Γ—64Γ—16, params 3200, input 128Γ—128Γ—8 karne pe params same) bhi hai. Yeh Week 8 ka topic hai, isliye yahan detail nahi.

17. Graded Assignment Week 6 (GA)

⚠ GA ka official key PDF me nahi tha β€” yeh answers maine khud solve kiye hain. Submit karne se pehle khud bhi verify kar lena.

GA Q1: $N_1$ me $K_1$ hidden layers, $N_2$ me $K_2$, har layer $n$ neurons, $K_1\gg K_2$. Higher complexity?
Answer: $N_1$ β€” zyada layers = zyada parameters aur non-linearities.
GA Q2: $E[(y-\hat f(x))^2]$ (slide 13) ke according true statements?
  1. Training error sufficiently complex model se zero ho sakta hai
  2. Training error irreducible error ki wajah se zero nahi ho sakta
  3. Test error zero ho sakta hai
  4. Test error irreducible error ki wajah se zero nahi ho sakta
Answer: (a), (d). Complex model training points (noise samet) ratt leta hai; test pe $\sigma^2$ bachega.
GA Q3: Model ka training error zero hai. L2 lagao to?
  1. Training error badh sakta hai
  2. Test error ghat sakta hai
  3. Kam important weights ko zero ke paas laakar complexity kam
  4. Training error ghat sakta hai
Answer: (a), (b), (c). Training error already 0, aur kam nahi ho sakta.
GA Q4–Q8: $\mathcal L(w,b)=0.5w^2+5b^2+1$
Q4: $\mathcal L(w^*,b^*)$?   Q5: $\nabla\mathcal L(w^*,b^*)$ ke elements ka sum?   Q6: $\det H$?   Q7: Loss kis param ke prati zyada sensitive?   Q8: "Regularization unregularized minimum $(w^*,b^*)$ ko nahi badalta" β€” True/False?
Minimum $(0,0)$.
Q4: 1.   Q5: 0 (minimum pe gradient zero).   Q6: $H=\begin{bmatrix}1&0\\0&10\end{bmatrix}$ β†’ $\det=\mathbf{10}$.
Q7: Eigenvalues 1 (direction $[1,0]$ = $w$) aur 10 (direction $[0,1]$ = $b$) β†’ bada eigenvalue = zyada curvature β†’ $b$.
Q8: True. EVD: $H=\text{diag}(1,10)$ β†’ $\tilde w=Q(\Lambda+\alpha I)^{-1}\Lambda Q^Tw^*$. Yahan $w^*=(0,0)$, to $\tilde w=QDQ^T\cdot\mathbf 0=(0,0)$ β†’ minimum nahi badla. (General me L2 minimum ko origin ki taraf shift karta hai; yahan minimum already origin pe hai, isliye koi change nahi.)

18. My notes (khali pages)

Free writing space. Use βž• Page in the toolbar to add more ruled pages.