Deep Learning β Week 6 (Hinglish Notes)
Regularization: BiasβVariance, Train vs Test error, True error & model complexity, L2 regularization, Dataset augmentation, Parameter sharing/tying, Noise at input/output (label smoothing), Early stopping, Ensembles (bagging), Dropout, Loss landscape & implicit regularization Β· Prof. Mitesh Khapra ke slides (Module 6.1β6.11) + transcripts + PYQs + GA Week 6
Page kaise use karein
Upar toolbar: Pen / Highlight / Line / Text / Eraser, colour aur size choose karo. Jo bhi likhoge woh browser me auto-save hota hai. Backup ke liye β¬ Save file / β¬ Load. π Practice dabao to answers blur; click karke dekho. Shortcuts: P pen Β· H highlight Β· L line Β· T text Β· E eraser Β· Esc read Β· Ctrl+Z / Ctrl+Y.
Contents
1. Poori kahani ek nazar me
Week 4β5 me humne seekha loss ko minimize kaise karein (optimizers). Week 6 ka sawaal ulta hai: training loss ko zero kar diya, to kya model achha hai? Jawab: nahi!
Professor ka main argument
Deep networks me
billions of parameters, data sirf
millions β
over-parameterized β asaani se
overfit (training data ratt lete hain). Isliye:
- Pehle samjho bias vs variance (simple vs complex model).
- Phir prove karo ki training error true error se kam (optimistic) dikhata hai, aur farak model complexity pe depend karta hai.
- To training me minimize karo: $\mathcal L_{train}(\theta)+\Omega(\theta)$, jahan $\Omega$ = complexity ka penalty β yahi regularization hai.
- Phir 8 tareeke: L2, data augmentation, parameter sharing, input noise, output noise, early stopping, ensembles, dropout.
2. Exam me kya aata hai padh ke shuru karo
| Topic | PYQ me kahan | Type |
| "Kaun se regularization / overfitting kam karte hain?" | Sep'24, Sep'25, ET Jan'26 (Γ2) | MSQ |
| Bias / variance numerically nikalna (models ki predictions se) | Jan'25, Jan'26 | SA |
| Graphs: loss curves / error vs complexity / decision boundaries β bias-variance | Sep'24, T2'25, ET Jan'25 | MCQ/MSQ |
| L2: purpose, $w^*$ aur Ξ» nikalna, shrinkage | Jan'25, T2'25, Sep'25 + GA | MCQ/SA |
| Dropout: test-time scaling numerical, training me kya hota hai | Sep'25, Jan'26, ET Jan'25 | MCQ/MSQ |
| Parameter tying (autoencoder) param count | T2'25 | SA |
| Soft targets (label smoothing) | Jan'25 | MCQ |
3. Bias aur Variance Module 6.1 numerical aata hai
Setup: asli function $f(x)$ ek sinusoid hai (hume pata nahi). Do models try karte hain:
- Simple (degree 1): $\hat f(x)=w_1x+w_0$
- Complex (degree 25): $\hat f(x)=\sum_{i=1}^{25}w_ix^i+w_0$
Training data se 500 points baar baar ($k$ times) alag sample karke alag alag model train karo, phir dekho:
- Simple models: ek doosre jaise dikhte hain (variance kam), lekin sab sinusoid se bahut door (underfitting β bias zyada).
- Complex models: ek doosre se bahut alag (variance zyada), lekin average sinusoid ke paas (bias kam).
Definitions
$$\text{Bias}(\hat f(x)) = E[\hat f(x)] - f(x)$$
$$\text{Variance}(\hat f(x)) = E\big[(\hat f(x)-E[\hat f(x)])^2\big]$$
$E[\hat f(x)]$ = alag alag training samples pe trained models ka average prediction.
| Bias | Variance | Naam |
| Simple model | High | Low | Underfit |
| Complex model | Low | High | Overfit |
β Numerical recipe (PYQ Jan'26)
3 models, $x=2$ pe predictions $1.1, 0.9, 1.0$; true $y=1$.
$E[\hat f]=\frac{1.1+0.9+1.0}{3}=1$ β Bias $=1-1=\mathbf 0$.
Variance $=\frac{(0.1)^2+(-0.1)^2+0^2}{3}=\frac{0.02}{3}=\mathbf{0.0067}$.
Dhyan: variance me $1/k$ (population formula) use hota hai, $1/(k-1)$ nahi. Variance ke liye true $y$ ki zaroorat hi nahi.
β Graph se pehchanna
Straight-line decision boundary = high bias. Bahut wiggly boundary jo har point ke around mudti hai = low bias, high variance. Smooth curve = beech me. Bias ordering: line > smooth curve > wiggly.
π Exam me aise poocha β abhi try karo
$x=2$ pe 3 models: 1.1, 0.9, 1.0; true $y=1$. Bias? Variance? Quiz-2 Jan'26
Bias 0; Variance $0.02/3=0.0067$.
True $y=[1,3,6,8]$; A $[1,4,7,8]$, B $[3,5,6,9]$, C $[2,3,5,7]$. Average variance? Quiz-2 Jan'25
Har point pe mean 2, 4, 6, 8; har point ka variance 2/3 β 0.67.
Boundaries: A seedhi line, B bahut wiggly, C smooth curve. Bias order? Quiz-2 Sep'24
A > C > B.
$K_1\gg K_2$ layers, same width. Zyada complex kaun? GA Q1
$N_1$.
4. Train error vs Test error Module 6.2
Naya point $(x,y)$ jo training me nahi dekha, uska expected squared error:
Biasβvariance decomposition
$$E[(y-\hat f(x))^2] = \text{Bias}^2 + \text{Variance} + \sigma^2\ (\text{irreducible error})$$
- $\sigma^2$ = data ka noise. Koi bhi model isko hata nahi sakta β test error kabhi zero nahi ho sakta.
- Training error complex model se zero ho sakta hai (model training points ratt leta hai β noise bhi).
Error vs complexity graph:
- Training error: complexity badhne pe lagatar girta hai.
- Test error: U-shape β pehle girta (bias kam), phir badhta (variance badhta).
- Left side (low complexity): dono high β high bias, low variance. Right side: train low, test high β overfit. Beech me sweet spot.
$$\text{train}_{err}=\frac1n\sum_{i=1}^{n}(y_i-\hat f(x_i))^2 \qquad \text{test}_{err}=\frac1m\sum_{i=n+1}^{n+m}(y_i-\hat f(x_i))^2$$
Complexity badhne pe train error over-optimistic ho jaata hai; validation error asli picture deta hai.
π Exam me aise poocha β abhi try karo
$E[(y-\hat f)^2]=\text{Bias}^2+\text{Var}+\sigma^2$ ke hisaab se: train error zero ho sakta hai? Test error? GA Q2
Train error: haan (complex model). Test error: nahi ($\sigma^2$ irreducible).
Error vs complexity: U-shape curve kaunsa? Monotonic girta kaunsa? Far-left point? ET Jan'25
U = test, girta = train, far-left = high bias, low variance.
Loss curves: dono badh rahe (M2), dono high pe plateau (M4), val U-shape (M3), dono low (M1)? Quiz-2 T2'25
M2 β LR too high, M4 β high bias, M3 β high variance, M1 β low bias + low variance.
Train error 10%, test error 56% β kya karein? (Adam / ReLU / input noise / Maxout / L2) ET Jan'26
Overfit β input noise, L2.
5. True error estimate & model complexity Module 6.2β6.3 derivation
Maano $y=f(x)+\varepsilon$, $\varepsilon\sim\mathcal N(0,\sigma^2)$. Hume chahiye true error $E[(\hat f(x)-f(x))^2]$, par $f$ pata nahi. To $\hat y-y$ (jo dikh raha hai) se relate karo:
$$E[(\hat y-y)^2]=E[(\hat f-f-\varepsilon)^2]=E[(\hat f-f)^2]-2E[\varepsilon(\hat f-f)]+E[\varepsilon^2]$$
Key equation
$$\underbrace{E[(\hat f-f)^2]}_{\text{true error}}=\underbrace{\tfrac1m\textstyle\sum(\hat y_i-y_i)^2}_{\text{empirical error}}-\underbrace{\sigma^2}_{\text{small const}}+2\underbrace{E[\varepsilon(\hat f-f)]}_{\text{covariance}}$$
(Expectation empirically = average, jaise $k$ matches ke goals ka average.)
| Case 1: Test data | Case 2: Training data |
Test points $\hat f$ banane me use nahi hue β $\varepsilon \perp (\hat f-f)$ β covariance $=0$. True error = test error + small constant. β Isliye hamesha alag validation set use karo. | Training points se hi $\hat f$ bana β $\varepsilon$ aur $\hat f-f$ independent nahi β covariance $\ne0$. Empirical train error true error se kam β sahi picture nahi deta. |
Stein's Lemma se:
$$\frac{2\sigma^2}{n}\sum_{i=1}^n\frac{\partial\hat f(x_i)}{\partial y_i}$$
$\partial\hat f(x_i)/\partial y_i$ kab bada? Jab observation me chhota change β prediction me bada change. Yeh complex model ki pehchaan hai (slide: ek point hilao, complex curve bahut hilti hai, simple line nahi).
Conclusion = regularization ka base
$$\text{true error}=\text{train error}+\text{small const}+\Omega(\text{model complexity})$$
To train karte waqt minimize karo: $\ \min_\theta\ \mathcal L_{train}(\theta)+\Omega(\theta)=\tilde{\mathcal L}(\theta)$.
L1, L2, early stopping, input noise β sab isi form ke instances hain.
π Exam me aise poocha β abhi try karo
S1: zero training loss β unseen data pe guaranteed achha. S2: regularization se train loss badh sakta hai par generalization error ghat sakta hai. ET May'25
S1 false, S2 true. (Train error true error se optimistic hota hai!)
6. L2 regularization (weight decay) Module 6.4 most asked
L2
$$\tilde{\mathcal L}(w)=\mathcal L(w)+\frac{\alpha}{2}\|w\|^2 \qquad \nabla\tilde{\mathcal L}(w)=\nabla\mathcal L(w)+\alpha w$$
$$w_{t+1}=w_t-\eta\nabla\mathcal L(w_t)-\eta\alpha w_t$$
Code me bas ek chhota term add β har step weight ko thoda zero ki taraf kheencho ("weight decay").
Geometric meaning (eigen analysis)
$w^*$ = bina regularization ka optimum ($\nabla\mathcal L(w^*)=0$). Taylor 2nd order:
$$\mathcal L(w)=\mathcal L(w^*)+\tfrac12(w-w^*)^TH(w-w^*)\ \Rightarrow\ \nabla\mathcal L(w)=H(w-w^*)$$
Regularized optimum $\tilde w$: $H(\tilde w-w^*)+\alpha\tilde w=0$:
Regularized solution
$$\tilde w=(H+\alpha I)^{-1}Hw^*=Q(\Lambda+\alpha I)^{-1}\Lambda Q^Tw^*$$
$H=Q\Lambda Q^T$ (symmetric PSD). Matlab: $w^*$ ko $Q^T$ se rotate β har component $i$ ko $\dfrac{\lambda_i}{\lambda_i+\alpha}$ se scale β $Q$ se wapas rotate.
- $\lambda_i\gg\alpha$ β factor $\approx1$ β woh direction bachi rehti (important direction).
- $\lambda_i\ll\alpha$ β factor $\approx0$ β woh direction khatam.
- $\alpha\to0$ β $\tilde w\to w^*$.
- Effective parameters $=\sum_i\frac{\lambda_i}{\lambda_i+\alpha}<n$.
β SaarSaare weights shrink hote hain (norm hamesha $\|\tilde w\|<\|w^*\|$), lekin kam important directions (chhota eigenvalue) zyada shrink. Weight vector rotate bhi hota hai. Regularization minimum ko badal deta hai (jab tak $w^*\ne0$).
β L1 vs L2 trap
L2 = sum of squares $\lambda\sum w_i^2$, weights chhote karta hai (exactly zero nahi).
L1 = sum of absolute values $\lambda\sum|w_i|$, sparsity (kuch weights exactly 0).
Ξ» choose karna: jo validation loss minimum de (training loss nahi β woh to Ξ»=0 pe sabse kam hoga).
β Quick numerical trick (PYQ T2'25)
$L_D=w_1^2+w_2^2-2w_1-2w_2+2=(w_1-1)^2+(w_2-1)^2$ β $w_D^*=(1,1)$.
L2 add karo $\frac\lambda2\|w\|^2$: $2w_1-2+\lambda w_1=0$ β $w_1=\frac{2}{2+\lambda}$. To $w^*=(0.5,0.5)$ ke liye $\lambda=2$. Aur $\lambda>0$ pe $\|w^*\|$ hamesha $<\|w_D^*\|$.
π Exam me aise poocha β abhi try karo
L2 ka primary purpose? Quiz-2 Jan'25
Weights ke squares ka sum loss me add karke overfitting rokna. (Absolute values / sparsity = L1.)
$L_D=w_1^2+w_2^2-2w_1-2w_2+2$, $+\frac\lambda2\|w\|^2$. $w_D^*$? $w^*=(0.5,0.5)$ ke liye Ξ»? Kabhi $\|w^*\|>\|w_D^*\|$? Quiz-2 T2'25
(1,1); Ξ» = 2; kabhi nahi (hamesha shrink).
Val loss Ξ» = 0.1 pe minimum, train loss Ξ» ghatne pe ghatta. Kya choose? Quiz-2 Sep'25
Ξ» = 0.1.
Training error already 0; L2 lagaya. Kya ho sakta hai? GA Q3
Train error badh sakta hai, test error ghat sakta hai, unimportant weights β ~0.
$\mathcal L=0.5w^2+5b^2+1$: min value, grad sum, det H, sensitive param, regularization se min badlega? GA Q4βQ8
1; 0; 10; $b$ (eigenvalue 10); is function ke liye min $(0,0)$ hi rahega β True.
7. Dataset augmentation Module 6.5
Idea: kuch transformations se label nahi badalta β "2" ko 20Β° ya 65Β° rotate karo, shift karo, blur karo, kuch pixels badlo β abhi bhi "2". To in se naya training data bana lo (task ki knowledge use karke).
- More data = better learning. Image classification / object recognition me bahut achha, speech me bhi.
- Kuch tasks me clear nahi hota ki aisa data kaise banayein.
8. Parameter sharing & tying Module 6.6
- Sharing: CNN me β same filter image ki alag positions pe; same weight matrix alag inputs pe.
- Tying: autoencoder me encoder aur decoder weights tied: $W_2=W_1^T$.
Kam independent parameters β kam complexity β regularization.
β PYQ: tied vs untied autoencoderInput $n$, hidden $0.3n$. Untied: $n\cdot0.3n+0.3n\cdot n=0.6n^2$. Tied: sirf $0.3n^2$. Ratio $n_1/n_2=\mathbf2$.
π Exam me aise poocha β abhi try karo
Autoencoder, hidden = 0.3 Γ input; untied vs tied ($W_2=W_1^T$) params ratio? Quiz-2 T2'25
$0.6n^2/0.3n^2=2$.
9. Input me noise Module 6.7
Input $\tilde x_i=x_i+\varepsilon_i$, $\varepsilon_i\sim\mathcal N(0,\sigma^2)$ (denoising autoencoder me dekha tha). Ise data augmentation bhi maan sakte ho. Simple linear network $\hat y=\sum w_ix_i$ ke liye:
$$\tilde y=\sum w_i\tilde x_i=\hat y+\sum w_i\varepsilon_i$$
$$E[(\tilde y-y)^2]=E[(\hat y-y)^2]+\underbrace{2E[(\hat y-y)\textstyle\sum w_i\varepsilon_i]}_{=0}+E[(\textstyle\sum w_i\varepsilon_i)^2]$$
Result
$$E[(\tilde y-y)^2]=E[(\hat y-y)^2]+\sigma^2\sum_i w_i^2$$
Yaani Gaussian input noise = L2 regularization (weight decay), jahan $\alpha\leftrightarrow\sigma^2$. (Cross terms 0 kyunki $\varepsilon_i$ independent aur zero-mean.)
π Exam me aise poocha β abhi try karo
Overfitting (train 10%, test 56%) β kya input noise help karega? ET Jan'26
Haan β input noise β L2 regularization.
10. Output me noise β label smoothing Module 6.8
Intuition: true labels pe pura bharosa mat karo, woh noisy ho sakte hain.
| Hard target (one-hot) | Soft target |
| $p=(0,0,1,0,\dots,0)$ | $p=\left(\tfrac{\varepsilon}{9},\tfrac{\varepsilon}{9},1-\varepsilon,\tfrac{\varepsilon}{9},\dots\right)$ |
10 classes: sahi class ko $1-\varepsilon$, baaki 9 me $\varepsilon$ barabar baant do ($\varepsilon/9$ each). Phir minimize $-\sum p_i\log q_i$. Model over-confident nahi hota.
β PYQ definitionSoft target = "a target value that is continuous or probabilistic rather than discrete". (0/1 wala = hard target.)
π Exam me aise poocha β abhi try karo
Output pe noise inject karne me "soft target" kya hai? Quiz-2 Jan'25
Continuous/probabilistic target (discrete 0/1 nahi).
11. Early stopping Module 6.9
AlgorithmValidation error track karo. Patience $p$: agar step $k$ pe pichhle $p$ steps me validation error improve nahi hua, to training roko aur step $k-p$ wala model return karo.
- Sabse effective aur most widely used regularizer. Doosre regularizers (L2) ke saath bhi use ho sakta hai.
- Intuition: $w_{t}=w_0-\eta\sum\nabla w_i$. Agar $\tau$ = max gradient, to $|w_{t+1}-w_0|\le\eta t|\tau|$. Yaani $t$ (steps) control karta hai $w$ kitna door ja sakta hai β exploration space limit.
- Math: quadratic approx + $w_0=0$ se: $w_t=Q[I-(I-\eta\Lambda)^t]Q^Tw^*$. L2 ka result: $\tilde w=Q[I-(\Lambda+\alpha I)^{-1}\alpha]Q^Tw^*$. Dono barabar agar
$$(I-\eta\Lambda)^t=(\Lambda+\alpha I)^{-1}\alpha$$
β early stopping β L2 regularization.
- Important param (bada gradient) jaldi badhta hai; unimportant (chhota gradient) $t$ steps me bada nahi ho paata β woh effectively shrink β same as weight decay.
π Exam me aise poocha β abhi try karo
Overfitting kaun rokta hai: Dropout, Early stopping, L2, MaxOut? ET Jan'26 MSQ
Dropout, Early stopping, L2.
12. Ensemble methods / Bagging Module 6.10
Alag models ke outputs combine karo (LR, SVM, Naive Bayes, ya same classifier with different hyperparameters / features / data samples).
Bagging: same classifier ke instances; dataset se sampling with replacement karke $T_1,\dots,T_k$ banao, $i$-th model ko $T_i$ pe train karo.
Bagging kab kaam karta hai?
$k$ models, error $\varepsilon_i$, $E[\varepsilon_i^2]=V$, $E[\varepsilon_i\varepsilon_j]=C$:
$$\text{mse}=E\Big[\Big(\tfrac1k\textstyle\sum\varepsilon_i\Big)^2\Big]=\frac1{k^2}\big(kV+k(k-1)C\big)=\frac{V}{k}+\frac{k-1}{k}C$$
- Perfectly correlated ($C=V$) β mse $=V$ β bagging se koi fayda nahi.
- Uncorrelated ($C=0$) β mse $=V/k$ β error $k$ guna kam.
- Average pe ensemble kam se kam apne individual members jitna achha hota hai.
13. Dropout Module 6.11 numerical aata hai
Problem: ensembles achhe hain lekin kai bade neural nets train karna mehenga, aur test time pe sabko combine karna real-time me infeasible. Dropout dono solve karta hai.
Dropout kya hai
Training me har node ko probability $p$ se retain karo (typically hidden $p=0.5$, input/visible $p=0.8$); drop hua node apne saare in/out connections ke saath temporarily hata do β thinned network.
$n$ nodes β $2^n$ possible thinned networks. Trick: (1) saare networks weights share karte hain, (2) har training instance / mini-batch pe naya thinned network sample karo.
- Backprop me sirf active weights update hote hain. Jo weight dono instances me active tha usko 2 updates, jo ek me tha usko 1.
- Har thinned network shayad hi train ho, lekin parameter sharing se koi parameter untrained nahi rehta.
- Har iteration me naya random mask β jo $t$ pe drop hua, woh $t+1$ pe active ho sakta hai.
- $p$ har layer ke liye alag ho sakta hai.
Test time
$2^n$ networks aggregate karna impossible. Isliye poora network use karo, aur har node ka output (ya uske outgoing weights) ko uske training me "on" rehne ke fraction $p$ se scale karo: $w\to pw$.
Kyu kaam karta hai: masking noise β hidden units co-adapt nahi kar paate (kisi doosre unit pe zyada depend nahi kar sakte, woh kabhi bhi drop ho sakta hai). Example: $h_i$ nose detect karke face pehchanta hai; $h_i$ drop hua to model ko redundantly nose encode karna ya doosre features (aankh) se face pehchanna seekhna padta hai β robust.
β PYQ Sep'25 type (test-time numerical)
$x=[2,1]$; hidden ReLU: $h_{11}=2(1)+1(0.5)=2.5$, $h_{12}=2(-1)+1(1)=-1\to0$. Dropout 0.5 β test pe scale: $[1.25,0]$. Output $\sigma(1\cdot1.25+1\cdot0)=\sigma(1.25)=\mathbf{0.777}$.
β Common traps
Dropout sparse weights nahi banata. Test time pe koi neuron drop nahi hota. Dropped set fixed nahi, har iteration random.
π Exam me aise poocha β abhi try karo
$x=[2,1]$, ReLU hidden ($w$: 1, 0.5 / β1, 1), output weights 1, 1, sigmoid, dropout 0.5. Test output? Quiz-2 Sep'25
$h=[2.5,0]$ β Γ0.5 β 1.25 β $\sigma(1.25)=0.77$.
Dropout + mini-batch GD: sparsity? layer-wise alag $p$? Iteration $t$ pe drop hua neuron, $t+1$ pe bhi update nahi hoga? Quiz-2 Jan'26
Sirf "$p$ har layer ke liye alag ho sakta hai" sahi hai.
Dropout $p=0.5$: training me random/fixed set? Test pe drop ya scale? ET Jan'25
Training: har iteration random set. Test: koi drop nahi, 0.5 se scale.
14. Loss landscape, grouping, implicit regularization
- Flat minimum β better generalization, sharp minimum β poor. Explicit regularization loss surface ko smooth karta hai.
| Data $x_i$ | Architecture $f(\cdot)$ | Optimizer $\nabla$ | Cost penalty $\mathcal L(\cdot)$ |
| Data augmentation, Noise injection | Dropout, Skip connections | SGD, Early stopping | L1, L2, Weight sharing, Pooling |
Explicit: L1, L2, data augmentation, noise injection, skip connections. Implicit: SGD, large/small initial LR, early stopping, weight sharing, pooling, dropout.
Random labelling experiment (Zhang et al.)
Training labels ko random shuffle karo (inputβoutput relation khatam). Chhota model (19 params) aise data ko fit nahi kar paata β training error high (expected). Lekin deep nets random labels pe bhi training error zero kar dete hain! Conclusion: regularizers tuned hon to generalization improve karte hain, lekin woh generalization ki fundamental wajah nahi lagte (sab regularizers hata do tab bhi nets achha perform karte hain). Over-parameterized models SGD/GD ke discrete steps se khud implicitly regularize hote hain.
π Exam me aise poocha β abhi try karo
Regularization kaun: data augmentation, activation functions, L2, input/output noise, optimization techniques? Quiz-2 Sep'24 MSQ
Data augmentation, L2, noise. (Activations aur optimizers nahi.)
Overfitting kaun kam karta hai: data aug, dropout, L2, Adam instead of SGD? Quiz-2 Sep'25 MSQ
Pehle teen.
15. One-page formula sheet
16. PYQs β fully solved
Yeh sab sawaal upar theory sections ke π boxes me already aa chuke hain. Yahan poore options ke saath. Official answer keys. π Practice se blur karke khud try karo.
A. Kaun se regularizers hain? (MSQ)
Quiz-2 Sep'24MSQ Β· 3 marks
Which of the following are regularization technique(s)?
- Dataset augmentation
- Activation functions
- L2 regularization
- Adding noise to the inputs or outputs
- Optimization techniques
- All of these
Answer: (a), (c), (d). Activation functions (sigmoid, ReLU) aur optimizers (Adam, RMSProp) regularizers nahi.
Quiz-2 Sep'25MSQ Β· 3 marks
Which techniques can be used to reduce model overfitting?
- Data augmentation
- Dropout
- L2 regularization
- Using Adam instead of SGD
Answer: (a), (b), (c).
ET Jan'26 ANMSQ Β· 2 marks
Which techniques prevent the model from overfitting?
- Dropout
- Early stopping
- L2 regularization
- MaxOut
Answer: (a), (b), (c). MaxOut ek activation function hai.
ET Jan'26 ANMSQ Β· 2 marks
Training error 10%, test error 56%. Which methods can reduce the test error?
- Adam
- ReLU activation
- Injecting noise at input
- Maxout
- L2 regularization
Answer: (c), (e). Bada gap = overfitting (high variance) β regularization chahiye.
B. Bias / Variance
Quiz-2 Sep'24MCQ Β· 3 marks
Decision boundaries: Model A = straight line, Model B = bahut wiggly (har point ke around), Model C = smooth moderate curve. Bias ka correct order?
- A > B > C
- B > C > A
- C > B > A
- A > C > B
Answer: (d) A > C > B. Simple = high bias.
Quiz-2 Jan'25SA Β· 3 marks important
Same architecture, 3 subsets pe trained models A, B, C. True $y=[1,3,6,8]$. Predictions: A $[1,4,7,8]$, B $[3,5,6,9]$, C $[2,3,5,7]$. Variance averaged over all test points?
Har point pe mean: $x_1:\frac{1+3+2}3=2$, $x_2:4$, $x_3:6$, $x_4:8$.
Variance per point: $x_1:\frac{1+1+0}{3}=\frac23$; $x_2:\frac{0+1+1}3=\frac23$; $x_3:\frac{1+0+1}3=\frac23$; $x_4:\frac{0+1+1}3=\frac23$.
Average $=\mathbf{0.67}$ (range 0.62β0.72). True $y$ variance me use nahi hota (woh bias ke liye hai).
Quiz-2 Jan'26Comprehension Q9β10
Model $y=w_0+w_1x$, 3 baar 5-point samples pe fit. $x=2$ pe $\hat f_1=1.1,\hat f_2=0.9,\hat f_3=1.0$, true $y=1$. (i) Bias? (ii) Variance (4 decimals)?
(i) $E[\hat f]=1$ β Bias $=\mathbf0$. (ii) $\frac{0.01+0.01+0}{3}=\mathbf{0.0067}$ (range 0.0064β0.0070).
Quiz-2 2025 T2MCQ Β· 3 marks tricky
4 loss curves (solid = train, dashed = val), 50 epochs, final model rakha:
$M_1$: dono low aur paas paas. $M_2$: dono
badhte ja rahe. $M_3$: train girta, val U-shape (bada gap). $M_4$: dono girke
high level pe plateau, chhota gap.
(A) LR too high (B) low bias high var (C) low bias low var (D) high bias low var
- M1βC, M2βA, M3βB, M4βD
- M1βD, M2βA, M3βC, M4βB
- M1βA, M2βB, M3βC, M4βD
- M1βD, M2βC, M3βB, M4βA
Answer: (a). Loss badh raha hai (train bhi) = LR too high, diverge. Dono high pe atke = underfit = high bias. β PDF me handwritten note M2/M4 ulta likha hai β key follow karo.
ET Jan'25MSQ Β· 3 marks
Error vs complexity plot: curve (1) U-shape, curve (2) monotonically girta (1 ke neeche), (3) = bahut low complexity pe dashed line.
- (1) test error
- (2) training error
- (3) high bias, low variance
- (1) training error
- (2) test error
- (3) low bias, high variance
Answer: (a), (b), (c). (3) = underfit.
ET May'25 ANMCQ Β· 2 marks
S1: Zero training loss wala model unseen data pe achha perform karega hi. S2: Regularization training loss badha sakta hai lekin generalization error kam.
- Both true
- S1 true, S2 false
- None of these
- S1 false, S2 true
Answer: (d).
C. L2
Quiz-2 Jan'25MCQ Β· 3 marks
Primary purpose of L2 regularization?
- Prevent overfitting by adding sum of absolute values of weights
- Encourage sparsity by setting some weights to zero
- Prevent overfitting by adding sum of squared values of weights to the loss
- Increase complexity by penalizing small weights
Answer: (c). (a), (b) = L1.
Quiz-2 2025 T2Comprehension Q240β242
$L_D(w)=w_1^2+w_2^2-2w_1-2w_2+2$, $L_R=\frac\lambda2\|w\|^2$, $L=L_D+L_R$.
(i) $w_D^*$? (ii) $w^*=(0.5,0.5)$ ho to Ξ»? (iii) Kya kisi Ξ»>0 ke liye $\|w^*\|>\|w_D^*\|$?
(i) $2w_i-2=0$ β $\mathbf{(1,1)}$.
(ii) $2w_i-2+\lambda w_i=0$ β $w_i=\frac{2}{2+\lambda}=0.5$ β $\lambda=\mathbf2$.
(iii) No, $w^*=\frac{2}{2+\lambda}(1,1)$ hamesha origin ki taraf shrink.
Quiz-2 Sep'25MCQ Β· 3 marks
L2, Ξ» β {0.001, 0.01, 0.1, 1}. Training loss Ξ» ghatne pe ghatta hai, validation loss Ξ» = 0.1 pe minimum. Kya choose karein?
Answer: Ξ» = 0.1 β lowest validation loss, best biasβvariance trade-off.
D. Dropout, label smoothing, tying
Quiz-2 Sep'25MCQ Β· 4 marks
2 inputs β 2 ReLU hidden β 1 sigmoid output. $w_{111}=1,w_{112}=0.5,w_{121}=-1,w_{122}=1,w_{211}=1,w_{212}=1$, no bias, dropout 0.5 on hidden. $x=[2,1]$. Expected output while testing?
- 0.77
- 1.25
- 2.5
- 0.92
Answer: (a) 0.77. $h=\text{ReLU}([2.5,-1])=[2.5,0]$ β Γ0.5 β $[1.25,0]$ β $\sigma(1.25)=0.777$. (Bina scaling: $\sigma(2.5)=0.92$ β trap option.)
Quiz-2 Jan'26MCQ Β· 2 marks
Feed-forward net me dropout + mini-batch GD. Correct statement?
- This will lead to sparsity in the trained weights
- The dropout probability $p$ can be different for each hidden layer
- Weights of neurons dropped at iteration $t$ will not get updated during iteration $t+1$
Answer: (b). (c) galat β har iteration naya mask.
ET Jan'25MSQ Β· 2 marks
Hidden layer pe dropout $p=0.5$. True?
- Training me random set drop; har iteration dynamically decide
- Inference me koi drop nahi; har activation 0.5 se scale
- Training me fixed set drop, pehle se decided
- Inference me random set drop
Answer: (a), (b).
Quiz-2 Jan'25MCQ Β· 3 marks
What is the soft target used when injecting noise at the output?
- Label that is either 0 or 1
- A target value that is continuous or probabilistic rather than discrete
- Random label for misclassified samples
- Label always equal to predicted output
Answer: (b).
Quiz-2 2025 T2SA Β· 2 marks
Autoencoder: hidden = 30% of input, no biases. Network-1 untied $W_1,W_2$; Network-2 tied $W_2=W_1^T$. $n_1/n_2$?
Answer: 2. $0.6n^2/0.3n^2$.
NoteWeek 6 PYQ PDF me ek CNN comprehension (Q237β239: 64Γ64Γ8 input, 16 kernels 5Γ5, stride 1, pad 2 β output 64Γ64Γ16, params 3200, input 128Γ128Γ8 karne pe params same) bhi hai. Yeh Week 8 ka topic hai, isliye yahan detail nahi.
17. Graded Assignment Week 6 (GA)
β GA ka official key PDF me nahi tha β yeh answers maine khud solve kiye hain. Submit karne se pehle khud bhi verify kar lena.
GA Q1: $N_1$ me $K_1$ hidden layers, $N_2$ me $K_2$, har layer $n$ neurons, $K_1\gg K_2$. Higher complexity?
Answer: $N_1$ β zyada layers = zyada parameters aur non-linearities.
GA Q2: $E[(y-\hat f(x))^2]$ (slide 13) ke according true statements?
- Training error sufficiently complex model se zero ho sakta hai
- Training error irreducible error ki wajah se zero nahi ho sakta
- Test error zero ho sakta hai
- Test error irreducible error ki wajah se zero nahi ho sakta
Answer: (a), (d). Complex model training points (noise samet) ratt leta hai; test pe $\sigma^2$ bachega.
GA Q3: Model ka training error zero hai. L2 lagao to?
- Training error badh sakta hai
- Test error ghat sakta hai
- Kam important weights ko zero ke paas laakar complexity kam
- Training error ghat sakta hai
Answer: (a), (b), (c). Training error already 0, aur kam nahi ho sakta.
GA Q4βQ8: $\mathcal L(w,b)=0.5w^2+5b^2+1$
Q4: $\mathcal L(w^*,b^*)$? Q5: $\nabla\mathcal L(w^*,b^*)$ ke elements ka sum? Q6: $\det H$? Q7: Loss kis param ke prati zyada sensitive? Q8: "Regularization unregularized minimum $(w^*,b^*)$ ko nahi badalta" β True/False?
Minimum $(0,0)$.
Q4: 1. Q5: 0 (minimum pe gradient zero). Q6: $H=\begin{bmatrix}1&0\\0&10\end{bmatrix}$ β $\det=\mathbf{10}$.
Q7: Eigenvalues 1 (direction $[1,0]$ = $w$) aur 10 (direction $[0,1]$ = $b$) β bada eigenvalue = zyada curvature β $b$.
Q8: True. EVD: $H=\text{diag}(1,10)$ β $\tilde w=Q(\Lambda+\alpha I)^{-1}\Lambda Q^Tw^*$. Yahan $w^*=(0,0)$, to $\tilde w=QDQ^T\cdot\mathbf 0=(0,0)$ β minimum nahi badla. (General me L2 minimum ko origin ki taraf shift karta hai; yahan minimum already origin pe hai, isliye koi change nahi.)
18. My notes (khali pages)
Free writing space. Use β Page in the toolbar to add more ruled pages.