BDS 761: Basic Flashcards¶
Lectures 01–04 · Quizzes 1–5 · Homeworks 1–5
190 short questions in 14 topics.
Try each question before moving to its answer. Use a few cards at a time; explain each answer in your own words.
Flashcards Index (1/2)¶
Topic links work in the notebook; slides links work in slideshow mode.
Flashcards Index (2/2)¶
Topic links work in the notebook; slides links work in slideshow mode.
- Preprocessing and Train/Test Data · slides
- Matrix Multiplication and Flat Arrays · slides
- PyTorch Tensor Basics · slides
- Reshaping, Vectorization, and Kronecker Products · slides
- Images, Batches, and Pixel Statistics · slides
- Nearest Neighbors and Voting · slides
- Image Patches for a Vision Transformer · slides
Question 1¶
In this course, what does it mean for a model to "learn"?
Answer 1¶
It uses data to estimate a rule or relationship, rather than having every rule written by hand.
Question 2¶
What is a feature?
Answer 2¶
An input measurement used by a model, such as a flower's petal length.
Question 3¶
What is a target or label in supervised learning?
Answer 3¶
The answer we want the model to predict, such as a flower's species.
Question 4¶
Predicting a house price: regression or classification?
Answer 4¶
Regression, because the target is a numerical value.
Question 5¶
Predicting a flower species: regression or classification?
Answer 5¶
Classification, because the target is a category.
Question 6¶
What does clustering try to find?
Answer 6¶
Groups of similar examples, usually without known class labels.
Question 7¶
What does dimensionality reduction try to do?
Answer 7¶
Represent data with fewer features while keeping useful information.
Question 8¶
For subjects in a study, we have measurements of their blood pressure, cognitive function, age, and "brain age" biomarker? What is the label (a.k.a. target) and what are the dependent variables in a supervised learning model?
Answer 8¶
Trick question. It is our choice, the label is whatever data variable we want to train the model to predict, using the rest as inputs.
Question 9¶
Why do we use array operations instead of a Python loop over every number?
Answer 9¶
Libraries can perform many operations efficiently together, including in parallel on suitable hardware.
Normal Distributions¶
27 basic questions.
Review: Quiz 1, HW 1.
Sources: BDS761_02_Regression_01_Simple, BDS761_02_Regression_02_LinearAlgebra
Question 1¶
What is the difference between a Normal and Gaussian distribution?
Answer 1¶
They are the same thing in different fields.
Question 2¶
$$ f(x)=\frac{1}{\sqrt{2\pi}\,\sigma} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) $$
In a one-dimensional Gaussian, what does $\mu$ describe?
Answer 2¶
The mean: the location of the center of the distribution.
Question 3¶
$$ f(x)=\frac{1}{\sqrt{2\pi}\,\sigma} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) $$
What do $\sigma$ and $\sigma^2$ mean?
Answer 3¶
$\sigma$ is the standard deviation; $\sigma^2$ is the variance. Both describe spread.
Question 4¶
$$ f(x)=\frac{1}{\sqrt{2\pi}\,\sigma} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) $$
A Gaussian has standard deviation 3. What is its variance?
Answer 4¶
$3^2 = 9$.
Question 5¶
What are the mean and variance of a standard normal distribution?
Answer 5¶
Mean 0 and variance 1.
Question 6¶
$$ f(x)=\frac{1}{\sqrt{2\pi}\,\sigma} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) $$
If we have a measurement $x=8$, with $\mu=2$, and $\sigma=3$, what is the standardized value?
Answer 6¶
$(x-\mu)/\sigma = (8-2)/3 = 2$.
Question 7¶
$$ f(x)=\frac{1}{\sqrt{2\pi}\,\sigma} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) $$
In a one-dimensional Gaussian, how many numbers specify one possible observation $x$?
Answer 7¶
One number. For example, $x=5$ is one possible observation.
Question 8¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
In an $n$-dimensional Gaussian, what does the dimension $n$ count?
Answer 8¶
The number of scalar variables in each observation vector. Each corresponds to one variable being modeled.
Question 9¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
In a multidimensional Gaussian random variable for which one observation is $\mathbf x=(3,7)^T$. What is $n$ above?
Answer 9¶
$n=2$.
Question 10¶
For one flower, petal length is 4 cm and petal width is 1 cm. Create a vector $\mathbf x$ containing these features.
Answer 10¶
$\mathbf x=\begin{pmatrix}4\\1\end{pmatrix}$. The first coordinate stores length; the second stores width.
Question 11¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
How do we model petal length and petal width together using a Gaussian?
Answer 11¶
The two measured features, petal length and petal width, are modeled as different random variables; hence they are represented by different dimensions and we use a 2D multidimensional Gaussian which models their variation jointly.
Question 12¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
We measure petal length and petal width for 100 flowers. For a Gaussian model of this data, what is the dimeension $n$ of the model?
Answer 12¶
$n=2$, because each flower has two modeled features. The 100 flowers are 100 observations, each represented by a two-coordinate vector.
Question 13¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
When an $n$-dimensional Gaussian models feature vectors, what are $n$ and $\mathbf x$?
Answer 13¶
$n$ is the number of features modeled together. $\mathbf x$ is one observation vector containing a value for each of those features. For petal length and width, $n=2$.
Question 14¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
For two features, what is the shape of the mean vector $\boldsymbol\mu$?
Answer 14¶
It has two entries, one mean per feature. As a column vector, its shape is $2\times1$.
Question 15¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
In the equation for a $n$-dimensional multidmensional Gaussian, what does $\boldsymbol\Sigma$ describe and what size is it?
Answer 15¶
The covariance of the variables, which gives spread of each feature and how pairs of features vary together. For $n$ features, it is $n\times n$.
Question 16¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
What is stored on the diagonal of a covariance matrix?
Answer 16¶
The individual feature variances.
Question 17¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
What does a positive off-diagonal covariance suggest?
Answer 17¶
The two features tend to be above their means together and below their means together.
Question 18¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
In the Gaussian formula, what do $|\boldsymbol\Sigma|$ and $\boldsymbol\Sigma^{-1}$ mean?
Answer 18¶
The determinant and the matrix inverse of the covariance matrix. The inverse requires a nonsingular covariance matrix.
Question 19¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
What does $\mathbf x-\boldsymbol\mu$ represent?
Answer 19¶
The difference between an observation and the mean, feature by feature.
Question 20¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
For $\boldsymbol\Sigma=I$, what does $(\mathbf x-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)$ become?
Answer 20¶
$\|\mathbf x-\boldsymbol\mu\|_2^2$: the squared Euclidean distance from the mean. Here $T$ means transpose and $I$ is the identity matrix.
Question 21¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}} \exp\!\left(-\frac{1}{2}(\mathbf x-\boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\right) $$
What does the factor $1/\sqrt{(2\pi)^n|\boldsymbol\Sigma|}$ do?
Answer 21¶
It scales the Gaussian density so that its integral over all observations is 1, to agree with a fundamental axiom of probability. We don't care about this term if we just want to find the location of the maximum.
Answer 22¶
A muldimensional Gaussian with $\mathbf\mu = \mathbf w$ and $\boldsymbol\Sigma=\mathbf Q$
Question 23¶
$$ f(\mathbf x)=\frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma|}}\exp(-q/2), \qquad q=(\mathbf x-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf x-\boldsymbol\mu) $$
In $\exp(-q/2)$, what happens to the value as a nonnegative $q$ increases?
Answer 23¶
It decreases. Observations farther from the mean in covariance-scaled distance have lower density.
Question 24¶
$$ f(x)=\frac{1}{\sqrt{2\pi}\,\sigma} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) $$
Is the height $f(x)$ of a continuous density the probability of observing exactly $x$?
Answer 24¶
No. Probabilities come from areas over intervals. For a continuous Gaussian, the probability of one exact value is 0.
Question 25¶
How can NumPy generate 1000 two-feature observations with mean $(-1,2)$, standard deviations $(3,1)$, and covariance 0.5?
Answer 25¶
rng = np.random.default_rng(0)
samples = rng.multivariate_normal([-1, 2], [[9, 0.5], [0.5, 1]], size=1000)
The result has shape (1000, 2). The off-diagonal value is covariance, not correlation.
Question 26¶
Suppose I want to model images of size $1024\times 768$ with a multidimensional normal distribution, what are the parameters of my statistical model?
Answer 26¶
$n=1024*768 = 786432$, $\boldsymbol\mu$ is a length-786432 vector describing the average image (flattened into a vector). $\boldsymbol\Sigma$ is a $786432\times 786432$ covariance matrix describing covariance between pixels pairwise.
Question 27¶
In machine learning terms, what are the features for the model of images
Answer 27¶
Individual pixels of an image
Simple Regression and Squared Error¶
16 basic questions.
Review: Quiz 1, HW 1.
Sources: BDS761_02_Regression_01_Simple
Question 1¶
What is the prediction equation for simple linear regression?
Answer 1¶
$\hat y = \beta_0 + \beta_1 x$. The hat is optional notation to indicate a predicted value.
Question 2¶
What does the $\beta_0$ represent mathematically?
Answer 2¶
The $y$-intercept or predicted value of $y$ when $x=0$.
Question 3¶
What does the $\beta_1$ represent mathematically?
Answer 3¶
The slope or change in the prediction when $x$ increases by 1.
Question 4¶
What is the residual in a regression model
Answer 4¶
The difference between a value predicted by the model and a value from data: $e=y-\hat y$
Question 5¶
What does ordinary least squares (OLS) minimize?
Answer 5¶
The sum of squared residuals: $\sum_i (y_i-\hat y_i)^2$.
Question 6¶
Why square residuals rather than just add them?
Answer 6¶
If we just added them, negative residuals would imply the total fit got better and counter positive residuals. We could alternatively take the absolute value to fix this, but it is discontinuous at zero.
Also it turns out squaring matches closely with Gaussian statistical models (which turn out to be great models to use in many many domains).
Question 7¶
Residuals are $[1,-2,0]$. What is the mean squared error (MSE)?
Answer 7¶
$\left(1^2+(-2)^2+0^2\right)/3 = 5/3$. MSE is the sum of squared errors divided by the number of observations.
Question 8¶
How is the Gaussian used in the simple regression model?
Answer 8¶
At a given $x$, $Y$ has mean $\beta_0+\beta_1x$ and noise variance $\sigma^2$, so P($Y\mid X=x) = \mathcal N(\beta_0+\beta_1x,\sigma^2)$.
Question 9¶
What does a likelihood describe?
Answer 9¶
How well different choices of parameters explain the observed data under a probability model. It is the overall joint probability of the data (which we can easily factor into a product of independent probabilities for each measurement when the data is presumed to be iid).
Question 10¶
What does maximum likelihood estimation (MLE) choose?
Answer 10¶
The parameter values that maximize the likelihood of the observed data.
Question 11¶
Why can we maximize the log-likelihood instead of the likelihood?
Answer 11¶
Log is increasing, so it keeps the same maximizer. It also turns products into sums.
Question 12¶
What does i.i.d. mean?
Answer 12¶
This describes a dataset, saying each measurement of data is independent (different measurement are independent of each other so the joint probability is the product of measurement probabilities) and identically distributed (every measurement has the same statistical distribution e.g., Gaussian, with same mean and variance).
Answer 13¶
The parameter value $\beta^*$ that makes $J(\beta)$ smallest, rather than the smallest value of $J$ itself. So
$$J(z) = J(\beta^*) \le J(\beta) \text{ for all possible } \beta$$
Question 14¶
For a differentiable squared-error objective, what is the calculus process for finding its minimizing coefficients analytically?
Answer 14¶
Differentiate with respect to each coefficient and set those derivatives equal to zero. This gives you a collection of equations to solve for the minimizing coefficients (or maximizing coefficients, or saddle point... technically you should check the second derivative as a good calculus student).
Question 15¶
What does the linear minimum mean squared error (LMMSE) estimator minimize?
Answer 15¶
The expected squared prediction error, $\mathbb E[(Y-\beta_0-\beta_1X)^2]$. OLS uses an observed dataset instead.
Question 16¶
Suppose we want to predict "brain age" given blood pressure, cognitive function, and age, using linear regression, give the regression model we would use, identifying what the variables mean.
Answer 16¶
$$ y_{\text{Brain age}} = \beta_0 + x_{\text{blood pressure}} \beta_1 + x_{\text{cognitive function}} \beta_2 + x_{\text{age}} \beta_3 $$
Question 1¶
For $M$ subjects, measuring $N$ features of each person, what is the shape of the data matrix (a.k.a., feature matrix or design matrix in machine learning)?
Answer 1¶
(M, N): one row per subject and one column per feature. We could define them as the transpose of this if we wanted, but the first way is the standard commonly used in machine learning.
Question 2¶
How do you turn $y_i=\beta_0+\beta_1x_i$ into a matrix equation for $M$ measurements ($i=1...M$), and known $\beta_0$ and $\beta_1$ ?
Answer 2¶
Put [1, x_i] in row $i$ of $A$, put $[\beta_0,\beta_1]^T$ in $z$, and put the observed $y_i$ values in $b$. Then $Az\approx b$.
Question 3¶
Why do we add a column of ones to a regression design matrix?
Answer 3¶
Multiplying that column by $\beta_0$ adds the same intercept to every prediction.
Question 4¶
If there are $M$ subjects and $N$ features, what is the design matrix shape when we include the $\beta_0$ term in the model it describes?
Answer 4¶
(M, N + 1). There are N + 1 coefficients to estimate.
Question 5¶
Using Python's zero-based indexing, what are $A_{i,0}$, $A_{i,j}$, $z_j$, and $b_i$ in an affine regression system?
Answer 5¶
$A_{i,0}=1$; $A_{i,j}$ for $j\ge1$ is subject $i$'s feature $j$; $z_j=\beta_j$; and $b_i=y_i$. The intercept is $z_0=\beta_0$.
Question 6¶
How can NumPy add an intercept column to a two-dimensional feature matrix X?
Answer 6¶
Create a ones vector and stack or concatenate it:
A = np.column_stack([np.ones(X.shape[0]), X])
Question 7¶
If A.shape == (5, 3) and beta.shape == (3,), what is the shape of A @ beta?
Answer 7¶
(5,): one prediction per subject.
Question 8¶
What does "overdetermined" mean for $Az=b$?
Answer 8¶
There are more equations than unknowns. An exact solution may not exist, so we minimize the squared residual.
Question 9¶
What does "underdetermined" mean for $Az=b$?
Answer 9¶
There are fewer equations than unknowns. If the equations are consistent, there are infinitely many exact solutions.
Question 10¶
Can every square matrix be inverted?
Answer 10¶
No. It must be nonsingular (have full rank). For a square matrix, this means its determinant is nonzero.
Question 11¶
What does the pseudoinverse let us do?
Answer 11¶
Compute a least-squares solution even for a rectangular or singular matrix. If several solutions minimize error, it selects the one with smallest Euclidean norm.
Question 12¶
How do you compute a least-squares coefficient vector using NumPy's pseudoinverse?
Assume your target vector is y and your data matrix is A.
Question 13¶
What is one row of the design matrix for the polynomial model $\hat y=\beta_0+\beta_1x+\beta_2x^2$?
Answer 13¶
[1, x, x**2] where x is a measurement value. The model is nonlinear in $x$, but still linear in the coefficients.
Question 1¶
What is overfitting?
Answer 1¶
Fitting the training data's noise or accidental details so closely that predictions on new data suffer.
Question 2¶
What does regularization add to a fitting objective?
Answer 2¶
A penalty that discourages some choices of coefficients, such as very large coefficients.
Question 3¶
What is a prior distribution on coefficients?
Answer 3¶
A model of plausible coefficient values before conditioning on the observed targets.
Question 4¶
What is a posterior distribution on coefficients?
Answer 4¶
The distribution after combining the prior with the evidence from the observed data.
Question 5¶
What does maximum a posteriori (MAP) estimation choose?
Answer 5¶
The coefficients at the maximum of the posterior. Unlike MLE, MAP includes a prior.
Question 6¶
In Bayes' rule, what two factors determine the posterior up to a constant?
Answer 6¶
Posterior is proportional to Likelihood times prior: $p(\beta\mid y,X)\propto p(y\mid\beta,X)p(\beta)$.
Question 7¶
Why can we ignore the denominator from Bayes rule the when maximizing the posterior over $\beta$?
Answer 7¶
It does not change when $\beta$ changes; it is a constant for this optimization.
Question 8¶
Which coefficient prior leads to ridge regression with Gaussian observation noise?
Answer 8¶
A zero-mean Gaussian prior. Its negative log contributes a squared $L_2$ penalty.
Question 9¶
What is the ridge regression objective?
Answer 9¶
$\|y-A\beta\|_2^2+\lambda\|\beta\|_2^2$. If the intercept is not penalized, the penalty includes only the other coefficients.
Question 10¶
What does increasing $\lambda\ge0$ do to ridge's penalty?
Answer 10¶
It makes large penalized coefficients more costly and encourages stronger shrinkage toward zero.
Question 11¶
If noise variance is $\sigma^2$ and an independent zero-mean Gaussian coefficient prior has variance $\tau^2$, what is ridge's $\lambda$ in the unscaled squared-error objective?
Answer 11¶
$\lambda=\sigma^2/\tau^2$. A tighter prior (smaller $\tau^2$) gives stronger regularization.
Question 12¶
Is the intercept usually penalized in ridge or LASSO?
Answer 12¶
No. It sets the baseline prediction. Typically only the feature coefficients are penalized.
Question 13¶
Which prior leads to LASSO with Gaussian observation noise?
Answer 13¶
Independent zero-centered Laplace priors, with density proportional to $\exp(-|\beta_j|/b)$ for scale $b>0$.
Question 14¶
What is the LASSO objective?
Answer 14¶
$\|y-A\beta\|_2^2+\lambda\|\beta\|_1$. The $L_1$ penalty sums absolute coefficient values, usually excluding the intercept.
Question 15¶
Which method can set coefficients exactly to zero: ridge or LASSO?
Answer 15¶
LASSO can. Ridge usually shrinks coefficients without making them exactly zero.
Question 16¶
How can ridge ridge be written as an ordinary least-squares problem using extra rows?
Answer 16¶
For a model with no intercept and with all coefficients penalized, append $\sqrt\lambda I$ to $A$ and zeros to $y$:
A_aug = np.vstack([A, np.sqrt(lam) * np.eye(A.shape[1])])
y_aug = np.concatenate([y, np.zeros(A.shape[1])])
beta = np.linalg.pinv(A_aug) @ y_aug
Here y is one-dimensional and lam >= 0. The added rows create the ridge penalty.
Question 1¶
What does a vector norm measure?
Answer 1¶
A vector's size or length, summarized as one nonnegative number.
Question 2¶
Is the number of entries in a vector the same as its Euclidean norm?
Answer 2¶
No. [3, 4] has two entries but Euclidean norm 5.
Question 3¶
What is the $L_1$ norm of $[-2,1,0]$?
Answer 3¶
$|-2|+|1|+|0|=3$.
Question 4¶
What is the $L_2$ norm of $[-2,1,0]$?
Answer 4¶
$\sqrt{(-2)^2+1^2+0^2}=\sqrt5$.
Question 5¶
What is the $L_\infty$ norm of $[-2,1,0]$?
Answer 5¶
$\max(2,1,0)=2$: the largest absolute entry.
Question 6¶
What is the so-called $L_0$ "norm" of $[-2,1,0]$?
Answer 6¶
2: the number of nonzero entries. This count is not a true mathematical norm.
Question 7¶
What is the finite $p$-norm expression for $p>0$?
Answer 7¶
$\|v\|_p=(\sum_i|v_i|^p)^{1/p}$. It is a true norm when $p\ge1$; for $0<p<1$ it is called a quasi-norm.
Question 8¶
What does the $p=0.5$ expression give for $[-2,1,0]$?
Answer 8¶
$(\sqrt2+1)^2=3+2\sqrt2\approx5.828$. The outer exponent is $1/0.5=2$.
Question 9¶
What is the $p=3$ norm of $[-2,1,0]$?
Answer 9¶
$(2^3+1^3+0^3)^{1/3}=9^{1/3}\approx2.080$.
Question 10¶
How do you compute the Euclidean norm of a one-dimensional NumPy array v?
Answer 10¶
length = np.linalg.norm(v)
Question 11¶
What is the dot product of $[1,2]$ and $[3,4]$?
Answer 11¶
$1(3)+2(4)=11$: multiply matching entries, then add.
Question 12¶
How is a real vector's squared Euclidean norm related to its dot product?
Answer 12¶
$\|v\|_2^2=v^Tv$. For a one-dimensional NumPy array, use v @ v.
Question 13¶
What is the Euclidean distance between $[0,0]$ and $[3,4]$?
Answer 13¶
$\|[3,4]-[0,0]\|_2=5$.
Question 14¶
How do you compute the distance from a one-dimensional vector a to b in NumPy?
Answer 14¶
distance = np.linalg.norm(a - b)
A distance compares two vectors; a norm measures one vector.
Question 15¶
What is the Manhattan distance between $[0,0]$ and $[3,4]$?
Answer 15¶
$|3|+|4|=7$. It uses the $L_1$ norm of the difference.
Question 16¶
For nonzero vectors, what does cosine similarity compare?
Answer 16¶
Their directions: $\dfrac{a^Tb}{\|a\|_2\|b\|_2} = \cos \theta$. Vectors pointing in the same direction have similarity 1, even if their lengths differ.
Question 17¶
What does the triangle inequality say about a norm?
Answer 17¶
$\|a+b\|\le\|a\|+\|b\|$: the length of the sum of vectors cannot exceed the sum of the lengths.
Question 1¶
What is the mean of $[1,2,3]$?
Answer 1¶
$(1+2+3)/3=2$.
Question 2¶
What does it mean to center a vector of observations?
Answer 2¶
Subtract its mean from each entry: $x_c=x-\bar x$.
Question 3¶
What is $[1,2,3]$ after centering?
Answer 3¶
$[-1,0,1]$. Its mean is 0.
Question 4¶
How can NumPy compute population variance using a norm?
Answer 4¶
xc = x - x.mean()
variance = np.linalg.norm(xc)**2 / x.size
Here x is a nonempty one-dimensional array. Use x.size - 1 for sample variance when there are at least two entries.
Question 5¶
If $y=2x$ and $x$ is not constant, what is the correlation between $x$ and $y$?
Answer 5¶
1, they increase together in an exact linear relationship.
Question 6¶
What is the root mean square (RMS) of a vector $v$ of length $n>0$?
Answer 6¶
$\sqrt{\sum_i v_i^2/n}=\|v\|_2/\sqrt n$. For a centered vector, it equals the population standard deviation.
Preprocessing and Train/Test Data¶
13 basic questions.
Review: HW 3, Quiz 5.
Sources: BDS761_03_Norms
Question 1¶
If X contains subjects in rows and features in columns, which axis computes one mean per feature?
Answer 1¶
axis=0 in NumPy. This combines the subjects within each feature column.
Question 2¶
How do you center every column of a two-dimensional NumPy array X?
Answer 2¶
Xc = X - X.mean(axis=0)
Question 3¶
What is broadcasting in this example: X - column_means?
Answer 3¶
NumPy applies the same row of column means to each row of X, without requiring a Python loop.
Question 4¶
How do you compute one Euclidean norm for each column of Xc?
Answer 4¶
scale = np.linalg.norm(Xc, axis=0)
Question 5¶
How do you scale each nonzero centered column to have Euclidean norm 1?
Question 6¶
How can you check that all nonconstant columns were normalized to Euclidean norm 1?
Answer 6¶
column_norms = np.linalg.norm(Xn, axis=0)
Those entries should be approximately 1; constant columns have norm 0. Check Xn.mean(axis=0) for means approximately 0.
Question 7¶
What is the difference between unit-norm normalization and standardization?
Answer 7¶
Unit-norm normalization divides by a chosen vector norm. Standardization subtracts the mean and divides by standard deviation. These scales are generally different.
Question 8¶
How do you standardize feature columns using population standard deviations in NumPy?
Answer 8¶
mu = X.mean(axis=0)
Xc = X - mu
sd = Xc.std(axis=0)
Z = Xc / sd # = (X - mu) / sd
Nonconstant columns have mean approximately 0 and population standard deviation approximately 1.
Question 9¶
Why must features and targets use the same shuffled row indices?
Answer 9¶
To keep each subject's features paired with that subject's correct target.
Question 10¶
How do you shuffle rows of X and a one-dimensional target array y together in NumPy?
Answer 10¶
Generate permuted indices, then use these to form shuffled versions of X and y
rng = np.random.default_rng(0)
idx = rng.permutation(len(y))
X_shuffled, y_shuffled = X[idx], y[idx]
Question 11¶
How do you split already shuffled data into 80% training and 20% test data?
Answer 11¶
Perform a python split at 80 percent (or 20 percent) of the length
cutoff = int(0.8 * len(y_shuffled))
X_train, X_test = X_shuffled[:cutoff], X_shuffled[cutoff:]
y_train, y_test = y_shuffled[:cutoff], y_shuffled[cutoff:]
Question 12¶
What are the training set and test set used for?
Answer 12¶
The training set fits the model. The test set measures performance on examples held out from fitting.
Question 13¶
For ordinary train/test evaluation, which data should determine preprocessing means and scales?
Answer 13¶
Training data only. Apply those same means and scales to test data to avoid using information from the test set during fitting.
Matrix Multiplication and Flat Arrays¶
9 basic questions.
Review: Quiz 4.
Sources: BDS761_04_PyTorch_01_Tensors, BDS761_02_Regression_02_LinearAlgebra
Question 1¶
If $A$ is $M\times N$ and $B$ is $N\times P$, what is the shape of $AB$?
Answer 1¶
$M\times P$. The inner sizes $N$ must match.
Question 2¶
How do you compute one entry $C_{m,p}$ of $C=AB$ via matrix multiplication?
Answer 2¶
$$C_{m,p}=\sum_{n=0}^{N-1} A_{m,n}B_{n,p}$$.
Take a row of $A$, a column of $B$, multiply matching entries, and add.
Question 3¶
For NumPy arrays, how do A * B and A @ B differ?
Answer 3¶
* multiplies entries element by element (with broadcasting when possible). @ performs matrix multiplication.
Question 4¶
How is [[1, 2], [3, 4]] flattened in row-major order?
Answer 4¶
[1, 2, 3, 4]: finish the first row before starting the second.
Question 5¶
Why do we constantly have to consider both zero-based and one-based indexing when implementing mathematics?
Answer 5¶
Because the mathematical convention from linear algebra is one-based while much of modern software operates zero-based.
Question 6¶
Using zero-based indices, where is matrix entry $A_{m,n}$ in a row-major array a for an $M\times N$ matrix?
Answer 6¶
a[m * N + n]. Each preceding row contributes N entries.
Question 7¶
Where is $B_{n,p}$ in row-major array b for an $N\times P$ matrix, using zero-based indices?
Answer 7¶
b[n * P + p]. The row width of $B$ is P.
Question 8¶
An $M\times N$ matrix has N=4. What flat index corresponds to row 2, column 1, using zero-based indices?
Answer 8¶
2 * 4 + 1 = 9.
Question 9¶
How do you express the matrix product using only flat row-major arrays a, b, c?
Answer 9¶
c[m * P + p] = sum(a[m * N + n] * b[n * P + p] for n in range(N))
Assume c has space for M * P entries and m, p are valid zero-based row and column indices.
PyTorch Tensor Basics¶
13 basic questions.
Review: Lecture foundations.
Sources: BDS761_01_Intro, BDS761_04_PyTorch_01_Tensors
Question 1¶
In machine learning programming, what is a tensor?
Answer 1¶
A multidimensional array of numbers. A vector has one axis, a matrix has two, a color image has 3, and an image batch can have four.
Question 2¶
What does the shape (2, 3, 4) tell you?
Answer 2¶
There are three axes, with sizes 2, 3, and 4. The tensor contains $2\times3\times4=24$ values.
Question 3¶
What does a tensor's dtype describe?
Answer 3¶
The type used to store its values, such as torch.float32 or torch.int64.
Question 4¶
What does a tensor's device describe?
Answer 4¶
Where its data lives, such as CPU memory or GPU memory.
Question 5¶
How do you create a floating-point tensor with values [1, 2, 3]?
Answer 5¶
x = torch.tensor([1, 2, 3], dtype=torch.float32)
Question 6¶
Which creates known zeros: torch.zeros(2, 3) or torch.empty(2, 3)?
Answer 6¶
torch.zeros(2, 3). torch.empty allocates memory without initializing its values.
Question 7¶
How do you get the Python number from a one-element tensor x?
Answer 7¶
x.item().
Question 8¶
What does the underscore mean in x.add_(1)?
Answer 8¶
The operation changes x in place. Here it adds 1 to every entry of x. "in place" means it does not return a new copy but changes the input data itself.
Question 9¶
How do you choose a CUDA GPU when one is available, otherwise the CPU?
Answer 9¶
device = 'cuda' if torch.cuda.is_available() else 'cpu'
Question 10¶
How do you move tensor x to a chosen device?
Answer 10¶
x = x.to(device)
Question 11¶
What must be true about the devices of two tensors used in an ordinary matrix multiplication?
Answer 11¶
They must be on the same device. Their matrix dimensions and numeric types must also be compatible.
Question 12¶
Does torch.from_numpy(a) copy the values of an ordinary NumPy array a?
Answer 12¶
No. The tensor shares memory with a; changing an entry through one can change the other.
Question 13¶
What is a tensor's stride?
Answer 13¶
The number of storage entries you step over when an index increases by 1 along an axis, as in the computation m*N +n, where the stride used is (N,1). A contiguous row-major (3, 4) tensor has strides (4, 1).
Reshaping, Vectorization, and Kronecker Products¶
11 basic questions.
Review: HW 4.
Sources: BDS761_04_PyTorch_01_Tensors
Question 1¶
What must stay the same when you reshape an array?
Answer 1¶
The total number of values. Six values can form (2, 3) or (3, 2), but not (2, 4).
Question 2¶
What does x.reshape(-1) do?
Answer 2¶
It makes a one-dimensional array or tensor. The -1 means infer the required size.
Question 3¶
What does transposing a two-dimensional matrix do?
Answer 3¶
It exchanges rows and columns. A (2, 3) matrix becomes (3, 2).
Question 4¶
In NumPy, how is flatten() different from ravel()?
Answer 4¶
flatten() returns a copy. ravel() returns a view "when possible", meaning for some situations it might also return a copy.
Question 5¶
Can PyTorch reshape() always be assumed to return a view without copying?
Answer 5¶
No. It can return a view or make a copy. view() requires compatible strides; after permuting axes, reshape() is often easier to use.
Question 6¶
For A = [[1, 2], [3, 4]], what are the row-major and column-major flattened values?
Answer 6¶
Row-major: [1, 2, 3, 4]. Column-major: [1, 3, 2, 4].
Question 7¶
Which flattening order is used in the lecture identity $\operatorname{vec}(ABC)=(C^T\otimes A)\operatorname{vec}(B)$?
Answer 7¶
Column-major order: order='F' in NumPy. Ordinary row-major flattening uses a different identity.
Question 8¶
Using column-major order, how do you turn a matrix into a vector and a length-4 vector into a $2\times2$ matrix?
Answer 8¶
def vec(A):
return A.reshape(-1, order='F')
def mat(v):
return v.reshape(2, 2, order='F')
This simple mat example is for exactly four entries. For length $n^2$, use shape (n, n).
Question 9¶
What does the Kronecker product $A\otimes B$ do?
Answer 9¶
It replaces each entry $A_{ij}$ with a block $A_{ij}B$. In NumPy, use np.kron(A, B).
Question 10¶
What is the shape of the Kronecker product of a $2\times3$ matrix and a $4\times5$ matrix?
Answer 10¶
$(2\cdot4)\times(3\cdot5)=8\times15$.
Question 11¶
How can you check the Kronecker identity in NumPy for compatible matrices A, B, and C?
Answer 11¶
left = (A @ B @ C).reshape(-1, order='F')
right = np.kron(C.T, A) @ B.reshape(-1, order='F')
left and right should be very similar in value (to numerical precision)
Images, Batches, and Pixel Statistics¶
15 basic questions.
Review: HW 4, Quiz 5.
Sources: BDS761_04_PyTorch_01_Tensors
Question 1¶
For an RGB image with shape (427, 640, 3), what do the axes mean?
Answer 1¶
Height, width, and color channel. Each pixel has three values: red, green, and blue.
Question 2¶
How do you combine two same-shaped RGB images into a NumPy batch?
Answer 2¶
batch = np.stack([img1, img2], axis=0)
For images of shape (427, 640, 3), the batch shape is (2, 427, 640, 3).
Question 3¶
For shape (B, H, W, C), which axes do you average to get one mean per color channel across the whole batch?
Answer 3¶
channel_means = batch.mean(axis=(0, 1, 2))
This leaves the channel axis, so the result has shape (C,).
Question 4¶
How do you get one standard deviation per color channel across a NumPy image batch?
Answer 4¶
channel_stds = batch.std(axis=(0, 1, 2), ddof=0)
Convert integer image data to floating point before later arithmetic such as subtraction or scaling.
Question 5¶
For shape (B, H, W, C), which axes produce one mean per image and color channel?
Question 6¶
What does (B, C, H, W) mean in the lecture's PyTorch image example?
Answer 6¶
Batch size, color channels, height, and width. This ordering puts channels before spatial dimensions.
Question 7¶
How do you convert a PyTorch tensor x from shape (B, H, W, C) to (B, C, H, W) using permute?
Answer 7¶
x = x.permute(0, 3, 1, 2)
Question 8¶
How do you reshape 100 grayscale images of shape (100, 1024, 1024) into a matrix of 100 subjects versus 1024*1024 pixel features?
Answer 8¶
X = batch.reshape(100, -1)
The result has shape (100, 1048576): one row per subject and one column per pixel position.
Question 9¶
For that matrix, should standardizing each pixel separately apply over subjects or over pixel positions (rows or columns)?
Answer 9¶
Over subjects/columns: axis=0 in NumPy or dim=0 in PyTorch. Each pixel position needs its own mean and scale across the 100 subjects.
Question 10¶
What is the advantage of processing a billion images in batches of 100?
Answer 10¶
Only a small part of the dataset must be in working memory at once. The full dataset should stay in disk-backed storage or a data loader.
Question 11¶
How can you iterate over batches of at most 100 subjects from an indexable dataset images?
Answer 11¶
for start in range(0, len(images), 100):
batch = images[start:start + 100]
The final batch can be smaller than 100.
Question 12¶
How do you compute per-pixel population standard deviations from a centered batch matrix Xc using NumPy linear algebra?
Answer 12¶
sd = np.linalg.norm(Xc, axis=0, keepdims=True) / np.sqrt(Xc.shape[0])
Xc is floating point, has mean-zero columns, and has one row per subject.
Question 13¶
Putting the steps together: how do you standardize pixel columns within one NumPy batch using norms?
Answer 13¶
X = batch.reshape(len(batch), -1).astype(np.float32)
Xc = X - X.mean(axis=0, keepdims=True)
sd = np.linalg.norm(Xc, axis=0, keepdims=True) / np.sqrt(len(X))
Z = Xc / np.where(sd == 0, 1, sd)
Nonconstant pixel columns have mean approximately 0 and population standard deviation approximately 1.
Question 14¶
For a floating PyTorch matrix X with subjects in rows, how do you compute population standard deviations by feature?
Answer 14¶
sd = X.std(dim=0, correction=0, keepdim=True)
correction=0 divides variance by the number of subjects. It matches NumPy's ddof=0 convention.
Question 15¶
Does using new means and scales for each batch give the same transform as using one mean and scale for the whole dataset?
Answer 15¶
No, the mean and variance estimates will be based on the small batch rather than the whole dataset
Nearest Neighbors and Voting¶
14 basic questions.
Review: HW 5.
Sources: BDS761_03_Norms, BDS761_04_PyTorch_01_Tensors
Question 1¶
What does a 1-nearest-neighbor classifier predict?
Answer 1¶
The class on an input based on finding the class of the closest labeled training example.
Question 2¶
What does $k$ mean in k-nearest neighbors (kNN)?
Answer 2¶
The number of nearby training examples used to make a prediction. For example k=5 means use the 5 closest, and take the majority class.
Question 3¶
Should kNN choose the largest or smallest distances?
Answer 3¶
The smallest distances: these examples are nearest to the input features.
Question 4¶
For one query vector q and NumPy training matrix X_train, how do you compute one Euclidean distance per training row?
Answer 4¶
d = np.linalg.norm(X_train - q, axis=1)
axis=1 combines the feature differences within each subject row.
Question 5¶
For a one-dimensional NumPy distance array d, how do you get the indices of the five nearest examples?
Question 6¶
If idx contains nearest-neighbor indices, how do you obtain their labels?
Answer 6¶
neighbor_labels = y_train[idx]
y_train must be a NumPy array or PyTorch tensor paired with the training rows.
Question 7¶
The five nearest labels are [0, 1, 1, 0, 1]. What does their vote predict?
Answer 7¶
Class 1: it has three votes; class 0 has two.
Question 8¶
How do you select the most frequent label from a one-dimensional NumPy array neighbor_labels?
Answer 8¶
labels, counts = np.unique(neighbor_labels, return_counts=True)
prediction = labels[np.argmax(counts)]
If counts tie, this selects the first label in NumPy's sorted labels array.
Question 9¶
Can using an odd k, such as 5, prevent every possible class-vote tie?
Answer 9¶
No. It prevents a tie between two classes, but three classes can have vote counts 2, 2, and 1. Use a consistent tie rule.
Question 10¶
What is the shape of torch.cdist(Q, X_train) when Q.shape == (8, F) and X_train.shape == (100, F)?
Answer 10¶
(8, 100): one distance for each query/training-subject pair. Use compatible floating-point tensors on the same device.
Question 11¶
For a two-dimensional PyTorch distance matrix d, how do you get the five nearest indices for each query?
Answer 11¶
idx = torch.topk(d, k=5, dim=1, largest=False).indices
largest=False selects smallest distances. Assume at least five training subjects.
Question 12¶
For a PyTorch matrix neighbor_labels with one query per row, how do you take a class vote for each query?
Answer 12¶
predictions = torch.mode(neighbor_labels, dim=1).values
Each prediction is the most frequent label in that row. A tie requires a consistent rule.
Question 13¶
If you evaluate a training subject against the same training dataset, why should you exclude that subject itself?
Answer 13¶
Its distance to itself is zero. Including it lets the classifier use the very label it is supposed to predict.
Question 14¶
Why can feature scaling matter for kNN?
Answer 14¶
A feature with much larger numerical values can dominate the distance, even when other features are useful.
Image Patches for a Vision Transformer¶
11 basic questions.
Review: HW 5.
Sources: BDS761_04_PyTorch_01_Tensors
Question 1¶
What does "patchifying" an image mean?
Answer 1¶
Dividing it into small blocks and representing each block as a vector of pixel values.
Question 2¶
How many values are in a $16\times16$ RGB patch?
Answer 2¶
$16\times16\times3=768$.
Question 3¶
How many nonoverlapping $16\times16$ patches fit in a $224\times224$ image?
Answer 3¶
$14\times14=196$ patches.
Question 4¶
What must be true of image height and width for the simple nonoverlapping reshape approach with patch size P?
Answer 4¶
Both must be divisible by P. Otherwise, crop or pad the image first.
Question 5¶
What does a patch array with shape (B, N, D) represent?
Answer 5¶
B images, N patches per image, and D values in each flattened patch.
Question 6¶
Homework 5 wants colors as the innermost dimension. For two RGB pixels, what should the patch vector start with?
Answer 6¶
[r1, g1, b1, r2, g2, b2]: all colors for pixel 1, then all colors for pixel 2.
Question 7¶
What is the final shape for two $224\times224$ RGB images patchified into $16\times16$ blocks?
Answer 7¶
(2, 196, 768).
Question 8¶
Starting with PyTorch x of shape (B, C, H, W), how do you expose patch-grid and within-patch coordinates?
Answer 8¶
grid = x.reshape(B, C, H // P, P, W // P, P)
The axes are batch, color, patch row, pixel row within patch, patch column, pixel column within patch.
Question 9¶
For that grid, which permutation puts patch-grid coordinates first and colors last?
Answer 9¶
ordered = grid.permute(0, 2, 4, 3, 5, 1)
The new axes are batch, patch row, patch column, pixel row, pixel column, color. This matches Homework 5's RGB ordering.
Question 10¶
How do you flatten ordered into one vector per patch?
Answer 10¶
patches = ordered.reshape(B, (H // P) * (W // P), P * P * C)
Colors change fastest within each vector, giving [r1, g1, b1, r2, g2, b2, ...].
Question 11¶
For a NumPy RGB image img with shape (H, W, 3), how do you make a list of nonoverlapping patch vectors with colors innermost?
Answer 11¶
H, W, C = img.shape
grid = img.reshape(H // P, P, W // P, P, C)
ordered = grid.transpose(0, 2, 1, 3, 4)
patches = ordered.reshape(-1, P * P * C)
Here P is a positive patch size dividing H and W. Each row is one patch vector.