Typical characteristics of the problem

In general, the following are characteristics of the problem that I'm generally interested in:

1. \(N\) not too large - potentially under 100k.
2. \(\alpha\) in the ballpark of 0.1% to 5%.

These kinds of problems are typical in security, fraud prevention, medicine, and other situations of attempting to detect harmful anomalous behavior.

The effect of label shift on calibration

Let's study this from the perspective of score distributions. Suppose that \(f_0(z)\) is the pdf of the distribution \(z | y=0\) and \(f_1(z)\) is the pdf of the distribution \(z | y=1\). For simplicity, assume these distributions are monotonic.

Suppose now that the base rate is \(P(y=1)=\alpha\). In this framework, a label shift can be represented simply as a change in \(\alpha\).

It is straightforward to calculate the calibration curve (as a function \(\alpha\)) as:

As is apparent from this formula, a change in \(\alpha\) will result in a change in calibration. The following graph provides an example:

Illustration of calibration curves changing with base rate

Let's consider the effect of this on decisionmaking. Going back to our disease example above, suppose that at model training/calibration time, \(\alpha=0.01\). Then a disease outbreak occurs and \(\alpha=0.1\). The decision rule being used based on the training data (with \(\alpha=0.01\)) says to treat any patient with raw \(z\) score of 0.65 or greater.

But once \(\alpha=0.1\), the actual infection probability of a person with \(z=0.65\) is nearly 40%. As per the loss function calculation earlier, we want to treat any patient with a 5.26% or greater chance of being sick!

In the literature, when making batch predictions, there's a known technique for solving this (see discussion [2]). The basic idea is the following. For a set of raw risk scores \(z_i, i=1\ldots N\), we know they are drawn from the distribution:

Thus, one can estimate \(\alpha\) via the maximum likelihood principle (although the literature describes a slightly different approach [3]):

Maximizing this is straightforward - take logs, compute \(\frac{\partial L}{\partial \alpha}\), use scipy.optimize.minimize.

Formal statement of the setup

In the training phase, we have a labelled data set \((\vec{x}_i, y_i), i=1\ldots N\) on which we can train any sort of model that generates risk scores \(z_i, i=1 \ldots N\). We will assume that in this data set, the risk scores \(z_i\) are drawn from \(f_0(z)\) if \(y_i=0\) and \(f_1(z)\) if \(y_i=1\).

In the prediction phase we will consider batch predictions. We receive a new set of \(\vec{x}\) and we can of course use the already trained classifier to generate risk scores \(z_i\). Our goal is for each data point \(z_i\) to generate a calibrated risk score \(c(z_i) \approx P(y_i=1|z_i)\).

Without label shift there is a standard approach to this that is implemented in sklearn as sklearn.calibration.CalibratedClassifierCV. Typically this involves running isotonic regression on a subset of the training data and the mapping \(c(z)\) is the result of this.

That does not work in this case because \(c(z)\) computed in the training phase will be for the wrong distribution. The figure Illustration of calibration curves changing with base rate illustrates this - isotonic calibration may correctly fit the curve \(c(z; \alpha=0.01)\) in the training phase. But if the right curve in the prediction phase is \(c(z; \alpha=0.1)\), that fit is not actually correct. This blog post aims to address that problem.

My method

The approach I'm taking is upgrading the maximum likelihood estimation to a max-aposteriori estimation.

I first parameterize the shape of the exceptional label \(f_1(z;\vec{q})\) with \(\vec{q} \in \mathbb{R}^m\). I then construct a Bayesian prior on it which is clustered near \(f_1(z)\). It is straightforwardly derived from Bayes rule that x:

For simplicity I'm taking \(P(\alpha, \vec{q}) = P(\vec{q})\), a uniform prior on \(\alpha\).

Once the posterior is computed, we can replace maximum likelihood with max-aposteriori estimation. This provides a plausible point estimate for \((\alpha, \vec{q})\) which we can then use for calibration.

Kernel Density Estimation on [0,1]

The first step is doing kernel density estimation in 1-dimension in a manner that respects the domain of the function. Gaussian KDE does NOT fit the bill here because the support of a gaussian kernel is \(\mathbb{R}\), not \([0,1]\). One approach (which is somewhat technical and I couldn't make performant) is using beta-function KDE instead [4]. An additional technical challenge with using traditional KDE approaches on this problem is that whatever approach is taken, it also needs to be fit into a max-likelihood/max-aposteriori type method.

I took a simpler approach and simply used linear splines in a manner that's easy to work with in scipy. Suppose we have node points \(\zeta_0=0, \zeta_1, \ldots, \zeta_m=1\). Then let us define the distribution \(f_1(z; \vec{q})\) as a normal piecewise linear function:

for \(z \in [\zeta_k,\zeta_{k+1}]\) with \(h_i\) defined as

and

I chose this parameterization because scipy.optimize.minimize doesn't do constrained optimization very well. With this parameterization, all values \(\vec{q} \in \mathbb{R}^m\) yield a valid probability distribution on \([0,1]\).

Python code implementing this is available in the linked notebook, implemented as PiecewiseLinearKDE. Calculations of \(\nabla_{\vec{q}} h_i(\vec{q})\) - used in numerical optimization - can also be found in that notebook. Most of it is straightforward.

Fitting a piecewise linear distribution to data is only a few lines of code:

from scipy.optimize import minimize
def objective(q):
    p = PiecewiseLinearKDE(zz, q)
    return -1*np.log(p(z)+reg).sum() / len(z)

def deriv(q):
    p = PiecewiseLinearKDE(zz, q)
    return -1*p.grad_q(z) @ (1.0/(p(z)+reg)) / len(z)

result = minimize(objective, jac=deriv, x0=np.zeros(shape=(len(zeta)-1,)), method='tnc', tol=1e-6, options={'maxiter': 10000})
result = PiecewiseLinearKDE(zeta, result.x)

The result is approximately what one might expect.

One useful coding trick to take away from this is our use of np.interp inside a number of methods of PiecewiseLinearKDE. Since the curve itself is computed as np.interp(x, self.nodes, self.h()), gradients of this w.r.t. q can then then be computed by applying np.interp(x, self.nodes, grad_h) where grad_h is the gradient of \(\vec{h}\) w.r.t. \(\vec{q}\). This then allows the efficient calculation of gradients of likelihood functions as seen in deriv above, simplifying what might otherwise be index-heavy code.

Computing a posterior and max-aposteriori estimation

Defining a prior on a function space - e.g. the space of all probability distributions on [0,1] - is not a simple matter. However, once we've chosen a parameterization for \(f_1(z; \vec{q})\), it becomes straightforward. Since \(\vec{q} \in \mathbb{R}^m\), the restriction of any reasonable prior onto this space is absolutely continuous w.r.t. Lebesgue measure, thereby eliminating any theoretical concerns.

The situation we are attempting to model is a small to moderate change in the distribution of \(f_1(z)\), particularly in regions where \(f_0(z)\) is small. So we will define the (unnormalized) prior to be:

where \(g(x) = \sqrt{1+x^2}-1\) is a basically just a smoothed out (differentiable) version of \(|x|\). We need a smooth version of \(|x|\) simply because when we do max-aposteriori later, a smooth curve makes numerical minimization easier.

This prior should not be thought of as a principled Bayesian prior, but merely one chosen for convenience and because it regularizes the method. If we ignore the smoothing, this is analogous to a prior that penalizes deviation from \(f_1(z)\) in the \(L^p(f_0(z) dz)\) metric. The measure \(f_0(z) dz\) is used to penalize deviation more in areas where \(f_0(z)\) is large. The parameter \(\beta\) represents the strength of the prior - larger \(\beta\) means that \(f_1(z; \vec{q})\) will remain closer to \(f_1(z)\).

One important note about the power \(p\). Because \(g(x) = O(x^2)\) as \(x \rightarrow 0\), choosing \(p=1\) does NOT actually generate any kind of sparsity penalty, in contrast to using \(|x|^1\).

The likelihood is (as per the above):

Computing the log of likelihood times prior (neglecting the normalization term from Bayes rule), we obtain:

The gradient of this with respect to \((\alpha, \vec{q})\) is:

Using this objective function and gradient, it is straightforward to use scipy.optimize.minimize to simultaneously find both \(\vec{q}\) and \(\alpha\).

Examples

Note: All of the examples here are computed in this Jupyter notebook. For more specific details on how they were performed, the notebook is the place to look.

Here's an example. I took a distribution of 97.7% negative samples, with a relatively simple prior distribution. I simulated a significant change of shape in the distribution of \(z\) scores of the positive class, which is illustrated in red in the graph below. As can be seen, the approximation (the orange line) is reasonably good. Moreover, we recover \(\alpha\) with reasonable accuracy - the measured \(\alpha\) was 0.0225 while the true \(\alpha\) was 0.0234.

(The histograms in the graph illustrate the actual samples drawn.)

Using the fitted curve to compute calibration seems to work reasonably well, although simple isotonic regression is another way to do it.

The advantage of using this method is on out of sample data with a significantly different distribution of positive cases. I repeated this experiment, but with \(\alpha=0.011\) and a marginally different distribution of positive cases.

The dynamically calculated calibration curve (the green) still behaves well, while the isotonic fit calculated for a different \(\alpha\) (unsurprisingly) does not provide good calibration.

Note that recalculating the isotonic fit is not possible, since that requires outcome data which is not yet available.

Estimating Bayes loss

The major use case for this method of calibration is reducing the loss of a decision rule due to model miscalibration. Consider a loss function which penalizes false positives and false negatives. Without loss of generality [6], such a loss function takes this form:

With this loss function, the optimal decision rule is to choose 1 (positive) whenever \(c(z) >= T\), otherwise choose 0 (negative).

Using the same example as above, we can compute the result of applying this decision rule using either isotonic calibration (static) or our dynamic rule to the test set. For almost every choice of threshold \(T\), the loss is significantly lower when using the dynamic calibration.

Other metrics

A method such as this should NOT be expected to improve ROC_AUC, and in fact in empirical tests this method does not. This is because ROC_AUC is based primarily on ordering of risk scores, and our calibration rule does not change the ordering.

The Brier Score - an explicit metric of calibration - does tend to increase with this method. This is of course completely expected. In my experiments, this method is less effective at generating a low Brier score than Isotonic calibration at least until either \(\alpha\) or \(\vec{q}\) changes.

The average precision score also tends to increase over multiple batches with different \(\alpha, \vec{q}\).

Comparison to more standard label shift methods

Another approach (the approach of papers linked in footnote [2]) is to simply fit \(\alpha\) and do not allow \(f_1(z)\) to change.

In experiments, I've noticed that fitting \(\alpha\) without allowing \(f_1(z)\) to change generally produces a more accurate estimate of \(\alpha\), even in situations where the true distribution differs significantly from \(f_1(z)\).

However, in spite of a more accurate estimate of \(\alpha\), the resulting calibration curves from fitting only \(\alpha\) do not tend to be as accurate. The curve that comes from fitting \(\alpha, \vec{q}\) is more accurate than the fit of \(\alpha\) alone:

Active learning

As one additional note, I'll mention that I have some work (which I'll write about soon) suggesting that if we can request labels for a subset of the data points, we can do reasonably efficient active learning of calibration curves. This appears to significantly improve accuracy and reduce the number of samples needed.

Notes

Formalizing the theory

The first step in answering this question is to formalize the theory. The most straightforward way I can think of to do this is through the lens of net present value, albeit with a modified discount rate.

This framework says that the value of any cash generating asset is given by:

In this sum, \(R_t\) is the cash flow in time period \(t\) and \(d_t\) is the discount factor of time \(t\).

Here's a fairly simple example - a US treasury bill guaranteed to pay a $100 coupon for 3 periods and then to pay a final $10,000 in the 4'th. In tabular form:

To complete the calculation, we need to time discount each cash payment. This is typically done by taking the risk free interest rate - say 5% - and applying that to each time period. For example:

Finally, the value of the bond is the sum of the R*d column, which is $9928.12 in this example.

Long term investors appear short term

Lets now consider a long term investor who is evaluating a blue chip, highly stable stock. This stock regularly has earnings of $100. Then one quarter, it misses earnings and only reports $75!

An investor infected by short-termism will significantly cut their evaluation of the company - since d=0 for all future periods, the value drops from $100 to $75, a 25% decrease.

Let us now consider a long term investor.

Over 18 years, the value of this revenue stream works out to be $6498. In contrast, had earnings for one quarter not been missed, it would be $6523, a difference of 0.4%. Thus, if there is a drop in share price of significantly more than 0.4%, one might hypothesize that this is due to the market taking a short termist view.

Let us now consider a long term investor who actively tries to think through cause and effect. Earnings decreased, and there must be some reason for it! The question to ask is therefore whether this reduction in a single quarter's earnings will continue into the future. We encounter a situation similar to the pharma stock discussed earlier:

In the R_bad scenario, the company will only be worth $4892 (a 25% decrease from it's previous value).

If the long term investor believes that there is a 40% chance of this occurring, then the value of the stock decreases to $5855.75, a 10% drop!

Even though the long term investor doesn't care much about a single quarter's earnings, he cares a lot about whether this predicts many more quarters of reduced earnings. This means that even long term investors behave in the manner that others describe as "short-termist".

As a result, both the short-termism theory and the long-termism theory make very similar predictions. The fact that stock prices move significantly in response to missed earnings estimates is insufficient to distinguish between these two theories.

Coronavirus as a natural experiment

Coronavirus provides a great natural experiment to help disambiguate test this theory. A large number of stocks are going to have several quarters of significantly reduced earnings, but there is very little uncertainty as to the reason.

We know that right now people don't go to $SBUX, $MCD or $CMG. This will be a big hit to their earnings. But on the flip side we know exactly why - people haven't suddenly decided they dislike Big Macs, they just don't want to catch coronavirus.

Meanwhile, we're being treated to articles like The Great Divide Between Stocks & The Economy which discuss a significant deviation between stock market performance and short term economic indicators.

One thing is pretty clear: the next couple of quarters of earnings are going to suck. But unlike many earnings misses, we know exactly why. And for most blue chip stocks in the S&P 500, we have great reason to believe that this is strictly temporary, and once coronavirus is dealt with things will go back to normal.

SPY over time.

Although SPY is nowadays driven predominantly by technology companies, even brick and mortar companies that are likely to survive the crisis have a similar share price: This is even true of solid brick and mortar companies such as McDonald's or Starbucks.

MCD over time.

SBUX over time.

If shareholders only care about the next quarter, there is no case to make that $SBUX is worth holding.

However, if shareholders are looking to the long term, then $SBUX and $MCD are pretty solid stocks to hold. If investors have a low discount rate, the impact of the next couple of quarters on LTV is not that big. As long as investors are convinced that there actually is a long term for the company, $SBUX and $MCD remain solid investments. The calculus of a long term investor is exactly as described before:

They may not be quite as large after the crisis as before - some locations will certainly close - but overall the revenue stream is likely to continue.

In contrast, other stocks such as $CCL (Carnival Cruise Lines) do not have such a rosy outlook. Unlike $SBUX, $CCL may simply go bankrupt - in that case, shareholders get nothing.

CCL over time.

$CCL is has a very uncertain future. It is likely that it's business will remain solid if it can survive through the crisis - it's August cruises are already fully booked. However, it is not clear that $CCL can actually survive the crisis without going through bankruptcy first. Ships are expensive to store and CCL has financing costs, and it's not clear it can do this without incoming revenue.

This makes it's valuation look a lot more uncertain, with both a good and bad outcome:

Thus, the stock price of $CCL is remaining low because it is unclear that it ever will come back. There is a high probability assigned to the bad outcome and thus the LTV is low.

All about liquidity injections

If I had a comment section, I would fully expect someone to talk about how this is all invalid because of the Fed.

But this betrays a fundamental misunderstanding of accounting. When "money printer go brrr", the actual mechanism by which this happens is that new debt is issued at favorable terms. Taking out a loan is not earnings so the short term forecast for a company impacted by Coronavirus is unchanged. A short term investor will be completely unmoved by this!

The money printer does have one very important effect - it increases the odds significantly that a company will survive the crisis and have a good long term returns. The next couple of quarters will suck, but the long term value of the company remains.

The Short-Termism theory has died of COVID

Coronavirus is a great natural experiment for a lot of things.

One of the most important things we can take away from it is the conclusion that equity markets are fundamentally focused on the long term value of the companies being traded. There are fast responses to problems with next quarter earnings, but these are primarily driven by the fact that problems in the short run tend to be indicative of more fundamental issues.

Now that we have a systematic example where we know that short run problems are strictly short run, we can safely disambiguate between short termism and long termism. The result is very clear; the market is predominantly focused on the long term.

Disclosure: Long $SBUX, $CCL.

Mathematically modeling a single test

To model this problem in the frequentist hypothesis testing framework, let us assume we have a continuous probability distributions \(f\) which is supported on \([0,\infty)\). As our null hypothesis - i.e. nothing unusual to report - we'll assume this distribution is absolutely continuous with respect to Lebesgue measure and that it has pdf \(f(x) dx\) which is monotonically decreasing, i.e. \(f(y) \leq f(x)\) for \(y \geq x\) (almost everywhere).

In contrast, for the alternative hypothesis - something worth flagging as potentially bad - I'll assume that the distribution is a mixture distribution with pdf \((1-\beta) f(x) + \beta s(x) dx\). Here \(s(x)\) is monotonically increasing, or more typically \(s(x) = \delta(x-x_0)\).

Observation: Consider a probability distribution \(f(x)\) that is monotonically decreasing. Then the cumulative distribution function \(F(x)=\int_0^x f(t) dt\) is concave. This can be proven by noting that it's derivative, \(F'(x) = f(x)\) is monotonically decreasing.

Our hypothesis test for distinguishing between the null and alternative hypothesis will be based on concavity. Specifically, if there are spikes in a histogram of the pdf of a distribution, then it's CDF may cease to be concave at the point of the spike. Here's an illustration. First, consider the empirical CDF of a distribution which is monotonically decreasing:

This graph is clearly concave. The red line illustrates a chord which must, by concavity, remain below the actual curve.

In contrast, a pdf with a spike in it will fail to be concave near the spike. Here's an illustration:

At \(x=10\) the chord (the red line) is above the graph of the CDF (the green line).

In mathematical terms, concavity of the true CDF can be expressed as the relation:

or equivalently:

Since we do not know \(F(x)\) exactly, we of course cannot measure this directly. But given a sample, we can construct the empirical CDF which is nearly as good:

Using the empirical CDF and the definition of concavity suggests a test statistic which we can use:

Our goal is to show that if this test statistic is sufficiently negative, then a spike must exist.

When \(q\) becomes negative, this shows that \(F_n(x)\) is non-concave. However, the empirical distribution function is by definition non-concave, as can be seen clearly when we zoom in:

Mathematically we can also see this simply by noting that \(1_{x \geq x_i}\) is not concave. However, this non-concavity has order of magnitude \(O(n^{-1})\), so to deal with this we can simply demand that \(q < -1/n\).

There is a larger problem caused - potentially - by deviation between the empirical distribution \(F_n(x)\) and the true, continuous and concave cdf \(F(x)\). This however can also be controlled and will be controlled in the next section.

Conclusion

At the moment this test is not all I was hoping for. It's quite versatile, in the sense of being fully nonparametric and assuming little beyond the underlying distribution being monotone decreasing. But while theoretically the convergence is what one would expect, in practice the constants involved are large. I can only detect spikes in histograms after they've become significantly larger than I'd otherwise like.

However, it's still certainly better than nothing. This method would have worked in several of the practical examples I described at the beginning and would have flagged issues earlier than than I detected them via manual processes. I do believe this method is worth adding to suites of automated anomaly detection. But if anyone can think of ways to improve this method, I'd love to hear about them.

I've searched, but haven't found a lot of papers on this. One of the closest related ones is Multiscale Testing of Qualitative Hypotheses.

Isotonic regression for binary data

Another issue facing the standard isotonic regression model is binary data - where y in [0,1]. Using RMS on binary data sometimes works (when there's lots of data and it's mean is far from 0 and 1), but it's far from optimal.

For this reason I wrote a class isotonic.BinomialIsotonicRegression which handles isotonic regression for the case of a binomial loss function.

As is apparent from the figure, this generates more plausible results for binary isotonic regression (in a case with relatively few samples) than the standard sklearn package. The result is most pronounced at the endpoints where data is scarcest.

Code is available

You can find the code on my github. It's pretty alpha at this time, so don't expect it to be perfect. Nevertheless, I'm currently using it in production code, in particular a trading strategy where the noise sensitivity of sklearn.isotonic.IsotonicRegression was causing me problems. So while I don't guarantee it as being fit for any particular purpose, I'm gambling :code:O($25,000) on it every week or two.

Appendix: Mathematical Details

This appendix explains the mathematical details of the methods, as well as technical details of the parameterization. It is mainly intended to be used as a reference when understanding the code.

The package uses maximum likelihood for curve estimation, and uses the Conjugate Gradient method (as implemented in scipy.optimize.minimize) to actually compute this maximum.

The basic math of a single gamble

Suppose we have the opportunity to make a gamble. If we make the gamble, we have a probability p of winning. If we win the gamble we earn a payoff of w, whereas if we lose we earn a payoff of l.

For example, the gamble might be "attend Lambda School", "go to college" or "buy clicks via google adwords which may or may not buy a product from our landing page". The payoff of a win (w) would be a higher paying job or a sale on a website. The loss of losing (l) would be the time and money spent on tuition or the money paid to Google for the click.

It's very easy to compute the expected value of this gamble is:

E[gamble] = p*w + (1-p)*l

Assuming no risk aversion - fairly typical when w and l are not that large - we should take this gamble whenever E[gamble] > 0. Doing some simple arithmetic, we discover that this gamble is positive sum whenever:

w/l > (1-p)/p

(If this is a repeated game we should use the Kelly Criterion or some similar rule. I'm going to focus on the non-repeated case in this post.)

Another useful rearrangement is we break even when p = l/(w+l).

Sounds simple, right?

It's important to note that there are 3 different ways we can alter the parameters to make a gamble worthwhile:

• Increase w - make the payoff more attractive.
• Increase p - make a winning bet more likely.
• Decrease l - make the cost of losing lower.

The latter case is actually pretty important; consider the limiting case of l == 0. In this case, there is a finite probability p of winning something, and even if you don't win you lose nothing. That's a good bet to take regardless of how low p is!

So far, nothing groundbreaking, right?

The basic math of a sorting rule

Lets suppose now that we have multiple gambles we can choose to enter or not - say N` gambles, with ``i <- 1..N representing a particular gamble. Moreover, for each gamble, we know the success probability p[i] of that particular gamble.

For example, we might be choosing which students to admit to Lambda School or Rutgers.

As another example, we might be choosing which adwords (and landing pages matched to the adwords) we wish to bid on for CPC.

We are now faced with the task of choosing which gambles we want to take. Armed with the decision rule we came up with above we have a simple way to decide this: if w/l > (1-p[i])/p[i] we will accept gamble i.

You can never raise your conversion rate

Here's an example that someone at a major travel website told me about. The customer acquisition funnel consists of paying Google for adwords on a cost-per-click basis. Once people reach the site, a certain fraction of them make a purchase.

Suppose we have several different ad channels, i.e. keywords that can be bid on. When a customer converts the profit is $50.

The channels available are: - A, with a CPC of $2 and a conversion rate of 10%. The cost per conversion here is $2/0.1 = $20, meaning the profit is $30 for people coming from this channel. - B, with a CPC of $1 and a conversion rate of 3%. The cost per conversion here is $33, meaning the profit is $16 for people coming from this channel. - C, with a CPC of $1 and a conversion rate of 1.5%. The cost per conversion here is $66, making this channel money losing.

Assuming these channels provide equal traffic, bids will be placed on channels A and B only. C is unprofitable so no bids are placed.

Now suppose the price of adwords falls by 30%; channel A now costs $1.4/click while B and C cost $0.7/click. At this point the cost per conversion of channel C has dropped to $0.7/0.015 = $46.67 < $50. At this time channel C gets switched on.

However, the net result of this is that the conversion rate has dropped from (10% + 3%)/2=6.5% to (10%+3%+1.5%)/3 = 4.8%. Profit margins have dropped too. The net result here is that traffic has increased by 50%, but the new 50% are low margin, barely profitable users with low conversion rates. These new users drag the average down even as they add profits to the bottom line.

On the flip side of this phenomenon, it's very difficult to raise conversion rates. Imagine a sitewide improvement that results in the conversion rate on every channel increasing by 40% (but prices remained the same). This raises the conversion rate of channel C to 2.1%, lowering the cost per conversion to $47.61 and making it profitable.

The average conversion rate goes to (14%+4.2%+2.1%)/3 = 6.76%, a mere 4% improvement over the old conversion rate of (10%+3%)/2=6.5%. If Channel C were twice the size of channel A, then the conversion rate would actually go down to (14%+4.2%+2x2.1%)/4 = 5.6%!

On the other hand it's important to look at aggregate numbers. In these scenarios, improving conversion rates has increased sales by 50-100%. The additional sales have significantly lower margin, but it's still an increase in profits.

Why Indian software developers are lower quality than Western ones

I've made this argument elsewhere in more detail, but I'll give a short summary here.

The cost of an American developer is approximately 3-10x the cost of a similar Indian developer. In my experience (I lived in India for many years), there are plenty of Indian developers and data scientists just as good as any in the US. (If you are seeking employees, I know of a good company that everyone is leaving from. I can connect you to people.)

Yet the reputation of Indian developers among Americans is quite low. People commonly talk about outsourced projects run by teams of idiots, and have generally strong criticisms of Indian developers in general.

I never had this experience, but I also subject anyone I hire in India to the same level of hiring rigor that I subject Americans to.

We can reconcile these two views simply by observing that as you lower the cost of a failure (e.g. making a bad hire), it is economically optimal to allow a higher rate of failure. This means that Indian companies competing with American companies should have a lower hiring bar, by the same logic as above. This will drag the average quality of Indian developers down; similarly low quality American developers would simply not get hired as developers.

Thus, the Indian software engineering market has lower quality simply because low costs have allowed a larger pool of people to enter.

(Note: I'm singling out India only because I lived and worked there for many years.)

Conclusion

The interplay between individual success probabilities, cost of failure, and aggregate success probabilities is complex. It is simply innumerate to say, as Austin Allred did, that "We will never be ok with 50% [success rate]." As the cost of failure drops, as it does with Lambda School, the socially optimal success rate also drops.

(To be clear, I believe @austen probably knows this and is merely attempting to placate innumerate journalists.)

Simple economics tells us that the lower the cost of failure, the more failed attempts should be made. When the cost of failure goes down, any critique of reduced success rates is hopelessly innumerate.

Structure of the strategy

The trading strategy I'm using is pretty straightforward. To begin with, I periodically query the REST API of a certain marketplace. The API returns a list of trading opportunities - securities \(S_i, i=1 \ldots N\) and their prices \(p_i\), along with characteristics \(\vec{x}^i\) of the security.

As a concrete example of systems like this, think of trading cryptocurrencies on an exchange (e.g. Coinbase Exchange API).

The price in question is an implicit market-based assessment of risk - i.e. there is a function \(r(p_i)\) which implicitly assigns each price to a corresponding risk level. Higher prices imply more risk.

My mental model for the market is as follows. The market is a machine learning model \(L(\vec{x}_i)\) which predicts risk, and then chooses a corresponding price for that risk. I do not know this model, however a very simple application of isotonic regression has enabled me to determine that the market prices are highly predictive (ROC_AUC is in the 70-80% ballpark).

I have additional data \(\vec{y}^i\) that I do not believe the market incorporates. So the way I'm attacking the problem is the following:

1. Query the REST API to get the standard market data, \((p_i, \vec{x}^i)\).
2. Compute an inner join between my data set to enrich the data set, \((p_i, \vec{x}^i) \mapsto (p_i, \vec{x}^i, \vec{y}^i)\).
3. Run a machine learning model on the enriched data set and generate improved risk scores \(q(p_i, \vec{x}^i, \vec{y}^i)\). In backtesting, these improved risk scores are more accurate than the risk scores \(r(p_i)\) generated from prices alone.
4. Purchase securities which have \(q(p_i, \vec{x}^i, \vec{y}^i) - r(p_i) < -T\) for some threshold \(T\). In simple terms, I'm purchasing securities when my assessment of their risk is significantly lower than the market's assessment of their risk.

The ML model used to predict prices is a relatively standard sklearn.pipeline.Pipeline - it's trained offline, saved as a pickle, and loaded periodically to execute step (4) above.

Azure Functions seemed like a simple and low cost way to run this strategy - no server deployment to worry about.

Setting up Azure Functions

The general structure of my application is as follows:

requirements.txt
trading_library/data/... # Data access methods
trading_library/models/... # Code to actually train/run the ML models
trading_library/config/__init__.py
trading_library/config/_fileconfig.py  # Config when running things locally. This has secrets!
trading_library/jobs/...  # Functions that do things

az_func/host.json
az_func/local.settings.json
az_func/get_data/function.json  # Each subfolder corresponds to a single Azure function.
az_func/get_data/__init__.py
az_func/...other_job.../{function.json, __init__.py}

deploy_az_functions.ps1

The first directory, trading_library is just ordinary python code. It's a library with assorted functions that are helpful for running my trading strategy. Some are intended to run in production, others I use locally when doing analysis and development. This directory is at the top level because I frequently run a Jupyter notebook here for development.

The directory az_func corresponds to the Azure Functions application. It was created as follows:

PS ..> func init az_func --python

The notation PS ..> means that this line represents a PowerShell command. Tangentially, as a long time Linux command line user, I must say that PowerShell is the most exciting innovation in command line UI that I've seen since zsh. If you haven't tried it yet, go check it out.

This is the command to initialize an Azure Functions project as described in Microsoft's docs. Each subfolder in az_func corresponds to a single function - think of each of these as a cron job. (In other setups, they can correspond to an HTTP endpoint or something similar.)

The contents are quite simple. Here is function.json:

{
  "scriptFile": "__init__.py",
  "bindings": [
      {
          "name": "my_function_argument",
          "type": "timerTrigger",
          "direction": "in",
          "schedule": "3 25 15 * * *",
          "runOnStartup": false
      }
  ]
}

The schedule parameter is in ordinary 6 option CRON format - the example above runs on the 3'rd second of the 25'th minute of the 15'th hour of every day.

The python code in __init__.py is also quite simple:

import logging
import azure.functions as azfunc
from __app__.trading_library.jobs import get_the_data

def main(my_function_argument):
    logging.info('Running get_data at time %s.', timer)
    get_the_data()  # Real work happens here
    logging.info("Finished get_data.")

Note how the parameter "name": "my_function_argument" in function_json corresponds to the argument main(my_function_argument) in python itself. The function won't work if you don't get this right.

As far as local code structure, that's basically everything you need to create an Azure function.

$shared_code_path = "az_func\trading_library"

if (Test-Path $shared_code_path){
    Remove-Item -Recurse -Force $shared_code_path
}

# Cleanup crud
Get-ChildItem .\trading_library\ -recurse -include __pycache__ | Remove-Item -recurse
Get-ChildItem .\trading_library\ -recurse -include *.pyc | Remove-Item
Get-ChildItem .\trading_library\ -recurse -include *.py~ | Remove-Item

# Copy shared code
Copy-Item -Recurse .\trading_library\ $shared_code_path\

Remove-Item .\az_func\trading_library\config\_fileconfig.py

# Publish the function
cd az_func
func azure functionapp publish trading_library
cd ..

if (Test-Path $shared_code_path){
    Remove-Item -Recurse -Force $shared_code_path
}

PS..> az keyvault create -n tradinglibrarykeyvault -g $resourceGroup

$service_principal = az ad sp create-for-rbac -n "http://mySP" --sdk-auth | ConvertFrom-Json

PS..> az keyvault secret set -n "keyvault-client-id" --vault-name $keyvaultname --value $($service_principal.clientId)
PS..> az keyvault secret set -n "keyvault-client-secret" --vault-name $keyvaultname --value $($service_principal.clientSecret)

# Grant keyvault access
$kv_identity = az functionapp identity assign -n trading_library -g trading_library_resource_group | ConvertFrom-Json # First create identity

az keyvault secret set -n "keyvault-tenant-id" --vault-name $keyvaultname --value $kv_identity.tenantId
az keyvault secret set -n "keyvault-name" --vault-name $keyvaultname --value $keyvaultname

az keyvault set-policy -n trading_library_keyvault -g $resourceGroup --object-id "$($kv_identity.principalId)" --secret-permissions get  # Assign the policy

foreach ($keyname in "keyvault-tenant-id","keyvault-name","keyvault-client-id","keyvault-client-secret") {  # Then provide the keyvault variables.
    $secretId = az keyvault secret show -n $keyname --vault-name lendingclubstratkeyvault --query "id"
    $keyname_var= $keyname.replace('-', '_')
    az functionapp config appsettings set -n lendingclubstrat -g lendingclubstrat --settings "$keyname_var=@Microsoft.KeyVault(SecretUri=$secretId^^)"
}

from trading_library.config import cfg

do_a_thing(cfg['trading_api_key'])

import config
import os

__all__=['cfg']

try:
    from ._fileconfig import _fileconfig
    filecfg = config.config_from_dict(_fileconfig)
except ImportError:
    filecfg = config.config_from_dict({})

if 'keyvault_name' in os.environ:  # On azure, we will have access to keyvault.
    from config.contrib.azure import AzureKeyVaultConfiguration

    azure_cfg = AzureKeyVaultConfiguration(
        az_client_id=os.environ['keyvault_client_id'],
        az_client_secret=os.environ['keyvault_client_secret'],
        az_tenant_id=os.environ['keyvault_tenant_id'],
        az_vault_name=os.environ['keyvault_name']
        )
    cfg = config.ConfigurationSet(filecfg, azure_cfg)  # Config looks to
else:
    cfg = filecfg

Other problems I ran into

import sys
import logging
sys.modules['my_copy_of_logging'] = logging
import my_copy_of_logging

assert (my_copy_of_logging == logging)  # returns True

import __app__.trading_library
import sys
sys.modules['trading_library'] = __app__.trading_library

Uploading built content /home/site/deployments/functionappartifact.squashfs -> https://trading_strategy.blob.core.windows.net/scm-releases/scm-latest-trading_strategy.zip?...
Remote build failed!

What's a "model"?

Fundamentally, a model is a piece of code that attempts to predict the future based on past data. In many data science applications this means that the model will be something complex; a deep neural network, a gradient boosting machine, or even a linear regressor. In this case our model is something simpler - simple a hash map endpoint_url -> rate_limit. Just as the model is simple, the process computing it is also quite easy. The most obvious way to do it would be to simply open up Jupyter, run the analysis once, and then hardcode the result into the production system.

The danger in simply doing this is that the future may not simply be a repeat of the past. Our goal is to rate limit the API in such a way as to minimize the impact on future non-malicious users. We are attempting to predict the future, specifically future usage of the API.

Just like any other data science model, we should therefore run a backtest to determine if our predictions would have worked well in the past had we used them.

SELECT endpoint_url,
       PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY daily_requests) OVER (PARTITION BY endpoint_url) AS num_requests_99pct
  FROM endpoint_usage
    WHERE (NOT malicious)
          AND (date < '2019-11-01')  -- This part is new
  GROUP BY endpoint_url

SELECT endpoint_url, COUNT(*) AS total_usage,
       SUM(CASE WHEN (daily_requests > num_requests_99pct) THEN 1 ELSE 0) AS num_rate_limited,
  FROM endpoint_usage
    INNER JOIN rate_limits ON (endpoint_usage.endpoint_url = rate_limits.endpoint_url)
  WHERE (NOT malicious)
        AND (date >= '2019-11-01') AND (date < '2019-12-01')
  GROUP BY endpoint_url

WITH rate_limits AS (SELECT endpoint_url,
                            PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY daily_requests) OVER (PARTITION BY endpoint_url) AS num_requests_99pct
                       FROM endpoint_usage
                         WHERE (NOT malicious)
                               AND (date < '2019-11-01')
                     GROUP BY endpoint_url)


SELECT endpoint_url, COUNT(*) AS total_usage,
      SUM(CASE WHEN (daily_requests > num_requests_99pct) THEN 1 ELSE 0) AS num_rate_limited,
  FROM endpoint_usage
    INNER JOIN rate_limits ON (endpoint_usage.endpoint_url = rate_limits.endpoint_url)
  WHERE (NOT malicious)
        AND (date >= '2019-11-01') AND (date < '2019-12-01')
  GROUP BY endpoint_url

Putting it into production

Instead of simply running this SQL query once and hard coding the result, we can have a much more robust system with a moderate amount of effort. We'll follow the exact same steps as putting a data science model into production:

1. Set up a cron job (or better, an Airflow dag to run the query (i.e. train the model) every month.
2. Set up a second cron job to run the backtest every month. This means that on Jan 1, the code should generate rate limits using data available up to Dec 1. Then it should check how many users exceeded those rate limits during Dec 1-Dec 31. The results of this should be posted to a slack channel or monthly email.
3. Ensure proper alerting. If the API usage in redshift drops below some expected level, raise an alert - the model might be broken due to insufficient data (or data collection might simply be broken). Similarly, if the number of non-malicious users getting rate limited exceeds an expected threshold (e.g. 3-5%) then an alert should be raised. Models are finicky things and sanity checks are needed.

(Obviously adjust "monthly" to whatever makes sense for your use case.)

Ultimately, the idea of running a train/test split on historical data and auto-update your parameters is a very powerful paradigm. It is normally applied to complex data science models but even many simple tasks can benefit from this process as well.

Empirical evidence: 5 North American Experiments

Let me begin with the empirical evidence. There have been 5 experiments on either Basic Income or Negative Income Tax in North America. I'll discuss each of them in turn, focusing on the labor force effects.

Theory: Why would this be true?

The common justification for why a Basic Income would not reduce labor supply is the following. Because a BI is given regardless of work, a person receiving a BI gains the same amount of money from working as they would gain if they did not work. This is often contrasted to means-tested welfare, which often has high implicit marginal tax rates due to the withdrawal of welfare benefits.

However, this verbal analysis ignores something very important: diminishing marginal utility.

In economics, people are modeled as making decisions based on utility - roughly speaking, the happiness you get from something - not on cash. And an important stylized fact, accepted by pretty much everyone, is that utility as a function of income is strictly concave down. In mathematical terms, that means that for any \(0 \leq \alpha \leq 1\):

Since more income is always better, we can also assume that \(U(I)\) is a strictly increasing function of income.

In pictures, this means that a person's utility function looks like this:

Now the choice to work is made by balancing the utility gained from income against the disutility from working:

Since the net utility is positive, this person will choose to work.

However, because the utility function is concave, if we start from a point further out (namely \(I_b\)), the utility gain from labor decreases. This can be illustrated in the following graph:

In the Basic Income regime, a person's utility gain from working is only \(U(I_b+I_w) - U(I_b)\), which is lower than \(U(I_w)-U(0)\).

In some cases, this decrease will result in the net utility gain from work being negative:

These are the people who are deterred from working.

Now the graphs I've given above are just an example. The clever reader might ask if it is true for every graph. I will prove in the appendix that a Basic Income always reduces the marginal utility from work if diminishing marginal utility is true.

Conclusion

How big is this affect? Journalists favorable to a Basic Income tend to talk about "only" a 10% drop in labor supply. Let me make an invidious comparison.

In 2008, the United States (and the world) suffered the Great Recession. To make a comparison, I've plotted the male employment to population ratio (approximated by taking the male civilian employment rate and dividing it by half of the US population) at the time of the great recession.

What would have happened if the Great Recession didn't occur, but we instead instituted a Basic Income in 2008?

To speculate about this, I assumed a baseline employment to population rate of 52% for men (the peak employment rate just before the recession). I then plotted for comparison the results of several Basic Income experiments focusing on the effects on men (though in a couple of cases that was not well disambiguated).

In the case of Seattle/Denver, I plotted the effect observed in each year. In the other cases, where yearly effects were not reported, I merely assumed a drop equal to the average reported drop.

The result can be seen above. The typical effects of a Basic Income are in the same ballpark as those of the Great Recession.

The conclusion we can draw from this is that all the available evidence suggests that a Basic Income will have a very large and negative effect on the economy.

We can also anticipate that the effect will be worse if people believe that a Basic Income is likely to be permanent. As can be seen by comparing the 3 and 5 year groups in the Seattle/Denver experiment, people assigned to a longer term BI reduced their work effort significantly more than those assigned to the short term BI.

Updates: Confirming the theory with out-of-time data!

Update on Keynesian Economics

A few commenters on reddit suggest that unemployment due to "low aggregate demand" is somehow different from reduced labor force participation due to Basic Income. However, this idea is based on either MMT or some weird newspaper columnist pop-Keynesianism; it is not in any way based on the economic mainstream.

Mainstream economist version of Keynesian theory says that in a recession, people do not work because they have sticky nominal wages but a shock has resulted in their real output dropping. Concretely, a worker has a nominal wage demand of \(W\) dollars. He used to produce \(K\) widgets at a cost of \(W/K\) each, but for whatever reason he can now only produce \(K_2 < K\) widgets. His real output is now \(K_2\) which has nominal value \(K_2 (W/K) = (K_2/K) W < W\).

In order to productively employ him the nominal wage must be reduced by a factor of \(K_2/K\). However, the worker refuses to work unless he is paid \(W\) dollars.

The Keynesian prescription of stimulating aggregate demand solves this problem by inflation; if the price of a single widget can be increased by a factor of \(K/K_2\), then the worker can again be paid a wage of \(W\) dollars.

In essense, stimulating aggregate demand is about tricking prideful workers into reducing their real wage demands so that they stop being lazy and go back to work.

In spite of the difference in mood affiliation, Keynesian economics claims that recessions reduce labor supply for the exact same reason a Basic Income does: workers refusing to work.

Appendix: Proof that Diminishing Marginal Utility implies a work disincentive from Basic Income

The choice to work can be framed as a question of utility maximization. Assuming one receives an income of \(I_w\) from work and an income of \(I_b\) from Basic Income, the utility of working is:

While the utility of not working is

Here \(U_w\) is the utility penalty that describes the unpleasantness of work. Let us define \(\Delta U_{bi}\) as the marginal utility gained by making the choice to work in a Basic Income regime:

In contrast, the marginal utility gained or lost from work in a non-Basic Income regime is:

The concavity relation (setting \(\alpha=I/(I_b+I_w)\)) tells us that for any \(0 \leq I \leq I_b+I_w\), we have:

Now if we compute the difference \(\Delta U_{bi} - \Delta U\), we discover:

If we substitute \(I=I_b\) and \(I=I_w\) into the concavity relation above, we discover:

Therefore:

This completes the proof that if Diminishing Marginal Utility is true, a Basic Income reduces the incentive to work.

Cool python trick #1: pandas.read_html

This is one of the coolest tricks I've learned in 2019. The pandas library has a method, read_html which takes a webpage as input and returns a list of dataframes containing the tables on that webpage.

So to answer the question about terrorism in 2017, I'm going to browse Wikipedia's List of Terrorist Incidents in 2017.

Sadly, there's a lot of terror attacks, so they have separate pages for each month. Each page looks like this: https://en.wikipedia.org/wiki/List_of_terrorist_incidents_in_January_2017.

Therefore, to extract the data I'll do this:

import pandas
def load_month(m):
    results = pandas.read_html('https://en.wikipedia.org/wiki/List_of_terrorist_incidents_in_' + m + '_2017')
    df = results[0]
    df.columns = df[0:1].values[0]
    return df[1:].copy()
data = []
for month in [datetime(2008, i, 1).strftime('%B') for i in range(1,13)]:
    data.append(load_month(month))
data = pandas.concat(data)

The function read_html is doing all the heavy lifting here.

The result of this is a dataframe listing a location, a perpetrator, a number of deaths/injuries, and a few more columns. It's not super clean, but at least it's pretty structured.

This read_html function is awesome because I needed to do literally no work parsing.

Cool python trick #2: the wikipedia module

In this data, there were 230 separate perpetrators listed after cleaning up some of the obvious data issues (e.g. some rows containing "Al Shabaab" and others containing "Al-Shabaab"). That's far too much for me to manually classify everything.

So instead I used the wikipedia module.:

import wikipedia

def get_summary(x):
    if x == 'Unknown':
        return None
    try:
        return wikipedia.page(x).summary
    except Exception:
        return None

count['perp_summary'] = count['perpetrator_cleaned'].apply(get_summary)

This gets me a summary of each terrorist group, assuming wikipedia can easily find it. For example, here's the result of get_summary('Al-shabaab'):

'Harakat al-Shabaab al-Mujahideen, more commonly known as al-Shabaab (lit. '"The Youth" or "The Youngsters", but can be translated as "The Guys"'), is a jihadist fundamentalist group based in East Africa. In 2012, it pledged allegiance to the militant Islamist organization Al-Qaeda.[...a bunch more...]

With a little bit of string matching (e.g. if the summary contains "Communist" or "Marxist", classify as "Communist"), I was able to classify assorted terrorist attacks into a few broad causes:

cause                       dead
Islam                       8170.0
Central Africal Republic    432.0
Communism                   310.0
Myanmar                     105.0
Congo                       85.0
Anarchy                     3.0
Far-right                   3.0
Far-left                    1.0

Some of these are broad catch-all terms that simply reflect my ignorance. For example, one group is called Anti-Balaka, and wikipedia explains them as a predominantly Christian rebel group "anti-balakas are therefore the bearers of grigris meant to stop Kalashnikov bullets". I lumped a bunch of similar groups into "Central African Republic", similarly for "Congo" and "Myanmar".

This classification scheme let me classify 92% of the 9933 deaths due to terrorism. Note that Islam alone accounted for at least 82%, and eyeballing the groups I didn't match it's probably higher.

There were also a number of attacks that I found very hard to classify, e.g. Patani independence or Fulani pastoralism. Key summary of the Fulani Pastoralism conflict: the Fulani people of Nigeria are mostly nomadic cow herders and they are getting into violent land disputes with non-Fulani farmers who don't want Fulani cows eating/trampling their crops. The world is a big place and it's full of all sorts of bad shit most folks have never heard of.

Conclusion

It looks like Donald Trump is right and AOC is wrong. Even if we take high end estimates of the number of people killed by white supremacists in 2017 (34 in the US according to the SPLC), it seems like a small problem compared to things like Anti-Balaka, Communism or Balochistan independence.

There are many individual terrorist groups that I imagine most readers have never heard of, such as Indian Naxalites (communists), which kill far more people than white supremacists.

Also, far more importantly for most of my readers, you can easily extract data from Wikipedia into a dataframe using pandas.read_html and the wikipedia module.

Methodology

You can find my python notebook here.

Correction: A previous version of this post described an "Independent Nasserite Movement (a Socialist pan-Arab nationalist movement)", which was a reference to Al Mourabitoun. However that might have been me getting confused by wikipedia results - I think the actual attack in 2017 was done by a different Al Mourabitoun which is just ordinary boring Islamist violence. So we probably need to add another 77 or so to the Islam row.

Also, at least one commenter noted that the SPLC counts 34 dead due to white nationalists, which is higher than I get from Wikipedia. I don't particularly trust the SPLC, but I do reference it above. It still doesn't really change the results. Fulani Pastoralism killed more people.

Particles explain pressure

If I take an arbitrary set of particles obeying Newtonian mechanics, and choose a sufficiently large number of them, then the apparent factor "Pressure" will typically explain the behavior of pistons. To support that statement, I want to show you some evidence from what happens with random, artificial patterns of particles, where we know where the data came from (my copy of Laundau-Lifshitz). So that you don't have to just take my word for this, I describe my procedure and link to a textbook on statistical mechancs where you can explore these arguments in detail.

Suppose that the gas inside the vessel is not a gas in the continuous sense having some intrinsic quantity pressure, but is actually a collection of a huge number of non-interacting particles obeying Newton's laws of motion. It can be shown that the vast majority of the time, provided the vessel has been at rest for a while, that the distribution of particle velocities is approximately the same in any particular cube of volume. Furthermore, the density of particle positions will be uniformly distributed throughout the vessel.

For simplicity, let us suppose the volume is cubic of length L, and one side of the volume is the piston. Consider now a single particle in the cube, moving with an x-velocity \(v_x\). This particle will cross the cube once every \(t=L/v_x\) units of time, and each time it hits the piston it will exert a force \(F=2mv_x\) every \(t\) units of time. Thus, on average the force will be \(F=2m v_x^2/L\).

The total force on the piston will be the sum of this quantity over all the particles in the vessel, namely \(F=2mN \bar{v}_x^2/L\). Here \(\bar{v}\) denotes the average velocity (in the x-direction) of a particle. If we divide this by area, we obtain \(F/A=2mN \bar{v}_x^2/L^3 = 2m \rho \bar{v}_x^2\). Here \(\rho = N/L^3\) is the density of particles per unit volume. I.e., we have derived that \(P=2m \rho \bar{v}_x^2\)!

Thus, we have determined that under these simple assumptions, pressure is nothing fundamental at all! Rather, pressure is merely a property derived from the number density and velocity of the individual atoms comprising the gas.

But - and I can hear people preparing this answer already - doesn't the fact that there are these correlations in forces on pistons mean that there must be a single common factor somewhere? To which question a definite and unambiguous answer can be given: No. It looks like there's one factor, but in reality all the real causes are about equal in importance and completely independent of one another.

(As a tangential note, several folks I've spoken about this article vaguely recollect that temperature is when the particles in a gas move faster. This is true; increasing temperature makes \(\bar{v}_x^2\) increase. If we note that \(\bar{v}_x^2 = C T\) - ignoring what the constant C is - and multiply our equation of pressure above by the volume of the vessel V, we obtain \(PV = \rho V 2m C T\). Since \(\rho V = N\), this becomes \(PV = N R T\), with \(R=2mC\). We have re-derived the fundamental gas law from high school chemistry from the kinetic theory of gases.)

Doing without P

The end result of the self-confirming circle of test construction is a peculiar beast. If we want to understand the mechanisms of how gases in a vessel work, how we can use it to power a locomotive, I cannot see how this helps at all.

Of course, if P was the only way of accounting for the phenomena observed in physical tests, then, despite all these problems, it would have some claim on us. But of course it isn't. My playing around with Boltzmann's kinetic theory of gases has taken, all told, about a day, and gotten me at least into back-of-the-envelope, Fermi-problem range.

All of this, of course, is completely compatible with P having some ability, when plugged into a linear regression, to predict things like the force on a piston or whether a boiler is likely to explode. I could even extend my model, allowing the particles in the gas to interact with one another, or allowing them to have shape (such as the cylindrical shape of a nitrogen molecule) and angular momentum which can also contain energy. By that point, however, I'd be doing something so obviously dumb that I'd be accused of unfair parody and arguing against caricatures and straw-men.

Shalizi is right, but only about a trivial philosophical point

I'll now stop paraphrasing Shalizi's article, and get to the point.

In physics, we call quantities like pressure and temperature mean field models, thermodynamic limits, and similar things. A large amount of the work in theoretical physics consists of deriving simple macroscale equations such as thermodynamics from microscopic fundamentals such as Newton's law of motion.

The argument made by Shalizi (and repeated by Taleb) is fundamentally the following. If a macroscopic quantity (like pressure) is actually generated by a statistical ensemble of microscopic quantities (like particle momenta), then it is a "statistical myth". Lets understand what "statistical myth" means.

The most important fact to note is that "statistical myth" does not mean that the quantity cannot be used for practical purposes. The vast majority of mechanical engineers, chemists, meteorologists and others can safely use the theory of pressure without ever worrying about the fact that air is actually made up of individual particles. (One major exception is mechanical engineers doing microfluidics, where the volumes are small enough that individual atoms become important.) If the theory of pressure says that your boiler may explode, your best bet is to move away from it.

Rather, "statistical myth" merely means that the macroscale quantity is not some intrinsic property of the gas but can instead be explained in terms of microscopic quantities. This is important to scientists and others doing fundamental research. Understanding how the macroscale is derived from the microscale is useful in predicting behaviors when the standard micro-to-macro assumptions fail (e.g., in our pressure example above, what happens when N is small).

As this applies to IQ, Shalizi and Taleb are mostly just saying, "the theory of g is wrong because the brain is made out of neurons, and neurons are made of atoms!" The latter claim is absolutely true. A neuron is made out of atoms and it's behavior can potentially be understood purely by modeling the individual atoms it's made out of. Similarly, the brain is made out of neurons, and it's behavior can potentially be predicted simply by modeling the neurons that comprise it.

It would surprise me greatly if any proponent of psychometrics disagrees.

One important prediction made by Shalizi's argument is that in fact, the psychometric g could very likely be an ensemble of a large number of independent factors; a high IQ person is a person who has lots of these factors and a low IQ person is one with few. Insofar as psychometric g has a genetic basis, it may very well be highly polygenic (i.e. the result of many independent genetic loci).

However, none of this eliminates the fact that the macroscale exists and the macroscale quantities are highly effective for making macroscale predictions. A high IQ population is more likely to graduate college and less likely to engage in crime. Shalizi's argument proves nothing at all about any of the highly contentious claims about IQ.

People won't pay for privacy

Even if you were willing to pay for it, privacy is illegal

Question: What does the government call a payment platform that provides privacy to it's users?

Answer: Money laundering.

Here in India, the bulk of the privacy intrusions I run into are coming from the government. It is government regulations which require me to submit passport photocopies/personal references/etc in order to get a SIM card. Tracking my wifi use by connecting my Starbucks WiFi to a phone number via OTP is another government regulation. Prohibitions against the use of encryption are generally pushed by national governments. Things were pretty similar in the US.

It is, of course, impossible for a service provider to satisfy the government's desire to spy on users without doing so itself.

The desire for the government to spy on users extends far beyond preventing money laundering. In the United States, Congress has demanded information and action from technology companies in order to prevent Russians from posting Pepe memes on Twitter or attempting to organize "Blacktivism" on Facebook. The Kingdom bans most encrypted communication, and many democratic nations (the US, India, UK, France) have politicians pushing in the same direction.

In the intermediary stages, there is a large amount of information that the government requires service providers to keep. This typically includes accounting details (required by tax departments), both purchase history as well as KYC information used by tax authorities to track down tax evaders (e.g., Amazon is required to keep and provide to the IRS tax related information about vendors using Amazon as a platform).

In many cases, censorship authorities require social networks and others to track and notify them about people posting illegal content (Nazi imagery, child pornography, Savita Bhabhi, anti-Islamic content).

Fundamentally, it is government regulations that shut down cryptocurrency exchanges in India. It is government regulations that ban encrypted communication in the Kingdom (at least partially), and it was politicians in the US and UK and India who want to move in the same direction.

Insofar as privacy preserving platforms might exist, it is far from clear whether governments will allow them to continue existing should they become popular.

The privacy preserving service has approximately three crypto geeks and seven zillion Ukrainians hackers laundering money on it

. . .if you're against witch-hunts, and you promise to found your own little utopian community where witch-hunts will never happen, your new society will end up consisting of approximately three principled civil libertarians and seven zillion witches. It will be a terrible place to live even if witch-hunts are genuinely wrong.

-- Scott Alexander

Unfortunately, this Scott Alexander quote explains very nicely what will happen when someone builds a moderately successful privacy preserving network.

If we built a privacy preserving payment network, it would be used for money laundering, drug sales and ransomware. If the Brave private browser/micropayment system ever approaches viability, it will be overrun by criminals laundering money through blogs about Ukrainian food.

If an ad network vowed to protect privacy, fraud would shoot up and good advertisers would leave. The few remaining customers would be selling penis enlargement pills, accepting the click fraud as the cost of doing business because no one else will work with them.

There are privacy preserving/censorship resistant social networks. They're full of Nazis.

This is a fundamental collective action problem, and no player in the game seems to have the ability change things. There are bad actors out there - fraudsters/scammers, terrorists laundering money, legal gun manufacturers moving money around, child pornographers, people who believe in evolution (even among humans), people advocating abandoning Islam, Russians posting Pepe memes, and journalists/revenge pornographers revealing truthful information that people want kept hidden. Any privacy preserving network, at it's core, allows these people to engage in these actions without interference.

And as any network approaches viability, it's early adopters will be these sorts of undesirables.

Make no mistake; I want this privacy preserving network to exist. I have no problem with teaching evolution and exploring it's consequences, advocating atheism over Islam, laundering drug money, or teaching people how to manufacture firearms. But I'm very much in a minority on this.

And if, like me, you want this privacy preserving network, the first step in making that happen is recognizing and acknowledging the very real barriers to making it happen.

Setting up the problem

To lay out the basics of the problem, consider two competing companies - Bangalore Pvt. Ltd. and Cupertino Inc. For concreteness, let us say that these two companies are both sotware companies catering to the global market and they are direct competitors.

The question now arises; how should Bangalore and Cupertino allocate their capital?

For a software company, there are two primary uses toward which capital can be directed:

• Marketing. Both Bangalore and Cupertino can direct an extra $1 of spending towards adwords, facebook ads, attendance at conferences, and similar things. Both companies will receive the same amount of exposure on their marginal dollar.
• Employees. Bangalore and Cupertino can both spend money on employees, but in this case they receive different returns on investment. In Bangalore, a typical employee might cost 100,000 rupees/month, whereas in Cupertino an employee might cost $100,000/year. This is approximately a 5x cost difference if we round up 1 lac rupees/month to $20,000/year.

Let us now model what the effect of each resource is on revenue.

It's a simple arithmetic identity is that revenue \(R\) is equal to:

The values

is the probability of any individual prospect making a purchase multiplied by the value of that purchase, and

is the number of prospects who can be reached by marketing as a function of money spent on it.

We choose this decomposition because it helps us understand the impact of two separate resources:

• The value \(L\) is mainly increased by spending money on additional labor. Engineers can build features, which increase value for customers and allow the product to be sold for more money. Marketers may improve the brand value, increasing the probability of a sale.
• The value \(M\) is increased by spending money on additional marketing. It's a simple machine - money is spent on facebook ads, conferences, TV commercials, and more people become exposed to the product.

We also choose this decomposition since it helps us avoid the Cambridge Controversy, which can under other circumstances make the model less well founded.

Diminishing returns to labor and marketing

To understand the relationship between resources and production, let us take the following exercise. Suppose we have a large set of projects, each with a certain cost and benefit. To begin with, lets discuss labor projects:

• Integrate the software with Salesforce, cost 100 hours, benefit $50/prospect.
• Come up with a more enterprisey-sounding brand, cost 40 hours, benefit $10/prospect.
• Slap some AI on top of the software, cost 2000 hours, benefit $60/prospect.
• etc...

Fundamentally, I'm making two important assumptions here:

1. The projects have no interdependencies.
2. The amount of labor required for each project is small compared to the overall amount of labor.

Let us assume the corporate strategy is to spend whatever amount of labor we have on this collection of projects in order of decreasing ROI. This means that if we sort the list of projects by ROI = benefit / cost, then the corporate strategy will be to take on the highest ROI projects first.

Here's a typical result. As noted above, the units on the y-axis are dollars per prospect.

Note that the xkcd plotting style is used to illustrate this is a schematic drawing, and should not be taken too literally.

The graph was made as follows:

data = pandas.DataFrame({ 'cost' : uniform(10,100).rvs(50), 'benefit' : uniform(1,100).rvs(50) })
data['roi'] = data['benefit'] / data['cost']
data = data.sort_values(['roi'], ascending=False)

step(cumsum(data['cost']), cumsum(data['benefit'])) #Like `plot(...)`, except that it shows steps at each data point.

As can be seen, no particular correlation between cost and benefit was assumed in order to get diminishing returns. Diminishing returns follows solely from the sorting operation, i.e. the choice to take on the highest ROI projects first.

One can similarly construct a diminishing returns curve on marketing spend. Note also that on the marketing side, many marketing channels (for example adwords) have their own diminishing returns curves built in.However, there's one very important distinction between labor and marketing. For the labor graph the X-axis is hours of labor, while for marketing the X-axis is amount of money spent.

The Cobb-Douglas Model

After observing the diminishing returns curve above, I thought it looked kind of like a \(t^{\alpha}\) for some \(0 < \alpha < 1\). So I decided to do a least squares fit using the model \(y(t) = a t^{\alpha}\). Using numpy, this can be accomplished in a fairly straightforward way using the minimize function from scipy:

def err(a):
    t = cumsum(data['cost'])
    y = cumsum(data['benefit'])
    return sum(pow(y - a[0]*pow(t,a[1]), 2))

x = minimize(err, [1.0, 1.0])

The result of this optimization yields \(\alpha = 0.518\), as well as a reasonably accurate best fit curve:

This kind of a graph shape is not an accident. I repeated ths experiment, but this time generating a different data set:

data = pandas.DataFrame({ 'cost' : expon(1).rvs(50), 'benefit' : expon(1).rvs(50) })

I repeated this experiment, but this time I used a different distribution of costs/benefits. The result was pretty similar, albeit with \(\alpha = 0.497\):

I suspect that there is some more interesting law of probability which is causing this result to occur, but I'm not entirely sure what.

If we substitute this back into the equation \(R = L \cdot M\), we arrive at the Cobb-Douglas model:

In the Cobb-Douglas model, the term \(A\) represents Total Factor Productivity.

Note: Normally, the use of the Cobb-Douglas model is somewhat problematic due to the Cambridge Controversy which points out the difficulties in assigning a single value to capital. However in this case capital is literally dollars which can be spent on marketing, so we can avoid the issue.

What should the optimal firm do?

Let us now suppose that both Bangalore Pvt. Ltd. and Cupertino Inc. have a fixed amount of capital \(C\) available for spending in the current period. These firms can convert capital into labor at the rates:

• Bangalore Pvt. Ltd.: 1 unit of capital converts to 1 unit of labor.
• Cupertino Inc.: 5 units of capital convert to 1 unit of labor.

Now let \(m\) represent the fraction of capital spent on marketing. Then we can rewrite our output (in Bangalore) as:

Whereas in Cupertino our output is:

Note that these outputs differ solely due to the presence of the \(5^{-\alpha}\) sitting in front. The dependence on \(m\) is unchanged. We can maximize this with simple calculus:

Solving this for \(m\) yields \(m = \beta / (\alpha + \beta)\).

In pictures, the following is what is happening:

As can be seen from the graph, the production function for both firms is the same, as is the capital allocation that maximizes production. All that differs is the level of production.

It's important to recognize what this means in business terms: the sole difference between Cupertino and Bangalore is that Bangalore has a higher total factor productivity. In terms of capital allocation, both firms should behave in the same way.

Secondly, this means that revenue in Bangalore Pvt. Ltd. will be higher than at Cupertino Inc. by a factor of \(5^{\alpha}\).

The third conclusion is that revenue per employee will be significantly lower at Bangalore Pvt. Ltd. Bangalore Pvt. Ltd. is devoting the same amount of capital to labor as Cupertino Inc., but it has 5x lower cost per employee. As a result, it will have 5x as many employees as Cupertino Inc. It's revenue is higher by a factor of \(5^{\alpha}\), but the number of employees is higher by a factor of \(5^1\). As a result, revenue per employee is lower by a factor of \(5^{\alpha-1}\) (recall that \(0 < \alpha < 1\)).

For example, assuming \(\alpha = 0.5\) (as it appeared to be in the synthetic examples I concocted above), this means Bangalore Pvt. Ltd. will have \(\sqrt{5} = 2.236\) as much revenue as Cupertino Inc., but the revenue per employee will be only 0.447 as large as that of Cupertino Inc.

What does this mean for the business?

It's often a bit difficult to translate abstract economic results into practical business advice. In this case, what the economic result implies is the following.

Because \(m\) is the same for both Bangalore Pvt. Ltd. and Cupertino Inc., both firms should spend approximately the same fraction of their capital on labor. This will result in Bangalore Pvt. Ltd. consuming more labor (i.e. having more employees, and having more labor hours), and moving further along the diminishing returns curve.

For example, if these competing firms are in the adtech business, then integrating with more ad networks might be a valuable way to increase their customer value. In this case, while Cupertino Inc. might integrate only with Adwords, Facebook and AppNexus, Bangalore Pvt. Ltd. might integrate with those networks as well as YouTube, Pornhub and other more niche sites. If these firms are in the business of selling an ecommerce widget, then Bangalore Pvt. Ltd. might provide a larger number of specialized themes than Cupertino Inc. In most software businesses there is value to be generated by repeating the same process for more data providers, more platforms, etc. Generally speaking, an Indian firm should make their product significantly broader than any corresponding western firm.

Similarly, on the marketing side, one might expect Bangalore Pvt. Ltd. to create a broader advertising surface. This might involve creating a larger number of landing pages, which would target smaller niches of customers. Similarly, one would expect more organic marketing as a fraction of total marketing.

At the micro level, the fundamental calculus is the following. For Cupertino Inc. to take on a project requiring 1 man-year of labor, the project must generate $100k in revenue to break even. In contrast, Bangalore Pvt. Ltd. can take on any project generating $20k in revenue or more. As a result, Bangalore Pvt. Ltd. should take on all the same projects as Cupertino Inc., in addition to projects generating between $20k-100k revenue.

Projects in this revenue range form a natural core competency for the Indian firm; simple economics forms a moat that few western firms can cross.

So in terms of practical business advice, the takeaway (for Indian firms) is the following: hire more people, and have them work on more marginal projects. It will lower your revenue/employee, but it will increase profits and help you capture business that western competitors are economically incapable of capturing.