Note: This website is moving to https://prediction-markets.neocities.org/. This post is cross-posted to write.as.

This post describes how to use the Python package zipline to backtest an equities trading strategy. Backtesting is the practice of evaluating a trading model or heuristic against past data. Although past performance is no guarantee of future performance, practitioners generally feel more confident in strategies which would have performed well had they been executed in the past (given information available at the time). Another excellent reason to use backtesting is to catch conspicuously bad mistakes (e.g., coding or math errors) that are likely to guarantee poor performance in the future.

This post covers the basics of using zipline in the context of trading equities. A future post will discuss using zipline to backtest a prediction markets trading strategy.

Overview

These instructions assume you are using zipline version 1.3.0.

Using zipline involves two distinct steps:

1. Loading data
2. Running a strategy which uses the data

Loading data from CSV

zipline can load arbitrary price data from a CSV file provided the price data is formatted in a specific way. Fortunately, the required format is relatively common. The format is called OHLCV and has eight columns: date, open, high, low, close, volume, split, and dividend. Do not worry about the split and dividend columns. They will be uniformly 0.0 and 1.0, respectively. The remaining columns are self-explanatory. Here is an example line in the file AAPL.csv.gz: 2012-01-03,58.485714,58.92857,58.42857,58.747143,75555200,0.0,1.0. The symbol associated with the price data is taken from the filename. In this case, the symbol is AAPL.

To make the data available to zipline for backtesting, we load (“ingest”) the data by putting the file into a daily subdirectory inside a directory and running the following command:

CSVDIR=/path/to/data python3 -m zipline ingest -b csvdir

where /path/to/data is replaced with the appropriate directory. If /path/to/data is our directory, then AAPL.csv.gz is located at /path/to/data/daily/AAPL.csv.gz. (AAPL.csv.gz can be found in the zipline repository.) Note the required daily subdirectory. If you have done everything correctly then your command should be greeted with a message which begins with Loading custom pricing data: ....

Testing a Trading Strategy

Now that we have loaded historical price data we can test trading strategies.

Let's test the following “strategy”: buy 10 shares of stock on Mondays, sell 10 shares on Wednesday. (Who knows, maybe we've discovered that people are pessimistic on Mondays and tend to systematically undervalue equities.)

To implement this strategy we need to figure out how to cast it into terms used by the zipline API. Fortunately for us the strategy is simple, so this will not be difficult. We know that zipline requires us to write a function which gets called every day, handle_data, where this function can buy and sell things. We can buy and sell shares with the order function. The difficult part here is querying zipline for the date(time), from which we calculate the day of the week. To retrieve the current datetime in our called-every-day function, we need to inspect an attribute, current_dt, of the data argument (an instance of BarData) which gets passed to our function. When dealing with daily data, as we are here, this datetime is the end of the current trading period for that day. Since AAPL is traded in the United States and trading closes at 16:00 New York time, an exemplary data.current_dt would be 2014-06-16 20:00:00+00:00.

Let's make a first attempt at writing a strategy. We will start with an empty initialize function (which zipline requires we define) and then craft our handle_data function. Here it is:

from zipline.api import order, record, symbol

def intitialize(context):
    pass

def handle_data(context, data):
    # weekday is a method of `datetime.datetime`, Mon = 0, Fri = 4
    weekday = data.current_dt.weekday()
    if weekday == 4:  # place order on Friday for Monday execution
        order(symbol('AAPL'), 10)
    elif weekday == 1: # place order on Tuesday for Wednesday execution
        order(symbol('AAPL'), -10)
    record(AAPL=data.current(symbol('AAPL'), 'price'))

If we save this code in a file strategy1.py we can backtest it over any period for which we have data. For example, we can test this minimal strategy over the first full trading week in January 2012 with:

zipline run -f strategy1.py -b csvdir --start 2012-1-6 --end 2012-1-13 -o strategy1_out.pickle

The output of a zipline run is a pandas data frame which can be read with the pandas.read_pickle function. The following lines of code will open the data frame and print the following:

• The opening value of our portfolio (i.e., the cash we start with)
• The closing price of AAPL on Monday (at which we buy 10 shares)
• The closing price of AAPL on Wednesday (at which we sell 10 shares)
• The closing value of our portfolio

import pandas as pd
df = pd.read_pickle("strategy1_out.pickle")
print('Opening `portfolio_value`:', df.iloc[0].portfolio_value)
print('AAPL on Monday:', df.iloc[1].AAPL)
print('AAPL on Wednesday:', df.iloc[3].AAPL)
print('Closing `portfolio_value`', df.iloc[5].portfolio_value)

which outputs:

Opening `portfolio_value`: 10000000.0
AAPL on Monday: 60.247
AAPL on Wednesday: 60.364
Closing `portfolio_value` 10000000.5469

So in particular one week we've earned a profit of 0.55 having made use of 602.247 ($10 \times 60.247$) (ignoring fees and slippage). In annualized terms this is a return of about 2.8% which is better than the risk-free rate in early 2012 (around 1.9%). Of course we only backtested the algorithm for one week. (Our strategy is also very risky.) To run this strategy for an entire year we would change the end date: zipline run -f strategy1.py -b csvdir --start 2012-1-6 --end 2012-1-28 -o strategy1_out.pickle.

Using zipline in a prediction market setting

To use zipline in a prediction market setting one needs correctly formated data. Obtaining current price information, including the bid-ask spread, is typically not difficult. PredictIt, for example, makes this data available in a machine-readable format (e.g., https://www.predictit.org/api/Market/3633/Contracts).

---

Note: This website is moving to https://prediction-markets.neocities.org/. This post is cross-posted to write.as.

#backtesting #zipline

This website is moving to https://prediction-markets.neocities.org/. You can find all the content here at the new address.

New material will be posted at the new website.

How bad can things get? A risk measure provides an answer.

Suppose you are presented with a wager costing 0.4 which returns 1 with probability 0.5 and otherwise zero. Such a wager has an expected return of 25%. You can make this wager once a day over the next 10 days. Suppose you do. How bad can things get? We know that there's a small probability ($0.5^{10}$) that you lose 4 (i.e., you lose all 10 wagers). And there's a larger probability, $10 \cdot 0.5^{10}$, that you lose 3 (i.e., win one, lose 9). Although these are useful pieces of information, we might wish for a single-number summary of the riskiness of our planned series of bets. Expected shortfall is one such measure of risk. It describes how much you should expect to lose if things go bad. $\newcommand{\VaR}{\text{VaR}} \newcommand{\E}{\text{E}}$

Expected shortfall, also known as conditional value at risk, is typically defined in terms of value at risk (VaR). Let's begin by defining value at risk. (VaR is also a risk measure, just not a particularly good one.) For a given set of wagers (“portfolio”) over a defined period of time and a specified confidence level α, the VaR is the (1-α)-quantile of the portfolio's loss distribution. If the profit-and-loss distribution is X then $\VaR_α(X)$ is the (1-α)-quantile of Y := -X. (Losses are negative, profits positive.) Suppose our portfolio X has a standard normal distribution. Then the VaR at level 0.05 of our portfolio is 1.645. 1.645 is the 0.95-quantile of -X. Suppose, alternatively, that our portfolio Z has a probability of 0.05 of losing 2 million or more. The VaR at level 0.05 of the portfolio is 2 million. 2 million is the 0.95-quantile of -Z.

Expected shortfall, like value at risk, is defined with respect to a period of time and a confidence level α. It is the expected loss in the worst α of cases. If the profit-and-loss distribution is X and X follows a continuous probability distribution then the expected shortfall of X at level α is the E S_α = E[– X | X ≤ – VaR_α(X)]. This is the left-tail conditional expectation below –$\VaR_α(X)$. Suppose again that our portfolio X has a standard normal distribution. Then the expected shortfall at level 0.05 of our portfolio is 2.0627. Expected shortfall, like value at risk, is a single-number measurement of how much we should expect to lose if things go bad.

Expected shortfall is a better measure of downside risk than VaR because expected shortfall gives us the average loss when things go bad. VaR, by contrast, tells you a single example of a bad outcome. Consider the following example of a portfolio X which has an expected 2% return. The portfolio costs 1 to acquire and returns 1.6 with probability 0.95, 0 with probability 0.04, and -50 with probability 0.01. The VaR at level 0.05 of this portfolio is 0. The expected shortfall at level 0.05 is 10. 0 is an example of a bad outcome (VaR). 10 is a measure of the average badness, given a bad outcome. Expected shortfall provides a richer description of downside risk than VaR. For this reason people tend to prefer it whenever it is available.

In a prediction market setting, use expected shortfall to characterize the downside risk of a portfolio of wagers. Consider the example of a portfolio which consists of 10 independent wagers. Each wager costs 0.5 and pays out 1 with probability 0.51. (This portfoilio has an expected return of 2%.) To calculate the expected shortfall we calculate the left-tail conditional expectation below $-\VaR_α(X) = 2$. Calculating this can be accomplished in at least two ways. Simulating the portfolio is likely the easiest way. We arrive at an expected shortfall of 3.21.

When considering different portfolios which we might purchase, it is natural to ask about the typical case: How much do we stand to gain (or lose) on average? It is also natural—or should be natural—to ask how bad things can get. Expected shortfall provides an answer. Expected shortfall describes how much you should expect to lose if things go bad.

---

This post is part of a series. The most recent post in the series is “Virtues of prediction markets: Useful baselines”. Learn when new posts appear by subscribing (RSS). You may also follow @prediction-markets@write.as in Mastodon or subscribe for email updates.

#predictionmarkets #expectedshortfall #conditionaltailrisk #valueatrisk

Participation in prediction markets has a social benefit: a market's prediction provides a useful baseline against which other predictions (outside the market) may be judged. A market's prediction can also be useful to individuals making decisions that depend on the outcome of the predicted event. In both cases, the social utility provided by the market remains even if the market's prediction is not perfect. The market is useful even if it only provides a general assessment of the relative probabilities of different outcomes.

Prediction markets provide baseline predictions about uncertain events. Consider the task of predicting which candidate will receive the US Democratic party's presidential nomination in 2020. In May 2019 there were 23 candidates for the nomination. The thought that each candidate is equally likely to win the nomination does not merit being described as a baseline prediction. Some candidates have a far better chance than others. Here the prediction market gives the public a general sense of which candidates are better positioned than others (Figure 1). And the public gets this for no cost.

Figure 1: The probability of one of these five candidates winning is 87%, in the judgement of PredictIt participants in late May 2019. There are many other candidates in this race. The prediction market here tells us that some candidates stand a far better chance than other candidates.

The utility of having a baseline prediction is easy to appreciate in the case of political events. With political events, we have political commentators and polling firms which deliver forecasts of what will happen. It is natural to ask whether or not these commentators or firms are skilled at forecasting. Given that polling is expensive and salaries for political commentators are non-zero, we anticipate that their predictions will approach or exceed a certain standard of utility. One standard we can use is the following: ask that their predictions tend to be more accurate than those made by prediction markets. Having prediction markets provide a standard here is particularly valuable because there are typically no other standards we can use. We cannot, for example, ask the commentator or firm to do better than the “prediction” which assumes each candidate has an equal chance of winning because virtually anyone can make better predictions than that. Prediction markets prove their worth by providing a baseline in settings where non-trivial baselines are scarce.

Consider participating in a prediction market. Even if your wagers fail to pay out, the public stands to benefit.

---

This post is part of a series. The most recent post in the series is “Are there sure bets on PredictIt?”. Learn when new posts appear by subscribing (RSS). You may also follow @prediction-markets@write.as in Mastodon or subscribe for email updates.

#predictionmarkets #predictit

This site reviews essential background knowledge required for trading on prediction markets. It also describes basic trading strategies.

Prediction markets aggregate and make visible informed opinion about future events. Prediction markets have proven capable of forecasting the outcome of political elections and scientific experiments. Although trading on prediction markets such as PredictIt and Betfair resembles trading on other electronic markets, some conventions and trading strategies are particular to prediction markets.

Contributors

• Andrew Lee, PhD, is a Brooklyn-based researcher with over a decade of experience in statistical consulting and software engineering. His involvement in prediction markets grew out of an interest in political forecasting. His first trades were made on the InTrade exchange in 2012. For the moment, he wagers that prediction markets benefit the public by gathering useful information from a diverse population.

Are there bets with guaranteed positive return on the prediction market PredictIt?

Yes and no. “Yes” because examples of such bets do exist. “No” because these bets have negative present-value adjusted returns. (Bets with guaranteed positive returns are called “sure bets.”)

Here is one example of a bet which does indeed have a guaranteed positive return. During the spring of 2019 (e.g., April 28 to May 2) you could purchase two contracts in the 2020 U.S. presidential election market on PredictIt which, together, paid off 100¢ with certainty and cost 99¢. Specifically, you could purchase the following two contracts:

1. a contract which costs 0.43 paying out 1 if the Republican Party candidate does not win, and
2. a contract which costs 0.56 paying out 1 if the Democratic Party candidate does not win (which pays out if the Republican wins).

So for 0.99 you can receive 1 if the Republican candidate wins or does not win. As the probability of this bet paying out is 100%, this contract is guaranteed to yield a profit of 0.01. As this particular market is one of the most liquid markets on PredictIt, you could purchase 1517 of each contract for a total of \$1501.83—PredictIt imposes a limit of \$850 invested in a single contract market—and be guaranteed to receive \$1517, making a profit of \$15.17. This bet qualifies as a sure bet.

Figure 1: A sure bet on PredictIt. The sure bet shown here was available during several days in May 2019. This sure bet consists of wagering against the Republican candidate winning and also against the Democratic candidate winning. (This second wager is equivalent to betting on the Republican candidate winning.) Making these wagers costs less than 1. The bet against Republican costs 43¢ (100¢ – 58¢ + 1¢) and the bet against Democrat costs 56¢ (100¢ – 45¢ + 1¢). (The spread, in both cases, was 1¢.) Purchasing one of each contract costs 99¢.

Before you get out your wallet, recall that purchasing a contract in this market pays out in November 2020. Taking into account the present value of the future income stream makes the return on the bet negative. As the risk-free interest rate in May 2019 was 2.31%, the present value of a cash flow of \$1,517 in 18 months is \$1,465.91 ( $\frac{1517}{(1+0.0231)^{18/12}}$ ). Hence the present-value-adjusted return on the investment of \$1501.83 is an unattractive -2.4%. You would be better off investing the \$1501.83 in a high-yield savings account.

---

This post is part of a series. The most recent post in the series is “The present value of a bet's payoff”. Learn when new posts appear by subscribing (RSS). You may also follow @prediction-markets@write.as in Mastodon or subscribe for email updates.

#predictionmarkets #predictit

Future income flows should always be expressed in present value terms. This practice is as important in the prediction market setting as it is elsewhere. The payoffs on winning prediction market bets frequently do occur many months from the present. Indeed, it is not uncommon to see prediction market bets concerning events that might occur two or more years in the future. If the wager was made using a currency which experiences inflation, then the payoff should be discounted to reflect the change in the value of the currency. When we calculate the value of the future payoff in terms of the value of the currency today, we are calculating the present value or the present discounted value of the future income flow.

The present value of a future income flow can be illustrated with an example. Suppose the date is January 1st and it is possible to earn 2% interest on a 1-year time deposit (aka certificate of deposit) with a major bank. How much should one be willing to pay on January 1st for a future income flow of 100 in one year? Clearly it must be less than 100 because it is possible to deposit 100 with the major bank and receive 102 in one year's time. Accepting a price of 100 would “lock in” a loss of 2. The commonly accepted answer is $\frac{1}{1 + r} 100 \approx 98.04$, where $r$ is equal to the “risk-free” rate (2%). We say that the present value of 100 in a year's time is 98.04. It would be wasteful to pay more than 98.04 for a future income flow of 100 because you would be better off depositing your money in the bank and withdrawing it, with interest, in a year's time.

(Energetic students willing to engage in light algebra may derive the present value formula using principles mentioned above.)

Using a present-value adjustment in the prediction market setting is relatively straightforward. Consider a contract that costs 0.98 today and pays out with certainty in one year. (It is not difficult to find contracts fitting this description on PredictIt.) Assuming a 2% risk-free rate, what is the return on this contract, in present value terms? The present value of 1 in one year is $\frac{1}{1+r} 1 \approx 0.9804$, making the return a measly 0.04%. In fact, the return is negative if we account for PredictIt's fees. What appeared on the surface to be a sure bet is, upon reflection, a losing wager.

#predictionmarkets #predictit

---

This post is part of a series. The most recent post in the series is “The riskiness of a bet in a prediction market”. Learn when new posts appear by subscribing (RSS). You may also follow @prediction-markets@write.as in Mastodon or subscribe for email updates.

Some bets are riskier than others. Consider the following two bets: (1) a gamble costing 0.45 on the outcome of a flip of a fair coin paying out 1 if the coin lands “heads” and (2) a gamble costing 0.75 on a die roll paying out 1 if the die shows a number other than six. The first gamble (on the coin flip) has a healthy expected return of 11%. This is the same expected return as the gamble on the die roll. These gambles differ, however, in their risk. If forced to choose between these two gambles, most people would choose the second as it is less risky.

A quick way to get an intuitive sense of risk is to do some simulation: evaluate the risk of an individual gamble by reflecting on the gain or loss associated with making the gamble 100 times. (Make the assumption here that outcomes of these repeated gambles are independent.) Figure 1 compares the distributions of returns associated with making each wager (the coin flip and the die roll bet) 100 times. The relative riskiness of the wagers is easy to see. Whereas in the coin flip case the probability of losing money over 100 independent bets is 14%, the probability of losing money in the die roll case over 100 independent bets is 1%.

Figure 1: Return on two different repeated (independent) gambles. Each figure shows the return associated with making a specific bet 100 times where the outcome of each bet is independent. The expected return of both bets is the same.

We can make this particular definition of risk precise by defining the riskiness of a gamble as the standard deviation of its return. (The standard deviation of a random variable is the square root of its variance.) A little calculation will show that the standard deviation of the return on the coin-flip bet is 111% and the standard deviation of the return on the die-roll bet is 50%. Without any detailed interpretation of these numbers, one thing is clear: the riskiness of the coin-flip bet is higher than the riskiness of the die-roll bet. This is consistent with what we observe in the simulation exercise above.

If the prospect of losing money concerns us more than the prospect of winning money—and there exist principled reasons for adopting such a stance—then we are “risk-averse”. We prefer the wager with the lower risk.

---

This post is part of a series. The most recent post in the series is “The expected return on a bet”. Learn when new posts appear by subscribing (RSS). You may also follow @prediction-markets@write.as in Mastodon or subscribe for email updates.

#predictionmarkets #risk

The return or rate of return is the standard way of characterizing the benefit associated with making an investment. The return on an investment is calculated by dividing an investment's net payoff by the cost of the investment. A bet on a prediction market is a kind of investment. Comparing bets in terms of their returns allows us to ignore irrelevant details and focus on the relative benefits of bets.

In general, an investment which costs $p$ to purchase and which yields $q$ at some later date has a return of $r = \frac{q – p}{p}$. For example, a bet which costs 10 today and pays out 12 tomorrow (with certainty) has a return of $r = \frac{12 – 10}{10} = 0.2$. Presented with bets that have equal risk, people tend to prefer bets with higher rates of return.

In order to compare investments which have uncertain payoffs, the expected return is typically used. The expected return is, as the name suggests, the expected value of the return. A bet costing $p$ which pays 1 tomorrow with probability $\pi$ and 0 otherwise has an expected return of $\frac{\pi – p}{p}$. (The variance of the return is $\frac{\pi(1-\pi)}{p^2}$.) A more concrete example would be a bet which costs 8 today and pays out 10 tomorrow with probability 90% and nothing otherwise. This bet has an expected return of $\frac{9-8}{8} = 0.125$. Again, presented with bets that have equal risk, people tend to prefer bets with higher expected returns.

#predictionmarkets #return

---

This post is part of a series. The most recent post in the series is “Bids, offers, and the bid-offer spread.” Learn when new posts appear by subscribing (RSS). You may also follow @prediction-markets@write.as using Mastodon.

A market is a tool for matching buyers with sellers. Electronic markets are a particular kind of market. This note introduces three concepts used in virtually all electronic markets in the early 21st century: bids, offers, and the bid-offer spread. This note discusses these concepts in the context of a prediction market, a particular kind of electronic market.

Sellers post offers to sell a quantity of a known product at a specified price (e.g., 50 metric tons of corn for 185 USD). A market aggregates and makes visible these offers for prospective buyers. In summaries of current market activity, the market typically makes visible the offer associated with the lowest price (“lowest offer”). Often there is no buyer interested in buying the product at this lowest offer. (In the jargon, we say there is no buyer willing to pay an offer.) In this situation, sellers leave an offer in the hopes that a buyer may eventually arrive who is interested in paying the quoted price. Provided an offer has not been paid, sellers may typically cancel offers at any time.

Buyers post bids to purchase a quantity of a known product at a specified price. Again, a market aggregates and makes visible these bids for prospective sellers. In summaries of current market activity, the market typically shows the highest bid. If there is no seller interested in selling the product at a price a buyer is willing to pay—-in the jargon, no seller is willing to give a bid—-the buyer leaves a bid in the hopes a seller may arrive who is interested in selling at the quoted price.

The difference between the price associated with the highest bid and the price associated with the lowest offer is the bid-offer spread. In a functioning market the bid price is lower than the offer price. Hence we use the term bid-offer spread (rather than the offer-bid spread). Large spreads are the norm. Only with heavily traded products does the spread tend to be small. A screenshot of summary information about three contracts traded on a prediction market in early 2019 is shown as Figure 1. In the screenshot you can see the following information for each contract: the price at which the last successful trade occurred, the lowest offer, and the highest bid.

Figure 1: Bid-offer spreads for three contracts concerning the 2020 United States presidential election. Bid and offer prices are not labeled according to standard conventions. The price in the second column of prices is associated with the lowest offer. The price in the third column is associated with the highest bid. The bid-offer spread for all three contracts is 0.01 USD.

Making skillful use of an electronic market requires being familiar with bids, offers, and the bid-offer spread. Buyers communicate how much they are willing to pay for a product by posting bids. Sellers indicate their willingness to sell by posting offers. Very frequently there is a gap between the highest price associated with a bid and the lowest price associated with an offer. This gap is the bid-offer spread. When a product is infrequently traded, the bid-offer spread tends to be large. Even when no trades occur, buyers and sellers learn something about supply of and demand for a product.

#predictionmarkets

---

This post is part of a series. This is the first post in the series. Learn when new posts appear by subscribing (RSS). You may also follow @prediction-markets@write.as using Mastodon.