
[Wordle](https://www.powerlanguage.co.uk/wordle/) has taken the world by storm. Here's a program to solve it efficiently.

The rules behind the game are straightforward. For each letter in the guess, the following rules apply:
- if the letter is in the correct spot, show a 🟩
- if the letter is in the word but not in the correct spot, show a 🟨
- if the letter is not in the word, show a ⬜️

For example, <span className="font-mono font-black tracking-widest"><span className="bg-green-400 text-green-900 dark:bg-green-700 dark:text-green-100">W</span>EARY</span> means that `W` is in the word and in the correct spot. <span className="font-mono font-black tracking-widest">P<span className="bg-yellow-300 text-yellow-900 dark:bg-yellow-600 dark:text-yellow-100">I</span>LLS</span> means that `I` is in the word but in the wrong spot. <span className="font-mono font-black tracking-widest">VAG<span className="bg-slate-300 text-slate-800 dark:bg-slate-600 dark:text-slate-100">U</span>E</span> means that `U` is not in the word.

The examples below are available on [Github](https://github.com/ryanrishi/wordle-solver).

# Recreating Wordle For Testing
Here's a class to hold the result of each letter in the guess.
```java
@Getter
@RequiredArgsConstructor
class LetterGuess {
  private @NonNull char letter;
  private boolean isInWord;
  private boolean isInWordAndInCorrectLocation;
}
```

Here's a representation of the game. Each game has an answer and provides a way to guess that answer.
```java
class Wordle {
  public static final int WORD_LENGTH = 5;

  private final String answer;

  @Getter
  private int numGuesses = 0;

  @Getter
  private boolean solved;

  public Wordle(String answer) {
    this.answer = answer.toLowerCase();
  }

  /**
   * Something that shouldn't stop the game, just something wrong with your guess.
   */
  static class BadGuessException extends IllegalArgumentException {
    public BadGuessException(String msg) {
      super(msg);
    }
  }

  public List<LetterGuess> guess(String guess) {
    if (guess.length() != WORD_LENGTH) {
      throw new BadGuessException(String.format("Guess must be %d characters long", WORD_LENGTH));
    }

    guess = guess.toLowerCase();
    ++numGuesses;

    if (answer.equals(guess)) {
      // victory
      solved = true;

      return Collections.emptyList();
    }

    // use a map of letter in answer => number of times this letter occurs in the answer
    Map<Character, Integer> letterFrequencies = getLetterFrequencyMap();
    List<LetterGuess> result = new ArrayList<>();

    // handle letters in correct location first
    for (int i = 0; i < guess.length(); i++) {
      char c = guess.charAt(i);
      LetterGuess letterGuess = new LetterGuess(c);

      if (c == answer.charAt(i)) {
        letterGuess.isInWord = true;
        letterGuess.isInWordAndInCorrectLocation = true;
        letterFrequencies.put(c, letterFrequencies.get(c) - 1);
      }

      result.add(letterGuess);
    }

    for (int i = 0; i < 5; i++) {
      char c = guess.charAt(i);

      if (letterFrequencies.getOrDefault(c, 0) > 0) {
        result.get(i).isInWord = true;
        letterFrequencies.put(c, letterFrequencies.get(c) - 1);
      }
    }

    return result;
  }

  private Map<Character, Integer> getLetterFrequencyMap() {
    Map<Character, Integer> letterFrequencies = new HashMap<>();
    for (char c : answer.toCharArray()) {
      letterFrequencies.putIfAbsent(c, 0);
      letterFrequencies.put(c, letterFrequencies.get(c) + 1);
    }

    return letterFrequencies;
  }
}
```

We can also create a driver program to try out the game:
```java
public static void main(String[] args) {
  Wordle wordle = new Wordle("price");

  Scanner scanner = new Scanner(System.in);
  while (!wordle.isSolved()) {
    String guess = scanner.nextLine();
    List<LetterGuess> result = wordle.guess(guess);

    System.out.println(result.stream().map(l -> {
      if (l.isInWordAndInCorrectLocation) {
        return "🟩";
      } else if (l.isInWord) {
        return "🟨";
      } else {
        return "⬜️";
      }
    }).collect(Collectors.joining()));
  }
}
```

```
Welcome to Wordle. Enter your guess below.
(1/6) > asdflkjasfdbpojaerb
Guess must be 5 characters long
(1/6) > about
⬜️⬜️⬜️⬜️⬜️
(2/6) > prize
🟩🟩🟩⬜️🟩
```

It won't cover cases like guesses that aren't 5 letters or aren't in the dictionary, but for our purposes, this will get the job done.

Before we start implementing a program to solve this, let's define an interface that each type of solver will implement.
```java
interface Solver {
  /**
   * @param wordle the game
   * @return the answer to the game, or null if the game could not be solved
   */
  String solve(Wordle wordle);
}
```

# Brute Force Approach
A naive approach is to check every single 5-letter word in the dictionary until the game is solved. In order to implement this, I've pulled in a dictionary from [dwyl/english-words](https://github.com/dwyl/english-words/blob/master/words_alpha.txt).

```java
class BruteForceSolver implements Solver {
  private Set<String> dictionary = new TreeSet<>();

  public BruteForceSolver() {
    // load the dictionary
    try (InputStream is = BruteForceSolver.class.getResourceAsStream("/words_alpha.txt")) {
      if (is == null) {
        throw new RuntimeException("Could not load file");
      }

      Scanner scanner = new Scanner(is);
      while (scanner.hasNext()) {
        String word = scanner.next();
        if (word.length() != 5) {
          // skip over words that aren't 5 letters long
          continue;
        }

        dictionary.add(word);
      }
    } catch (IOException ioe) {
      ioe.printStackTrace();
      throw new RuntimeException("Error loading dictionary");
    }
  }

  @Override
  String solve(Wordle wordle) {
    for (String word : dictionary) {
      wordle.guess(word);

      if (wordle.isSolved()) {
        return word;
      }
    }

    // couldn't solve the game
    return null;
  }
}
```

Let's test it out.
```java
@Test
void testBruteForce_about() {
  String answer = "about";
  Wordle wordle = new Wordle(answer);
  Solver solver = new BruteForceSolver();
  assertEquals(answer, solver.solve(wordle));
  System.out.println("brute force - about: " + wordle.getNumGuesses());
}

@Test
void testBruteForce_zorro() {
  String answer = "zorro";
  Wordle wordle = new Wordle(answer);
  Solver solver = new BruteForceSolver();
  assertEquals(answer, solver.solve(wordle));
  System.out.println("brute force - zorro: " + wordle.getNumGuesses());
}
```
```
brute force - about: 60
brute force - zorro: 15907
```

Guessing `about` took 60 guesses since it was at the start of the dictionary, but guessing `zorro` required 15,907 guesses because this program has to guess almost every word in the dictionary before finding the solution.


# Iterative Approach
To improve, we can leverage the result from each guess in order to inform our future guesses.

Thinking back to the rules of the game, we can improve our solution by doing the following:
- If the letter is in the word and in the correct location (🟩), filter out any words that don't have that letter in that location
- If the letter is in the word but _not_ in the correct location (🟨), filter out any words that don't have that letter in the word
- If the letter is not in the word (⬜️), filter out any words that _do_ have that letter

For example, if the answer is `light` and our first guess is `grout` (🟨⬜️⬜️⬜️🟩), we would do the following:
- Filter out any words that don't end in `t` (🟩). This eliminates words like `horse` and `biker` but keeps words like `blast` and `pleat`
- Filter out any words that don't have a `g` (🟨). This eliminates words like `built` and `bulbs` but keeps words like `egret` and `ought`
- Filter out any words that contain letters that aren't in the solution (⬜️). In this example, eliminate any words that contain a `r`, `o`, or `u`
- Guess the first word in the set of remaining words
- Continue using the result of each guess to filter out future guesses

Let's put this into code.
```java
class IterativeSolver implements Solver {
  @Override
  public String solve(Wordle wordle) {
    Set<String> futureGuesses = new LinkedHashSet<>(seeds);
    futureGuesses.addAll(dictionary);
    Set<Character> lettersNotInSolution = new HashSet<>();
    List<Character> lettersInSolution = new ArrayList<>();

    while (!wordle.isSolved() && !futureGuesses.isEmpty()) {
      String guess = futureGuesses.iterator().next();
      futureGuesses.remove(guess);

      List<LetterGuess> result = wordle.guess(guess);

      if (wordle.isSolved()) {
        return guess;
      }

      Set<Character> lettersInSolutionAndInCorrectLocation = result.stream()
              .filter(LetterGuess::isInWordAndInCorrectLocation)
              .map(lg -> lg.letter)
              .collect(Collectors.toSet());

      for (int i = 0; i < 5; i++) {
        LetterGuess letter = result.get(i);
        final int finalI = i;

        if (letter.isInWordAndInCorrectLocation) {
          // 🟩
          lettersInSolution.add(letter.letter);
          futureGuesses.removeIf(word -> word.charAt(finalI) != letter.letter);
        } else if (letter.isInWord) {
          // 🟨
          lettersInSolution.add(letter.letter);
          futureGuesses.removeIf(word -> word.charAt(finalI) == letter.letter); // since this letter is in the wrong spot
        } else {
          // ⬜️
          if (!lettersInSolution.contains(letter.letter)) {
            lettersNotInSolution.add(letter.letter);
          }
        }
      }

      // if solution is "feeds" and guess is "guess", the first "s" will return ⬜️, but there is still an "s" in the answer and in correct location
      lettersNotInSolution.removeAll(lettersInSolutionAndInCorrectLocation);

      if (!lettersInSolution.isEmpty() || !lettersNotInSolution.isEmpty()) {
        futureGuesses.removeIf(word -> {
          Set<Character> lettersInWord = word.chars().mapToObj(i -> (char) i).collect(Collectors.toSet());

          if (lettersInWord.parallelStream().anyMatch(lettersNotInSolution::contains)) {
            return true;
          }

          if (!lettersInWord.containsAll(lettersInSolution)) {
            return true;
          }

          return false;
        });
      }
    }

    // couldn't solve the puzzle
    return null;
  }
}
```

I use a `LinkedHashSet` to store future guesses because it provides predictable iteration order; that is, elements are iterated over in the same order in which they were inserted. `LinkedHashSet` also provides [constant-time performance](https://en.wikipedia.org/wiki/Time_complexity) for `add(element)`, `contains(element)`, and `remove(element)` methods, meaning that those methods will take the same amount of time whether the collection has 10 elements or 10 million elements.

```java
@Test
void test20220127_iterative_goodSeed() {
  String answer = "mount";
  Wordle wordle = new Wordle(answer);
  Solver solver = new IterativeSolver(Collections.singletonList("about"));
  assertEquals(answer, solver.solve(wordle));
  System.out.println("2022-01-27 (iterative, good seed): " + wordle.getNumGuesses());
}

@Test
void test20220127_iterative_badSeed() {
  String answer = "mount";
  Wordle wordle = new Wordle(answer);
  Solver solver = new IterativeSolver(Collections.singletonList("adieu"));
  assertEquals(answer, solver.solve(wordle));
  System.out.println("2022-01-27 (iterative, bad seed): " + wordle.getNumGuesses());
}

@Test
void test20220128_iterative_badSeed() {
  String answer = "perky";
  Wordle wordle = new Wordle(answer);
  Solver solver = new IterativeSolver(Collections.singletonList("about"));
  assertEquals(answer, solver.solve(wordle));
  System.out.println("2022-01-28 (iterative, bad seed): " + wordle.getNumGuesses());
}

@Test
void test20220128_iterative_goodSeed() {
  String answer = "perky";
  Wordle wordle = new Wordle(answer);
  Solver solver = new IterativeSolver(Collections.singletonList("pesky"));
  assertEquals(answer, solver.solve(wordle));
  System.out.println("2022-01-28 (iterative, good seed): " + wordle.getNumGuesses());
}
```
```
2022-01-27 (iterative, good seed): 4
2022-01-27 (iterative, bad seed): 5
2022-01-28 (iterative, good seed): 4
2022-01-28 (iterative, bad seed): 7
```

This approach is an improvement from the brute force approach, but there's still an issue&mdash; it will still guess the words in dictionary order If the solution is `tight`, this will guess the following words in this order after determining the answer ends in `ight`:
- `bight` (a bend in the coast or a loop in a rope)
- `dight` (archaic, past tense of dress?)
- `eight`
- `fight`
- `hight` (archaic, named)
- `light`
- `might`
- `night`
- `pight` (archaic, past tense of pitch)
- `right`
- `sight`
- `tight`

Here is a test case to demonstrate that worst-case scenario.
```java
@Test
void testTight_iterative_badSeed() {
  String answer = "tight";
  Wordle wordle = new Wordle(answer);
  wordle.setDebug(true);
  Solver solver = new IterativeSolver(Collections.singletonList("bight"));
  assertEquals(answer, solver.solve(wordle));
  System.out.println("tight (iterative, bad seed): " + wordle.getNumGuesses());
}
```

I've added some code to print the guess and result to emphasize the problem with this approach.
```
bight
⬜️🟩🟩🟩🟩
dight
⬜️🟩🟩🟩🟩
eight
⬜️🟩🟩🟩🟩
fight
⬜️🟩🟩🟩🟩
hight
⬜️🟩🟩🟩🟩
light
⬜️🟩🟩🟩🟩
might
⬜️🟩🟩🟩🟩
night
⬜️🟩🟩🟩🟩
pight
⬜️🟩🟩🟩🟩
right
⬜️🟩🟩🟩🟩
sight
⬜️🟩🟩🟩🟩
tight
🟩🟩🟩🟩🟩
tight (iterative, bad seed): 12
```

Some of those are reasonable guesses (`light` was the answer in Wordle #226), but some words in there are archaic words that are unlikely to be a Wordle answer.


# Iterative Approach Using Word Frequency
We can improve on the above solution by guessing more common words first. Peter Norvig, the Director of Research at Google, has compiled a [list](http://norvig.com/ngrams/) of the most common ~300,000 words in English. We can use this list in order to guess more common words prior to guessing obscure, archaic words.

Here is what the beginning of the list looks like:
```
about	1226734006
other	978481319
which	810514085
their	782849411
there	701170205
first	578161543
would	572644147
these	541003982
click	536746424
price	501651226
```

We don't care how often a word occurs, but do care about the order in which each word occurs.

Let's modify our iterative approach to try more common words first.
```java
class IterativeApproachWithWordFrequency implements Solver {
  private Collection<String> seeds = Collections.emptySet();
  private LinkedHashSet<String> futureGuesses = new LinkedHashSet<>();

  public IterativeSolverWithWordFrequency(Collection<String> seeds) {
    futureGuesses.addAll(seeds);
    try (InputStream is = IterativeSolverWithWordFrequency.class.getResourceAsStream("/ngrams_count_1w.txt")) {
      if (is == null) {
        throw new RuntimeException("Could not load file");
      }

      Scanner scanner = new Scanner(is);
      while (scanner.hasNext()) {
        String word = scanner.next().split("\\s+")[0];  // extract just the word from each line
        if (word.length() != 5) {
          continue;
        }

        futureGuesses.add(word);
      }
    }
    catch (IOException ioe) {
      ioe.printStackTrace();
      throw new RuntimeException("Error loading ngrams");
    }
  }

  // `solve` method is same as iterative approach
}
```

Let's test it out.
```java
@Test
void testTight_iterativeFrequency_badSeed() {
  String answer = "tight";
  Wordle wordle = new Wordle(answer);
  wordle.setDebug(true);
  Solver solver = new IterativeSolverWithWordFrequency(Collections.singletonList("bight"));
  assertEquals(answer, solver.solve(wordle));
  System.out.println("tight (iterative + frequency, bad seed): " + wordle.getNumGuesses());
}
```
```
light
⬜️🟩🟩🟩🟩
right
⬜️🟩🟩🟩🟩
night
⬜️🟩🟩🟩🟩
might
⬜️🟩🟩🟩🟩
eight
⬜️🟩🟩🟩🟩
fight
⬜️🟩🟩🟩🟩
tight
🟩🟩🟩🟩🟩
tight (iterative + frequency, bad seed): 7
```

Not bad&mdash; taking word frequency into account, we were able to improve a worst-case scenario by almost 50%. It's not enough to win the game in 6 turns, but it's enough to efficiently beat the game for most words.


# Elimination Approach
There's another approach that I haven't coded but have thought about. Some people use their second guess to guess a word that has none of the same characters as their first guess in order to get a better sense of what letters are in the answer. If the answer is `about` and their first guess is `tough` (⬜️🟨🟨⬜️⬜️), they would ignore the fact that there's an `ou` in the answer and instead guess something like `races` in order to see if there are common letters like `r` or `s` in the answer.

I'm curious how this approach stacks up against an iterative approach. [Reach out to me](https://twitter.com/ryanrishi) if you or someone else ends up writing a program that uses this approach.


Wordle has taken the world by storm. Here's a program to solve it efficiently.

Solving Wordle

AWS key pairs allow you to put your SSH key on EC2 instances when the are created, allowing you to SSH into an instance with a public key instead of a password. If you have multiple [Terraform workspaces](https://www.terraform.io/docs/state/workspaces.html), eg. `staging` and `production`, using the same key pair in multiple workspaces can cause some problems.

### The Problem
Consider the following Terraform block in the `staging` workspace:
```hcl
resource "aws_key_pair" "key_pair" {
  key_name   = "ryan"
  public_key = "ssh-rsa AAAAB...."
}

resource "aws_instance" "web" {
  key_name = aws_key_pair.key_pair.key_name
  # rest of definition
}
```

If you want to use the same key pair in the `production` workspace, you might switch workspaces and import the key pair:
```shell-session
$ terraform workspace show
production
$ terraform import -var-file=production.tfvars aws_key_pair.key_pair ryan
aws_key_pair.key_pair: Importing from ID "ryan"...
aws_key_pair.key_pair: Import prepared!
  Prepared aws_key_pair for import
aws_key_pair.key_pair: Refreshing state... [id=ryan]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
```

This will import the key pair, but there is a problem&mdash;running `terraform plan` in the `production` workspace shows that the key pair will be recreated:
```hcl
# aws_key_pair.key_pair must be replaced
+/- resource "aws_key_pair" "key_pair" {
      ~ fingerprint = "ca:0b:36:48:39:69:86:21:8a:15:da:c2:10:f0:91:a7" -> (known after apply)
      ~ id          = "ryan" -> (known after apply)
        key_name    = "ryan"
      ~ key_pair_id = "key-0ef79e456a8daa98a" -> (known after apply)
      + public_key  = "ssh-rsa AAAAB...." # forces replacement
      - tags        = {} -> null
}
```

Notice the "forces replacement" next to the `public_key`. Running `terraform import` imported the key pair but _not_ the `public_key` for the key pair&mdash;this is because the [AWS API does not expose the public key](https://github.com/terraform-providers/terraform-provider-aws/issues/1092).

We can confirm that the Terraform state does not have the public key by running `terraform state show`:
```hcl
$ terraform state show aws_key_pair.key_pair
# aws_key_pair.key_pair:
resource "aws_key_pair" "key_pair" {
    fingerprint = "ca:0b:36:48:39:69:86:21:8a:15:da:c2:10:f0:91:a7"
    id          = "ryan"
    key_name    = "ryan"
    key_pair_id = "key-0ef79e456a8daa98a"
    tags        = {}
}
```

### The Solution
We’re going to update `terraform.tfstate` to include public key. I *do not* recommend this approach if you’re using Terraform as a team, as editing the Terraform state file by hand can lead to unexpected behaviors if a change to the infrastructure is made while you’re doing this.

```hcl
$ terraform workspace show
staging
# get the public key
$ terraform state show aws_key_pair.key_pair
# aws_key_pair.key_pair:
resource "aws_key_pair" "key_pair" {
    fingerprint = "ca:0b:36:48:39:69:86:21:8a:15:da:c2:10:f0:91:a7"
    id          = "ryan"
    key_name    = "ryan"
    key_pair_id = "key-0ef79e456a8daa98a"
    public_key  = "ssh-rsa AAAAB...."
    tags        = {}
}
```

Make a note of the `public_key` from above, and then switch to the production workspace:
```shell-session
# switch to production workspace
$ terraform workspace select production
```

#### Edit Terraform state file
The next step is to get the Terraform state file. Your mileage may vary depending on which [Terraform backend](https://www.terraform.io/docs/backends/index.html) you're using. In my case, I use the S3 backend, so I need to copy the production state file so I can edit it:
```shell-session
$ aws s3 cp s3://terraform/env:/production/terraform.tfstate production.tfstate
```

Find the resource with type `aws_key_pair` named `key_pair`. Add the public key from above to the `public_key` attribute:
```diff
{
  "mode": "managed",
  "type": "aws_key_pair",
  "name": "key_pair",
  "provider": "provider.aws",
  "instances": [
    {
      "schema_version": 1,
      "attributes": {
        "fingerprint": "ca:0b:36:48:39:69:86:21:8a:15:da:c2:10:f0:91:a7",
        "id": "ryan",
        "key_name": "ryan",
        "key_name_prefix": null,
        "key_pair_id": "key-0ef79e456a8daa98a",
-       "public_key": null
+       "public_key": "ssh-rsa AAAAB....",
        "tags": {}
      },
      "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
    }
  ]
}
```

Finally, we need to re-upload the state file:
```shell-session
$ aws s3 cp production.tfstate s3://terraform/env:/production/terraform.tfstate
```

To ensure the public key was added to the key pair state, we can run `terraform plan` to verify that a second key pair won’t be created:
```shell-session
$ terraform workspace show
production
$ terraform plan -var-file=production.tfvars
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_key_pair.key_pair: Refreshing state... [id=ryan]
aws_vpc.vpc: Refreshing state... [id=vpc-066836488ca1736cc]
aws_internet_gateway.internet_gateway: Refreshing state... [id=igw-063ceed82c4a9575b]
aws_subnet.subnet: Refreshing state... [id=subnet-00223b10865ba9fcd]
aws_security_group.security_group: Refreshing state... [id=sg-00b43e691cb489f74]
aws_route_table.route_table: Refreshing state... [id=rtb-08856ff0b342d789d]
aws_instance.web: Refreshing state... [id=i-088d61e604d233686]
aws_route_table_association.subnet_association: Refreshing state... [id=rtbassoc-05bfaf2437b843a0c]
aws_eip.elastic_ip: Refreshing state... [id=eipalloc-09baa591c07a453a7]

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.
```

### Conclusion
That's it! Even though importing the AWS key pair didn't import the public key, we can manipulate the Terraform state to include the public key so that the resource is not recreated.

While I went with the above solution for my use case, there are other solutions to this problem:
- set up each Terraform workspace to use different AWS accounts
- use different key pairs for each workspace, naming them like `ryan-staging` and `ryan-production`


Importing key pairs into a Terraform workspace can lead to some unexpected results. This post outlines the problem and provides a solution by manipulating the Terraform state.

Managing the Same AWS Key Pair in Multiple Terraform Workspaces


In December, I attended [Edward Tufte's course on Presenting Data and Information](https://www.edwardtufte.com/tufte/courses) in San Francisco. Here are my key takeaways from the course.

<Blockquote
  name="Edward Tufte"
>
  Hold an open mind, but not an open head.
</Blockquote>

### The purpose of data is to make people smarter&mdash; so don't dumb it down
When presenting information, chances are the audience has already been filtered by some homogenizing process&mdash;your colleagues, people that have made a conscious decision to come to your presentation, etc.

Here is an example of an appropriate amount of data from the National Weather Service.

| ![National Weather Service](/img/tufte/weather.png "National Weather Service") |
| :--: |
| _National Weather Service. [Source](https://forecast.weather.gov/MapClick.php?lat=37.796&lon=-122.4206)_ |

Most visitors to the National Weather Service website care about the temperature and conditions (eg. mostly cloudy). This information is accurately and concisely presented via a large "59&deg;F" and image representing mostly cloudy conditions.

However, more data points are available to constituents that require more information&mdash;as a pilot, the wind speed, barometric pressure, dewpoint, and visibility are some factors that determine if I'm flying or staying on the ground (these measurements are probably from the [SFO Terminal Area Forecast](https://aviationweather.gov/taf/board?ids=KSFO&format=expanded)).


### Minimize design-figuring-out time, maximize content-reasoning time
<Blockquote
  name="Edward Tufte"
  citationLink="https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001V7">
  Diagrammatic pictures allow more focused, tighter commentary on pictures, compared to text or captions outside the picture frame.
</Blockquote>

The following pie chart has a high cognitive load, because the user has the impediment of visually encoding the chart data with the pie type.
![Favorite Pie](/img/tufte/pies.png "Favorite Pie")

To further the damage from the pie chart without direct labeling, it may be impossible to read the chart as a colorblind person or when printed or photocopied in grayscale.

On the other hand, the following scatterplot uses direct labeling to minimize content-reasoning time.

| ![Direct labeled Iris data](/img/tufte/iris-scatter.png "Direct labeled Iris data") |
| :--: |
| Direct labeled _Iris_ data, demonstrated from the [`directlabels`](https://tdhock.github.io/directlabels/) R package. [Source](https://tdhock.github.io/directlabels/) |


### It's better to be approximately right than precisely wrong
I'll let the following excerpt from Tufte's <i>Beautiful Evidence</i> speak for itself.

| ![Sparklines: Intense, Simple, Word-Sized Graphics](/img/tufte/sparklines.jpg "Sparklines: Intense, Simple, Word-Sized Graphics") |
| :--: |
| Edward Tufte. _Beautiful Evidence, p.47_. [Source](https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001OR) |

And finally, here's a fun satire on all the no-nos of presenting data and information.
<VideoContainer
  src="https://www.youtube-nocookie.com/embed/fP-7rhb-qMg"
  title="Viz-O-Matic"
/>


Some learnings from Edward Tufte's "Presenting Data and Information" course

Edward Tufte: Presenting Data and Information


import Image from 'next/image';

I recently migrated my personal website hosting from a [DigitalOcean VPS](https://www.digitalocean.com/products/droplets/) to [Amazon AWS S3 + CloudFront](https://aws.amazon.com/premiumsupport/knowledge-center/cloudfront-serve-static-website/). As part of this, I wanted to use [Terraform](https://www.terraform.io/) to manage infrastructure-as-code.

### Requirements
My website is built using [Jekyll](https://jekyllrb.com/). Jekyll is a static site generator, which makes it a perfect candidate to serve from a CDN such as CloudFront. In addition to CloudFront, I would need to store the site in S3, update my DNS to point to CloudFront, and generate a TLS certificate that CloudFront can use.

<Callout type="info">
  As of early 2021, my website is using Next.js instead of Jekyll, and no longer hosted on AWS.
</Callout>

### Building Out the Infrastructure
I had played with Terraform a few months ago, and wanted to try it out in a real-world situation. I followed [this tutorial](https://medium.com/runatlantis/hosting-our-static-site-over-ssl-with-s3-acm-cloudfront-and-terraform-513b799aec0f) to write the Terraform code needed to accomplish this task.

#### S3 Bucket
The generated static site needs to be stored somewhere. S3 is an object storage service designed with 99.999999999% (11 9’s) of data durability and relatively cheap prices.

The Terraform code blow creates a S3 bucket that allows anyone to read objects from the bucket:
```hcl
resource "aws_s3_bucket" "www" {
  bucket = var.www_domain_name
  acl    = "public-read"
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AddPerm",
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::${var.www_domain_name}/*"]
    }
  ]
}
POLICY

  website {
    index_document = "index.html"
    error_document = "404.html"
  }
}
```

#### ACM Certificate
To serve my website over HTTPS, I need a TLS certificate for my domain. [AWS ACM](https://aws.amazon.com/certificate-manager/) provides free public TLS certificates.
```hcl
resource "aws_acm_certificate" "certificate" {
  domain_name               = "*.${var.root_domain_name}"
  validation_method         = "EMAIL"
  subject_alternative_names = [var.www_domain_name]
}
```

#### CloudFront CDN
A CDN caches content at the edges of a network, which helps reduce load on the origin server and reduce and latency due to geographic location. When somebody visits my website, the CDN checks if has the file they are requesting, and if not, will fetch it from the S3 bucket.
```hcl
resource "aws_cloudfront_distribution" "www_distribution" {
  origin {
    domain_name = aws_s3_bucket.www.website_endpoint
    origin_id   = var.www_domain_name

    custom_origin_config {
      http_port               = "80"
      https_port              = "443"
      origin_protocol_policy  = "http-only"
      origin_ssl_protocols    = ["TLSv1", "TLSv1.1", "TLSv1.2"]
    }
  }

  enabled             = true
  default_root_object = "index.html"

  default_cache_behavior {
    viewer_protocol_policy  = "redirect-to-https"
    compress                = true
    allowed_methods         = ["GET", "HEAD"]
    cached_methods          = ["GET", "HEAD"]
    target_origin_id        = var.www_domain_name
    min_ttl                 = 0
    default_ttl             = 86400
    max_ttl                 = 31536000

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }
  }

  aliases = [var.www_domain_name]

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    acm_certificate_arn = aws_acm_certificate.certificate.arn
    ssl_support_method  = "sni-only"
  }
}
```

#### Domain Name
I use [Google Domains](https://domains.google/) as my DNS provider. In order to route traffic from `www.ryanrishi.com` to the CloudFront distribution, I created a [AWS Route 53](https://aws.amazon.com/route53/) zone and record:
```hcl
resource "aws_route53_zone" "zone" {
  name = var.root_domain_name
}

resource "aws_route53_record" "www" {
  zone_id = aws_route53_zone.zone.zone_id
  name    = var.www_domain_name
  type    = "A"

  alias {
    name                   = aws_cloudfront_distribution.www_distribution.domain_name
    zone_id                = aws_cloudfront_distribution.www_distribution.hosted_zone_id
    evaluate_target_health = false
  }
}
```

While DNS propagates, I can get the AWS nameservers from the Terraform state so I can configure Google Domains:
```hcl
$ terraform state show aws_route53_zone.zone
# aws_route53_zone.zone:
resource "aws_route53_zone" "zone" {
    comment       = "Managed by Terraform"
    force_destroy = false
    id            = "Z1ETY..."
    name          = "ryanrishi.com."
    name_servers  = [
        "ns-1188.awsdns-20.org",
        "ns-161.awsdns-20.com",
        "ns-1993.awsdns-57.co.uk",
        "ns-689.awsdns-22.net",
    ]
    tags          = {}
    zone_id       = "Z1ETY..."
}
```

### Deploying the Website
The final step is to deploy the website to the S3 bucket. I use [Travis CI](https://travis-ci.org/) to build and deploy my code.

This snippet of code will deploy to the preconfigured S3 bucket whenever a change is pushed to my `master` branch:
```yaml
deploy:
  provider: s3
  skip_cleanup: true
  access_key_id: $AWS_ACCESS_KEY_ID
  secret_access_key: $AWS_SECRET_ACCESS_KEY
  bucket: $S3_BUCKET
  local-dir: _site
  acl: public_read
  on:
    repo: ryanrishi/website
    branch: master
```

After deploying, I want to invalidate the CloudFront cache. If I skipped this step, changes to my website may not show up for 24 hours because the `default_ttl` for the CloudFront distribution is set to 86400 seconds.
```yaml
after_deploy:
  - aws configure set preview.cloudfront true
  - aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRIBUTION_ID --paths "/*"
```

You can see the full Travis CI configuration [here](https://github.com/ryanrishi/website/commit/fcc70801ece6d09b494f6026476213696eb59e65#diff-354f30a63fb0907d4ad57269548329e3).

### Benchmark
Moving from my own VPS in New York to a CDN shaved the load time from 1.2s to 200ms. Visitors in California used to have higher latency due to cross-country travel, but now that it is served via a CDN, geographic location plays a much smaller factor&mdash;the content will be served from the closest edge of the network topology.

| ![Waterfall when serving from a VPS](/img/waterfall-vps.png "Waterfall when serving from a VPS") |
| :--: |
| _Waterfall when serving from a VPS_ |

| ![Waterfall when serving from a CDN](/img/waterfall-cdn.png "Waterfall when serving from a CDN") |
| :--: |
| _Waterfall when serving from a CDN_ |


### Next Steps
I'm currently running `terraform apply` on my laptop&mdash; I'd rather run that in a continuous integration pipeline.

Using Terraform was challenging at first, but in the end it seems like a great way to track infrastructure changes. I'd like to continue learning how to manage infrastructure using Terraform, including migrating some other VPSs to EC2/Kubernetes.

You can see the final Terraform code [here](https://github.com/ryanrishi/devops/tree/f858d1448d5105df3a231c91fc60882b82d4d05a).


I recently migrated my personal website infrastructure from a VPS to AWS S3 + CloudFront, with infrastructure managed by Terraform.

Migrating My Website to AWS Using Terraform

For those of you who know me well, you know that I should not be singing&mdash;I should stick to my guns, which are percussion and recording. Even though the extent of my vocal ability rarely leaves the confines of my shower, I ended up auditioning for an a cappella group at school, and to my surprise, I ended up making the cut!

Five of the a cappella groups on campus decided to join forces and make the first-ever Santa Clara University A Cappella Christmas Album, and I was to produce that album.

### Preparation
To prepare, I listened to *a lot* of a cappella&mdash;so much that YouTube still suggests a cappella videos on a regular basis. I researched best practices in recording multiple voices, and found out that isolating voices is very important so that autotune can work its magic on a clean signal. I discussed my plan of action with the recording teacher at school, and he told me that he has a friend that produces a cappella albums. Cool! Everything was on track. I went home and Googled the name of his friend, [Bill Hare](http://billhareacappella.com). It turns out he produces some of the best-known a cappella artists, from Pentatonix to Street Corner Symphony.

Further discussion with the recording teacher led to a field trip of about eighteen people (a mix of recording students and a cappella members) to the studio of Bill Hare. While there, the group learned about Bill's experience with a cappella groups, including recording practices, mixing/mastering techniques, and live performance techniques.

One interesting practice Bill does is to record vocalists using a MIDI backing track. When recording multiple vocalists, I had previously had them sing at once and use that recording as a backing track as I record individual voices. However, with that practice, vocalists can be a few cents off from the true pitch, which will lead to them being off from the true pitch of the song when they record later. With MIDI, however, the sounds are exactly in tune. It never crossed my mind before, but in hindsight, it makes a lot of sense.

### On the first day of recording, my true love gave to me...
...the largest a cappella group on campus&mdash;17 members! I booked the school studio for four hours, and got there an hour early to set up three microphones in an isolation booth. The back of the microphones faced each other, since the null point of a cardioid microphone is in the back. There was a little bit of bleed into each microphone, but it was an acceptable amount.

I recorded the remaining groups over the next five days. On the sixth day of recording, I recorded the last group. It was time to start editing and mixing!

### Editing/Mixing
I had already started to mix the groups I recorded first. My process was as follows:

1. bounce out each individual track&mdash;this creates a new audio file based on the best composite take, thus keeping the original audio intact
2. edit individual tracks&mdash;I started by lining up each note for rhythm, then applied autotune where necessary. This also included standard mixing techniques, such as EQ and compression
3. bus each voice part to their respective sections&mdash;this resulted in 4 to 8 busses (soprano 1, soprano 2, alto 1, alto 2, etc.). This makes it easier to get a good section blend and then mix SATB voices
4. create reverb for the ensemble&mdash;each section bus had an auxiliary send to a reverb track. I wasn't sure if I should add reverb at the end of the mixing stage or at the beginning of the mastering phase. I experimented with both and had better results with keeping it at the end of the mixing phase.

My final mix levels were somewhere in the -6 dB to -3 dB range. I knew that I would end up putting some mastering compression to make the overall mix more lively.

### Mastering
For mastering, I wanted it to sound pretty produced and not like a live a cappella performance. My reasoning for this is because in live performance, it's much more theatrical and the audio is only one component of the overall performance. For studio a cappella recordings, I found that there's a fair amount of compression. Many of the reference tracks I used in this project had RMS levels in the -10 dB to -5 dB range. That's pretty high, which means that there's a good deal of mastering compression on the tracks.

I ended up using the [Waves L3 Ultramaximizer](http://www.waves.com/plugins/l3-ultramaximizer) on a mastering session with a -9 dB threshold and a -0.1 dB output ceiling. This added a lot of compression, which makes the tracks sound more intimate and lively. It applies compression to the top-end of the audio signal, thus allowing the track to be played louder without clipping. I felt a little funny doing this, because I felt like I was contributing to the [Loudness Wars](http://dynamicrangeday.co.uk/about/). (I'm working on a blog post about the Loudness Wars, stay tuned!)

I spent the majority of my Thanksgiving break working on this album. I had to have everything done by the day after Thanksgiving, as that's when the leaders planned to burn the CDs. There were a few late nights working on this album, but it was all worth it since I enjoy working on projects like this. On the twelfth day of the album (the day after Thanksgiving), everything was done! I uploaded the mastered tracks to Google Drive, and my work here was done.

### Reflection
Overall, it was a great project to work on. It was great recording, editing, and mixing several of my friends and seeing it become something I could be proud of. If I was to do it all over again, I would do a few things differently:

* start this earlier&mdash;12 days is not enough to record, edit, mix, and master 9 songs
* have the groups make the MIDI backing tracks for their own arrangements
* work with someone else in the editing/mixing process&mdash;you start to hear (more) voices in your head when mixing a cappella for hours on end. I spend a little more than 60 hours on this project, the majority of which was spend in editing and mixing.

The leaders of the a cappella groups started distributing the CDs the Monday after Thanksgiving. 150 CDs were ordered, and they sold out within 8 days! I was stoked that so many people were interested in this album. After all, CDs do make great stocking stuffers!

This CD was a great stepping stone for me, because I'm also working on producing [my band's first album](http://facebook.com/JaggedLightBand), which is set to release early 2016.

Thank you do Bill Hare, Jeff Hanson, and Dean-Austin Mayor for your help on the album, and to all the a cappella groups on campus for contributing to the album!

Thanks for reading!

<div className="col-lg-8 col-lg-offset-2 col-md-8 col-md-offset-2">
  <iframe width="100%" height="450" scrolling="no" frameBorder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/239762077&amp;auto_play=false&amp;hide_related=false&amp;show_comments=true&amp;show_user=true&amp;show_reposts=false&amp;visual=true"></iframe>
</div>


Recording and producing a Christmas a cappella album in two weeks

Solving Wordle

Managing the Same AWS Key Pair in Multiple Terraform Workspaces

Edward Tufte: Presenting Data and Information

Migrating My Website to AWS Using Terraform

The Twelve Days of an A Cappella Christmas Album