[Wordle](https://www.powerlanguage.co.uk/wordle/) has taken the world by storm. Here's a program to solve it efficiently.

The rules behind the game are straightforward. For each letter in the guess, the following rules apply:
- if the letter is in the correct spot, show a 🟩
- if the letter is in the word but not in the correct spot, show a 🟨
- if the letter is not in the word, show a ⬜️

For example, WEARY means that `W` is in the word and in the correct spot. PILLS means that `I` is in the word but in the wrong spot. VAGUE means that `U` is not in the word.

The examples below are available on [Github](https://github.com/ryanrishi/wordle-solver).

# Recreating Wordle For Testing
Here's a class to hold the result of each letter in the guess.
```java
@Getter
@RequiredArgsConstructor
class LetterGuess {
 private @NonNull char letter;
 private boolean isInWord;
 private boolean isInWordAndInCorrectLocation;
}
```

Here's a representation of the game. Each game has an answer and provides a way to guess that answer.
```java
class Wordle {
 public static final int WORD_LENGTH = 5;

 private final String answer;

 @Getter
 private int numGuesses = 0;

 @Getter
 private boolean solved;

 public Wordle(String answer) {
 this.answer = answer.toLowerCase();
 }

 /**
 * Something that shouldn't stop the game, just something wrong with your guess.
 */
 static class BadGuessException extends IllegalArgumentException {
 public BadGuessException(String msg) {
 super(msg);
 }
 }

 public List<LetterGuess> guess(String guess) {
 if (guess.length() != WORD_LENGTH) {
 throw new BadGuessException(String.format("Guess must be %d characters long", WORD_LENGTH));
 }

 guess = guess.toLowerCase();
 ++numGuesses;

 if (answer.equals(guess)) {
 // victory
 solved = true;

 return Collections.emptyList();
 }

 // use a map of letter in answer => number of times this letter occurs in the answer
 Map<Character, Integer> letterFrequencies = getLetterFrequencyMap();
 List<LetterGuess> result = new ArrayList<>();

 // handle letters in correct location first
 for (int i = 0; i < guess.length(); i++) {
 char c = guess.charAt(i);
 LetterGuess letterGuess = new LetterGuess(c);

 if (c == answer.charAt(i)) {
 letterGuess.isInWord = true;
 letterGuess.isInWordAndInCorrectLocation = true;
 letterFrequencies.put(c, letterFrequencies.get(c) - 1);
 }

 result.add(letterGuess);
 }

 for (int i = 0; i < 5; i++) {
 char c = guess.charAt(i);

 if (letterFrequencies.getOrDefault(c, 0) > 0) {
 result.get(i).isInWord = true;
 letterFrequencies.put(c, letterFrequencies.get(c) - 1);
 }
 }

 return result;
 }

 private Map<Character, Integer> getLetterFrequencyMap() {
 Map<Character, Integer> letterFrequencies = new HashMap<>();
 for (char c : answer.toCharArray()) {
 letterFrequencies.putIfAbsent(c, 0);
 letterFrequencies.put(c, letterFrequencies.get(c) + 1);
 }

 return letterFrequencies;
 }
}
```

We can also create a driver program to try out the game:
```java
public static void main(String[] args) {
 Wordle wordle = new Wordle("price");

 Scanner scanner = new Scanner(System.in);
 while (!wordle.isSolved()) {
 String guess = scanner.nextLine();
 List<LetterGuess> result = wordle.guess(guess);

 System.out.println(result.stream().map(l -> {
 if (l.isInWordAndInCorrectLocation) {
 return "🟩";
 } else if (l.isInWord) {
 return "🟨";
 } else {
 return "⬜️";
 }
 }).collect(Collectors.joining()));
 }
}
```

```
Welcome to Wordle. Enter your guess below.
(1/6) > asdflkjasfdbpojaerb
Guess must be 5 characters long
(1/6) > about
⬜️⬜️⬜️⬜️⬜️
(2/6) > prize
🟩🟩🟩⬜️🟩
```

It won't cover cases like guesses that aren't 5 letters or aren't in the dictionary, but for our purposes, this will get the job done.

Before we start implementing a program to solve this, let's define an interface that each type of solver will implement.
```java
interface Solver {
 /**
 * @param wordle the game
 * @return the answer to the game, or null if the game could not be solved
 */
 String solve(Wordle wordle);
}
```

# Brute Force Approach
A naive approach is to check every single 5-letter word in the dictionary until the game is solved. In order to implement this, I've pulled in a dictionary from [dwyl/english-words](https://github.com/dwyl/english-words/blob/master/words_alpha.txt).

```java
class BruteForceSolver implements Solver {
 private Set<String> dictionary = new TreeSet<>();

 public BruteForceSolver() {
 // load the dictionary
 try (InputStream is = BruteForceSolver.class.getResourceAsStream("/words_alpha.txt")) {
 if (is == null) {
 throw new RuntimeException("Could not load file");
 }

 Scanner scanner = new Scanner(is);
 while (scanner.hasNext()) {
 String word = scanner.next();
 if (word.length() != 5) {
 // skip over words that aren't 5 letters long
 continue;
 }

 dictionary.add(word);
 }
 } catch (IOException ioe) {
 ioe.printStackTrace();
 throw new RuntimeException("Error loading dictionary");
 }
 }

 @Override
 String solve(Wordle wordle) {
 for (String word : dictionary) {
 wordle.guess(word);

 if (wordle.isSolved()) {
 return word;
 }
 }

 // couldn't solve the game
 return null;
 }
}
```

Let's test it out.
```java
@Test
void testBruteForce_about() {
 String answer = "about";
 Wordle wordle = new Wordle(answer);
 Solver solver = new BruteForceSolver();
 assertEquals(answer, solver.solve(wordle));
 System.out.println("brute force - about: " + wordle.getNumGuesses());
}

@Test
void testBruteForce_zorro() {
 String answer = "zorro";
 Wordle wordle = new Wordle(answer);
 Solver solver = new BruteForceSolver();
 assertEquals(answer, solver.solve(wordle));
 System.out.println("brute force - zorro: " + wordle.getNumGuesses());
}
```
```
brute force - about: 60
brute force - zorro: 15907
```

Guessing `about` took 60 guesses since it was at the start of the dictionary, but guessing `zorro` required 15,907 guesses because this program has to guess almost every word in the dictionary before finding the solution.


# Iterative Approach
To improve, we can leverage the result from each guess in order to inform our future guesses.

Thinking back to the rules of the game, we can improve our solution by doing the following:
- If the letter is in the word and in the correct location (🟩), filter out any words that don't have that letter in that location
- If the letter is in the word but _not_ in the correct location (🟨), filter out any words that don't have that letter in the word
- If the letter is not in the word (⬜️), filter out any words that _do_ have that letter

For example, if the answer is `light` and our first guess is `grout` (🟨⬜️⬜️⬜️🟩), we would do the following:
- Filter out any words that don't end in `t` (🟩). This eliminates words like `horse` and `biker` but keeps words like `blast` and `pleat`
- Filter out any words that don't have a `g` (🟨). This eliminates words like `built` and `bulbs` but keeps words like `egret` and `ought`
- Filter out any words that contain letters that aren't in the solution (⬜️). In this example, eliminate any words that contain a `r`, `o`, or `u`
- Guess the first word in the set of remaining words
- Continue using the result of each guess to filter out future guesses

Let's put this into code.
```java
class IterativeSolver implements Solver {
 @Override
 public String solve(Wordle wordle) {
 Set<String> futureGuesses = new LinkedHashSet<>(seeds);
 futureGuesses.addAll(dictionary);
 Set<Character> lettersNotInSolution = new HashSet<>();
 List<Character> lettersInSolution = new ArrayList<>();

 while (!wordle.isSolved() && !futureGuesses.isEmpty()) {
 String guess = futureGuesses.iterator().next();
 futureGuesses.remove(guess);

 List<LetterGuess> result = wordle.guess(guess);

 if (wordle.isSolved()) {
 return guess;
 }

 Set<Character> lettersInSolutionAndInCorrectLocation = result.stream()
 .filter(LetterGuess::isInWordAndInCorrectLocation)
 .map(lg -> lg.letter)
 .collect(Collectors.toSet());

 for (int i = 0; i < 5; i++) {
 LetterGuess letter = result.get(i);
 final int finalI = i;

 if (letter.isInWordAndInCorrectLocation) {
 // 🟩
 lettersInSolution.add(letter.letter);
 futureGuesses.removeIf(word -> word.charAt(finalI) != letter.letter);
 } else if (letter.isInWord) {
 // 🟨
 lettersInSolution.add(letter.letter);
 futureGuesses.removeIf(word -> word.charAt(finalI) == letter.letter); // since this letter is in the wrong spot
 } else {
 // ⬜️
 if (!lettersInSolution.contains(letter.letter)) {
 lettersNotInSolution.add(letter.letter);
 }
 }
 }

 // if solution is "feeds" and guess is "guess", the first "s" will return ⬜️, but there is still an "s" in the answer and in correct location
 lettersNotInSolution.removeAll(lettersInSolutionAndInCorrectLocation);

 if (!lettersInSolution.isEmpty() || !lettersNotInSolution.isEmpty()) {
 futureGuesses.removeIf(word -> {
 Set<Character> lettersInWord = word.chars().mapToObj(i -> (char) i).collect(Collectors.toSet());

 if (lettersInWord.parallelStream().anyMatch(lettersNotInSolution::contains)) {
 return true;
 }

 if (!lettersInWord.containsAll(lettersInSolution)) {
 return true;
 }

 return false;
 });
 }
 }

 // couldn't solve the puzzle
 return null;
 }
}
```

I use a `LinkedHashSet` to store future guesses because it provides predictable iteration order; that is, elements are iterated over in the same order in which they were inserted. `LinkedHashSet` also provides [constant-time performance](https://en.wikipedia.org/wiki/Time_complexity) for `add(element)`, `contains(element)`, and `remove(element)` methods, meaning that those methods will take the same amount of time whether the collection has 10 elements or 10 million elements.

```java
@Test
void test20220127_iterative_goodSeed() {
 String answer = "mount";
 Wordle wordle = new Wordle(answer);
 Solver solver = new IterativeSolver(Collections.singletonList("about"));
 assertEquals(answer, solver.solve(wordle));
 System.out.println("2022-01-27 (iterative, good seed): " + wordle.getNumGuesses());
}

@Test
void test20220127_iterative_badSeed() {
 String answer = "mount";
 Wordle wordle = new Wordle(answer);
 Solver solver = new IterativeSolver(Collections.singletonList("adieu"));
 assertEquals(answer, solver.solve(wordle));
 System.out.println("2022-01-27 (iterative, bad seed): " + wordle.getNumGuesses());
}

@Test
void test20220128_iterative_badSeed() {
 String answer = "perky";
 Wordle wordle = new Wordle(answer);
 Solver solver = new IterativeSolver(Collections.singletonList("about"));
 assertEquals(answer, solver.solve(wordle));
 System.out.println("2022-01-28 (iterative, bad seed): " + wordle.getNumGuesses());
}

@Test
void test20220128_iterative_goodSeed() {
 String answer = "perky";
 Wordle wordle = new Wordle(answer);
 Solver solver = new IterativeSolver(Collections.singletonList("pesky"));
 assertEquals(answer, solver.solve(wordle));
 System.out.println("2022-01-28 (iterative, good seed): " + wordle.getNumGuesses());
}
```
```
2022-01-27 (iterative, good seed): 4
2022-01-27 (iterative, bad seed): 5
2022-01-28 (iterative, good seed): 4
2022-01-28 (iterative, bad seed): 7
```

This approach is an improvement from the brute force approach, but there's still an issue&mdash; it will still guess the words in dictionary order If the solution is `tight`, this will guess the following words in this order after determining the answer ends in `ight`:
- `bight` (a bend in the coast or a loop in a rope)
- `dight` (archaic, past tense of dress?)
- `eight`
- `fight`
- `hight` (archaic, named)
- `light`
- `might`
- `night`
- `pight` (archaic, past tense of pitch)
- `right`
- `sight`
- `tight`

Here is a test case to demonstrate that worst-case scenario.
```java
@Test
void testTight_iterative_badSeed() {
 String answer = "tight";
 Wordle wordle = new Wordle(answer);
 wordle.setDebug(true);
 Solver solver = new IterativeSolver(Collections.singletonList("bight"));
 assertEquals(answer, solver.solve(wordle));
 System.out.println("tight (iterative, bad seed): " + wordle.getNumGuesses());
}
```

I've added some code to print the guess and result to emphasize the problem with this approach.
```
bight
⬜️🟩🟩🟩🟩
dight
⬜️🟩🟩🟩🟩
eight
⬜️🟩🟩🟩🟩
fight
⬜️🟩🟩🟩🟩
hight
⬜️🟩🟩🟩🟩
light
⬜️🟩🟩🟩🟩
might
⬜️🟩🟩🟩🟩
night
⬜️🟩🟩🟩🟩
pight
⬜️🟩🟩🟩🟩
right
⬜️🟩🟩🟩🟩
sight
⬜️🟩🟩🟩🟩
tight
🟩🟩🟩🟩🟩
tight (iterative, bad seed): 12
```

Some of those are reasonable guesses (`light` was the answer in Wordle #226), but some words in there are archaic words that are unlikely to be a Wordle answer.


# Iterative Approach Using Word Frequency
We can improve on the above solution by guessing more common words first. Peter Norvig, the Director of Research at Google, has compiled a [list](http://norvig.com/ngrams/) of the most common ~300,000 words in English. We can use this list in order to guess more common words prior to guessing obscure, archaic words.

Here is what the beginning of the list looks like:
```
about	1226734006
other	978481319
which	810514085
their	782849411
there	701170205
first	578161543
would	572644147
these	541003982
click	536746424
price	501651226
```

We don't care how often a word occurs, but do care about the order in which each word occurs.

Let's modify our iterative approach to try more common words first.
```java
class IterativeApproachWithWordFrequency implements Solver {
 private Collection<String> seeds = Collections.emptySet();
 private LinkedHashSet<String> futureGuesses = new LinkedHashSet<>();

 public IterativeSolverWithWordFrequency(Collection<String> seeds) {
 futureGuesses.addAll(seeds);
 try (InputStream is = IterativeSolverWithWordFrequency.class.getResourceAsStream("/ngrams_count_1w.txt")) {
 if (is == null) {
 throw new RuntimeException("Could not load file");
 }

 Scanner scanner = new Scanner(is);
 while (scanner.hasNext()) {
 String word = scanner.next().split("\\s+")[0]; // extract just the word from each line
 if (word.length() != 5) {
 continue;
 }

 futureGuesses.add(word);
 }
 }
 catch (IOException ioe) {
 ioe.printStackTrace();
 throw new RuntimeException("Error loading ngrams");
 }
 }

 // `solve` method is same as iterative approach
}
```

Let's test it out.
```java
@Test
void testTight_iterativeFrequency_badSeed() {
 String answer = "tight";
 Wordle wordle = new Wordle(answer);
 wordle.setDebug(true);
 Solver solver = new IterativeSolverWithWordFrequency(Collections.singletonList("bight"));
 assertEquals(answer, solver.solve(wordle));
 System.out.println("tight (iterative + frequency, bad seed): " + wordle.getNumGuesses());
}
```
```
light
⬜️🟩🟩🟩🟩
right
⬜️🟩🟩🟩🟩
night
⬜️🟩🟩🟩🟩
might
⬜️🟩🟩🟩🟩
eight
⬜️🟩🟩🟩🟩
fight
⬜️🟩🟩🟩🟩
tight
🟩🟩🟩🟩🟩
tight (iterative + frequency, bad seed): 7
```

Not bad&mdash; taking word frequency into account, we were able to improve a worst-case scenario by almost 50%. It's not enough to win the game in 6 turns, but it's enough to efficiently beat the game for most words.


# Elimination Approach
There's another approach that I haven't coded but have thought about. Some people use their second guess to guess a word that has none of the same characters as their first guess in order to get a better sense of what letters are in the answer. If the answer is `about` and their first guess is `tough` (⬜️🟨🟨⬜️⬜️), they would ignore the fact that there's an `ou` in the answer and instead guess something like `races` in order to see if there are common letters like `r` or `s` in the answer.

I'm curious how this approach stacks up against an iterative approach. [Reach out to me](https://twitter.com/ryanrishi) if you or someone else ends up writing a program that uses this approach.

Wordle has taken the world by storm. Here's a program to solve it efficiently.

Solving Wordle

AWS key pairs allow you to put your SSH key on EC2 instances when the are created, allowing you to SSH into an instance with a public key instead of a password. If you have multiple [Terraform workspaces](https://www.terraform.io/docs/state/workspaces.html), eg. `staging` and `production`, using the same key pair in multiple workspaces can cause some problems.

### The Problem
Consider the following Terraform block in the `staging` workspace:
```hcl
resource "aws_key_pair" "key_pair" {
  key_name   = "ryan"
  public_key = "ssh-rsa AAAAB...."
}

resource "aws_instance" "web" {
  key_name = aws_key_pair.key_pair.key_name
  # rest of definition
}
```

If you want to use the same key pair in the `production` workspace, you might switch workspaces and import the key pair:
```shell-session
$ terraform workspace show
production
$ terraform import -var-file=production.tfvars aws_key_pair.key_pair ryan
aws_key_pair.key_pair: Importing from ID "ryan"...
aws_key_pair.key_pair: Import prepared!
  Prepared aws_key_pair for import
aws_key_pair.key_pair: Refreshing state... [id=ryan]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
```

This will import the key pair, but there is a problem&mdash;running `terraform plan` in the `production` workspace shows that the key pair will be recreated:
```hcl
# aws_key_pair.key_pair must be replaced
+/- resource "aws_key_pair" "key_pair" {
      ~ fingerprint = "ca:0b:36:48:39:69:86:21:8a:15:da:c2:10:f0:91:a7" -> (known after apply)
      ~ id          = "ryan" -> (known after apply)
        key_name    = "ryan"
      ~ key_pair_id = "key-0ef79e456a8daa98a" -> (known after apply)
      + public_key  = "ssh-rsa AAAAB...." # forces replacement
      - tags        = {} -> null
}
```

Notice the "forces replacement" next to the `public_key`. Running `terraform import` imported the key pair but _not_ the `public_key` for the key pair&mdash;this is because the [AWS API does not expose the public key](https://github.com/terraform-providers/terraform-provider-aws/issues/1092).

We can confirm that the Terraform state does not have the public key by running `terraform state show`:
```hcl
$ terraform state show aws_key_pair.key_pair
# aws_key_pair.key_pair:
resource "aws_key_pair" "key_pair" {
    fingerprint = "ca:0b:36:48:39:69:86:21:8a:15:da:c2:10:f0:91:a7"
    id          = "ryan"
    key_name    = "ryan"
    key_pair_id = "key-0ef79e456a8daa98a"
    tags        = {}
}
```

### The Solution
We’re going to update `terraform.tfstate` to include public key. I *do not* recommend this approach if you’re using Terraform as a team, as editing the Terraform state file by hand can lead to unexpected behaviors if a change to the infrastructure is made while you’re doing this.

```hcl
$ terraform workspace show
staging
# get the public key
$ terraform state show aws_key_pair.key_pair
# aws_key_pair.key_pair:
resource "aws_key_pair" "key_pair" {
    fingerprint = "ca:0b:36:48:39:69:86:21:8a:15:da:c2:10:f0:91:a7"
    id          = "ryan"
    key_name    = "ryan"
    key_pair_id = "key-0ef79e456a8daa98a"
    public_key  = "ssh-rsa AAAAB...."
    tags        = {}
}
```

Make a note of the `public_key` from above, and then switch to the production workspace:
```shell-session
# switch to production workspace
$ terraform workspace select production
```

#### Edit Terraform state file
The next step is to get the Terraform state file. Your mileage may vary depending on which [Terraform backend](https://www.terraform.io/docs/backends/index.html) you're using. In my case, I use the S3 backend, so I need to copy the production state file so I can edit it:
```shell-session
$ aws s3 cp s3://terraform/env:/production/terraform.tfstate production.tfstate
```

Find the resource with type `aws_key_pair` named `key_pair`. Add the public key from above to the `public_key` attribute:
```diff
{
  "mode": "managed",
  "type": "aws_key_pair",
  "name": "key_pair",
  "provider": "provider.aws",
  "instances": [
    {
      "schema_version": 1,
      "attributes": {
        "fingerprint": "ca:0b:36:48:39:69:86:21:8a:15:da:c2:10:f0:91:a7",
        "id": "ryan",
        "key_name": "ryan",
        "key_name_prefix": null,
        "key_pair_id": "key-0ef79e456a8daa98a",
-       "public_key": null
+       "public_key": "ssh-rsa AAAAB....",
        "tags": {}
      },
      "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
    }
  ]
}
```

Finally, we need to re-upload the state file:
```shell-session
$ aws s3 cp production.tfstate s3://terraform/env:/production/terraform.tfstate
```

To ensure the public key was added to the key pair state, we can run `terraform plan` to verify that a second key pair won’t be created:
```shell-session
$ terraform workspace show
production
$ terraform plan -var-file=production.tfvars
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_key_pair.key_pair: Refreshing state... [id=ryan]
aws_vpc.vpc: Refreshing state... [id=vpc-066836488ca1736cc]
aws_internet_gateway.internet_gateway: Refreshing state... [id=igw-063ceed82c4a9575b]
aws_subnet.subnet: Refreshing state... [id=subnet-00223b10865ba9fcd]
aws_security_group.security_group: Refreshing state... [id=sg-00b43e691cb489f74]
aws_route_table.route_table: Refreshing state... [id=rtb-08856ff0b342d789d]
aws_instance.web: Refreshing state... [id=i-088d61e604d233686]
aws_route_table_association.subnet_association: Refreshing state... [id=rtbassoc-05bfaf2437b843a0c]
aws_eip.elastic_ip: Refreshing state... [id=eipalloc-09baa591c07a453a7]

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.
```

### Conclusion
That's it! Even though importing the AWS key pair didn't import the public key, we can manipulate the Terraform state to include the public key so that the resource is not recreated.

While I went with the above solution for my use case, there are other solutions to this problem:
- set up each Terraform workspace to use different AWS accounts
- use different key pairs for each workspace, naming them like `ryan-staging` and `ryan-production`


Importing key pairs into a Terraform workspace can lead to some unexpected results. This post outlines the problem and provides a solution by manipulating the Terraform state.

Managing the Same AWS Key Pair in Multiple Terraform Workspaces

In December, I attended [Edward Tufte's course on Presenting Data and Information](https://www.edwardtufte.com/tufte/courses) in San Francisco. Here are my key takeaways from the course.

<Blockquote
 name="Edward Tufte"
>
 Hold an open mind, but not an open head.
</Blockquote>

### The purpose of data is to make people smarter&mdash; so don't dumb it down
When presenting information, chances are the audience has already been filtered by some homogenizing process&mdash;your colleagues, people that have made a conscious decision to come to your presentation, etc.

Here is an example of an appropriate amount of data from the National Weather Service.

| ![National Weather Service](/img/tufte/weather.png "National Weather Service") |
| :--: |
| _National Weather Service. [Source](https://forecast.weather.gov/MapClick.php?lat=37.796&lon=-122.4206)_ |

Most visitors to the National Weather Service website care about the temperature and conditions (eg. mostly cloudy). This information is accurately and concisely presented via a large "59&deg;F" and image representing mostly cloudy conditions.

However, more data points are available to constituents that require more information&mdash;as a pilot, the wind speed, barometric pressure, dewpoint, and visibility are some factors that determine if I'm flying or staying on the ground (these measurements are probably from the [SFO Terminal Area Forecast](https://aviationweather.gov/taf/board?ids=KSFO&format=expanded)).


### Minimize design-figuring-out time, maximize content-reasoning time
<Blockquote
 name="Edward Tufte"
 citationLink="https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001V7">
 Diagrammatic pictures allow more focused, tighter commentary on pictures, compared to text or captions outside the picture frame.
</Blockquote>

The following pie chart has a high cognitive load, because the user has the impediment of visually encoding the chart data with the pie type.
![Favorite Pie](/img/tufte/pies.png "Favorite Pie")

To further the damage from the pie chart without direct labeling, it may be impossible to read the chart as a colorblind person or when printed or photocopied in grayscale.

On the other hand, the following scatterplot uses direct labeling to minimize content-reasoning time.

| ![Direct labeled Iris data](/img/tufte/iris-scatter.png "Direct labeled Iris data") |
| :--: |
| Direct labeled _Iris_ data, demonstrated from the [`directlabels`](https://tdhock.github.io/directlabels/) R package. [Source](https://tdhock.github.io/directlabels/) |


### It's better to be approximately right than precisely wrong
I'll let the following excerpt from Tufte's Beautiful Evidence speak for itself.

| ![Sparklines: Intense, Simple, Word-Sized Graphics](/img/tufte/sparklines.jpg "Sparklines: Intense, Simple, Word-Sized Graphics") |
| :--: |
| Edward Tufte. _Beautiful Evidence, p.47_. [Source](https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001OR) |

And finally, here's a fun satire on all the no-nos of presenting data and information.
<VideoContainer
 src="https://www.youtube-nocookie.com/embed/fP-7rhb-qMg"
 title="Viz-O-Matic"
/>

Some learnings from Edward Tufte's "Presenting Data and Information" course

Edward Tufte: Presenting Data and Information


This module creates the necessary resources to run [paperless-ng](https://paperless-ng.readthedocs.io/en/latest/index.html) on AWS.

The resources managed by this module are:
- Aurora RDS (serverless, Postgres)
- ElastiCache (Redis)
- EC2
- Security groups, EBS attachments, passwords, etc. for the above

Source code available [here](https://github.com/ryanrishi/paperless-ng-terraform).


Terraform module to set up paperless-ng on AWS.

paperless-ng Terraform

import LoudnessWars from '../../components/projects/loudness-wars';

The [Loudness Wars](https://en.wikipedia.org/wiki/Loudness_war) refers to the dramatic increase in the perceived loudness of a song over the past 40 years. The loud parts haven't necessarily become louder, but instead the effective loudness (called the root mean square, or RMS) of recorded music has increased. This has been done by compressing the loud parts of an audio signal, and then increasing the overall loudness of the signal. The loud parts are now more or less as loud as they were before dynamic compression, but now that overall loudness has increased.

I wanted to quantify this notion that music is getting increasingly louder. I did so by scraping the year-end Billboard charts for the past 50 years, searching Spotify for those tracks, and using [Spotify's audio analysis API](https://developer.spotify.com/console/get-audio-analysis-track/) to get the loudness for each track, then visualized the data by plotting the loudness and release date for each track.

<LoudnessWars />

Some notes about the data:
- For some tracks, Spotify puts out a new version of the album with a new release date, but keeps the original loudness. For other tracks, Spotify puts out a remastered or deluxe version that is louder, but keeps the original release date. I've manually removed these entries, but if you do come across a track that looks suspicious (like a Michael Jackson track released in 2017), please <a href="/contact">contact me</a> to remove it. Thank you!
- [Spotify applies some loudness normalization](https://artists.spotify.com/help/article/loudness-normalization), so listening to a very quiet song on Spotify versus a very loud song on Spotify won't be as noticeable as listening to those two songs on another medium such as a CD.

Why does this matter? In short, everything being loud results in listener fatigue. It's kind of like eating the same meal every single day&mdash; you will (hopefully) get tired of eating the same thing. The same is true with our ears&mdash; if everything is loud, we get tired and nothing really sounds loud anymore.

This project utilized [Spotipy](https://github.com/plamere/spotipy) to pull information about tracks, and [d3](https://github.com/d3/d3) to visualize the data. Source code for this project is available [here](https://github.com/ryanrishi/loudness-wars).

## Further Reading
If you're interested in learning more about the loudness wars, here are some good resources.
- [The Loudness Wars: Why Music Sounds Worse](https://www.npr.org/2009/12/31/122114058/the-loudness-wars-why-music-sounds-worse)
- [Dynamic Range Day](https://dynamicrangeday.co.uk/)
- [They Really Don’t Make Music Like They Used To](https://www.nytimes.com/2019/02/07/opinion/what-these-grammy-songs-tell-us-about-the-loudness-wars.html) - this article has a great visualization of the average loudness vs. the peak loudness for many songs

A closer look at how recorded music has been getting increasingly louder over the past 50 years.

Loudness Wars


I built a bot that scrapes [recreation.gov](https://www.recreation.gov/) for campsite available for dates I've input. When it finds a campsite availability, it texts me so that I can hop online to book it. This project is 50% a pet project to play around with Python and related tech and 50% "hey, this is actually kinda useful".

This project started as a [fork of bri-bri/yosemite-camping](https://github.com/ryanrishi/yosemite-camping) running on a VPS with a cron job, but over the years has evolved into a more robust application that allows me to:
- add campsites outside of Yosemite
- add desired reservations via a CLI
- use [recreation.gov's new RIDB API](https://ridb.recreation.gov/docs) (the old one is no longer supported), using an [OpenAPI](https://github.com/OpenAPITools/openapi-generator)-generated Python client
- use built-in scheduling

Future improvements include:
- automatically booking a reservation when a campsite becomes available
- better scaling&mdash; separating the config management, the scraper, and the notification system
- Dockerizing it
- better handling around rate limiting for the RIDB API

Source code available [upon request](/contact). Built using Python, Selenium, SQLite, and Twilio.


I wrote a bot that texts me when a Yosemite reservation I'm interested in becomes available.

Ryan Rishi.

Blog.

Solving Wordle

Managing the Same AWS Key Pair in Multiple Terraform Workspaces

Edward Tufte: Presenting Data and Information

Projects.

paperless-ng Terraform

Loudness Wars

Letscamp