Last week I ended up spending more time than I expected on debugging.
I spent most of the time figuring out why a library did not accept my freshly generated RSA private key.
I was supposed to provide the RSA private key as an environment variable
RSA_PRIVATE_KEY. I wanted to store the RSA private key in a file
private-key.pem so my first attempt became
RSA_PRIVATE_KEY=\"$(cat private-key.pem)\" docker-compose up -d, which failed due to decoding issues by the library.
It would appear that the library was very strict on the format of the private key.
After a lot of debugging, I managed to figure out how I was supposed to provide the private key and I will try to share my learnings here.
During the debugging, I learned a ton about a few UNIX tools.
I realized that we have a great toolbox in front of us all day which we should try to get to know a little better.
This toolbox is part of Unix Development Tools and comes pre-installed on most UNIX systems (including MacOS).
I will go through a subset of these tools and explain how they helped me debug the decoding issue with the private key.
The private-key.pem referenced in the following command was generated by
openssl genrsa -out private-key.pem 3072
File command classifies the specified file.
The command determines the file type, and by providing the -I flag, the tool displays the mime file type.
The mime type corresponds to the media type and can help you verify that the file is of the type/format you expect it to be.
» file private-key.pem
private-key.pem: PEM RSA private key
» file -I private-key.pem
private-key.pem: text/plain; charset=us-ascii
The above output from
File was reasonable, it shows that the file is of the type that we expect.
file private-key.pem would have returned
private-key.pem: ASCII text, it would have been a red flag indicating that the file is not of the expected type.
By viewing the content of
private-key.pem, I could see that it contained linebreaks, and I was afraid that those linebreaks caused the issue.
I wanted to replace the linebreaks with the newline character
\n, and I used
sed from the toolbox to achieve that.
Sed reads the specified input stream and modifies it according to the given commands or regex.
» sed 's/$/\\n/' private-key.pem
-----BEGIN RSA PRIVATE KEY-----\n
-----END RSA PRIVATE KEY-----\n
s (substitute) was used.
This command changes all occurrences of something to something else.
The syntax of the
sed s command is:
sed s/before/after, meaning that
s/$/\\n/ substitutes the end of the line (indicated by
$) with with actual value
Unfortunately, this substitution did not solve my problem.
The next tool to grab from the toolbox was
Hexdump which displays the file content in different formats.
One of the formats is hexadecimal.
Hexdump is very helpful in displaying hidden characters that are difficult to identify manually, for instance, linebreaks, newlines, tabs, and spaces.
» hexdump -C private-key.pem
00000000 2d 2d 2d 2d 2d 42 45 47 49 4e 20 52 53 41 20 50 |-----BEGIN RSA P|
00000010 52 49 56 41 54 45 20 4b 45 59 2d 2d 2d 2d 2d 0a |RIVATE KEY-----.|
00000020 4d 49 49 47 34 77 49 42 41 41 4b 43 41 59 45 41 |MIIG4wIBAAKCAYEA|
00000970 48 51 56 59 59 55 42 77 0a 2d 2d 2d 2d 2d 45 4e |HQVYYUBw.-----EN|
00000980 44 20 52 53 41 20 50 52 49 56 41 54 45 20 4b 45 |D RSA PRIVATE KE|
00000990 59 2d 2d 2d 2d 2d 0a |Y-----.|
-C flag sets the output format to hexadecimal. Each row contains 16 characters, each displayed as their hexadecimal representation.
The first value is
2d, and by looking up
2d in an ascii-table, I could see that
2d corresponds to the
- character, meaning that we have no unwanted characters at the beginning of the file.
The last binary value was
0a which represents
LF which is the character that indicates the end of the line.
It is not the same as newline
\n or carriage return
These characters can be tricky to handle, but I had already tried to replace them with newlines using sed without any luck.
Running out of ideas for a potential next step I decided to test with a copy of the
I copied the content of
private-key.pem by running
pbcopy < private-key.pem, and I opened my editor and pasted the content into
To my surprise, the application started when I ran it with
private-key2.pem, the COPY of
To understand why this was possible, I grabbed a new tool from the toolbox,
This tool compares files line by line.
» diff private-key.pem private-key2.pem
The output of
Diff above shows a minor difference between the files.
39a40 represents the change: starting at row
39, add line(s) until reaching row
Diff command in reversed order shows the same thing but from the perspective of
» diff private-key2.pem private-key.pem
40d39 represents the change: starting at row
40, delete line(s) until row
39. The output from
Diff can be seen as instructions for making the files identical.
To conclude, all that was missing was a
\n at the end of the file. The key generation command excluded the necessary new line, but my editor decided to add it.
A lot of work for just a missing newline, but I realized that the included toolbox is quite powerful.
We should all try to get familiar with the tools, you never know when you may need them in the future.
There are a few more tools I want to mention before I move on.
Allows one to use the output from one command as input to another command.
» sed 's/$/\\n/' private-key.pem | hexdump
0000000 2d2d 2d2d 422d 4745 4e49 5220 4153 5020
0000010 4952 4156 4554 4b20 5945 2d2d 2d2d 5c2d
0000020 0a6e 494d 4749 4135 4249 4141 434b 5941
Search for information in files.
» cat private-key.pem | grep RSA
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
Show the manual and instructions of the given command.
» man grep
GREP(1) General Commands Manual GREP(1)
grep, egrep, fgrep, rgrep, bzgrep, bzegrep, bzfgrep, zgrep, zegrep, zfgrep
– file pattern searcher
grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color[=when]]
[--colour[=when]] [--context[=num]] [--label] [--line-buffered]
[--null] [pattern] [file ...]
The grep utility searches any given input files, selecting lines that match
one or more patterns. By default, a pattern matches an input line if the
regular expression (RE) in the pattern matches the input line without its
trailing new line. An empty expression matches every line. Each input line
that matches at least one of the patterns are written to the standard