2018/05/30 - Revision history

Woozle at 18:58, 20 November 2022

2022-11-20T18:58:52Z

← Older revision		Revision as of 18:58, 20 November 2022
Line 1:		Line 1:
			{{page/date\|nav=[[2018/05/20\|prev]] .. [[2018/05/30\|today]] .. [[2018/06/22\|next]] }}
	==Solved==		==Solved==
	There were, I think, two main problems:		There were, I think, two main problems:
	* The [[Let's Encrypt]] configuration had not been properly set up after the [[~~User:Woozle/toot.cat/~~2018/04/10\|server migration in April]]		* The {{l/htyp\|Let's Encrypt}} configuration had not been properly set up after the [[2018/04/10\|server migration in April]]
	* [[Nginx]] was correctly configured to respond to IPv6 on port 443 (https), but ''not'' on port 80 (http).		* {{l/htyp\|Nginx}} was correctly configured to respond to IPv6 on port 443 (https), but ''not'' on port 80 (http).
	** This was the problem that took the most time to identify, as it caused the [[letsencrypt]] command to fail even though the URL appeared to be returning the proper file.		** This was the problem that took the most time to identify, as it caused the {{l/htyp\|letsencrypt}} command to fail even though the URL appeared to be returning the proper file.

	Confounding factors included:		Confounding factors included:
	* [[HSTS]] was turned on (this seems to be part of the SSL cert, ''not'' something in Nginx), so existing browser sessions insisted on self-redirecting from http to https even when I had turned that off in Nginx. '''Workaround''': only attempt access in an anonymous browser window. (Using [[wget]] to do the test also might have worked – and ultimately, it was use of wget from an IPv6-enabled server that provided the final clue.)		* [[HSTS]] was turned on (this seems to be part of the SSL cert, ''not'' something in Nginx), so existing browser sessions insisted on self-redirecting from http to https even when I had turned that off in Nginx. '''Workaround''': only attempt access in an anonymous browser window. (Using {{l/htyp\|wget}} to do the test also might have worked – and ultimately, it was use of wget from an IPv6-enabled server that provided the final clue.)
	* My home internet (and apparently everyone else's) uses only IPv4, while Let's Encrypt uses IPv6 if it is available – so Let's Encrypt was consistently reporting a problem that we weren't seeing, leading us to think (basically) that we were misunderstanding the messages.		* My home internet (and apparently everyone else's) uses only IPv4, while Let's Encrypt uses IPv6 if it is available – so Let's Encrypt was consistently reporting a problem that we weren't seeing, leading us to think (basically) that we were misunderstanding the messages.

Woozle: Woozle moved page Woozle/toot.cat/2018/05/30 to 2018/05/30 without leaving a redirect: indended location

2022-11-20T18:00:34Z

Woozle moved page Woozle/toot.cat/2018/05/30 to 2018/05/30 without leaving a redirect: indended location

← Older revision	Revision as of 18:00, 20 November 2022
(No difference)

Woozle: 7 revisions imported: moved from HTYP

2022-11-20T17:56:43Z

7 revisions imported: moved from HTYP

← Older revision	Revision as of 17:56, 20 November 2022
(No difference)

Woozle: /* Solved */ 6/28 tidying

2018-06-28T10:18:24Z

Solved: 6/28 tidying

← Older revision		Revision as of 10:18, 28 June 2018
Line 11:		Line 11:
	Cleanup to do:		Cleanup to do:
	* It's not clear whether Let's Encrypt is now set up properly; I did the challenge installation by hand.		* It's not clear whether Let's Encrypt is now set up properly; I did the challenge installation by hand.
	* I turned off ''all'' http -> https redirects in Nginx in order to be absolutely certain that Nginx wasn't causing the redirects I was still seeing (because of HSTS).		* <s>I turned off ''all'' http -> https redirects in Nginx in order to be absolutely certain that Nginx wasn't causing the redirects I was still seeing (because of HSTS).</s>
			** '''2018-06-28''' This was causing full-page 403 errors for some users; I reverted the Nginx config file and it [https://toot.cat/@news/100281876187861211 seems to be fixed now].
			*** commented out the port 80 no-redirect block
			*** uncommented the old port 80 redirect block
			*** added an IPv6 listen directive to the uncommented block
			*** <code>/etc/init.d/nginx reload</code>
	* <s>A user [https://gist.github.com/renatolond/5102b210849beaef0a443d5d93f4636a now reports] some IPv6 issues on the https side as well. ([https://rm.vbz.net/issues/124 Addressing this] first.)</s>		* <s>A user [https://gist.github.com/renatolond/5102b210849beaef0a443d5d93f4636a now reports] some IPv6 issues on the https side as well. ([https://rm.vbz.net/issues/124 Addressing this] first.)</s>
	===Relevant Files===		===Relevant Files===

Woozle: /* Relevant Files */ another one

2018-05-31T14:32:51Z

Relevant Files: another one

← Older revision		Revision as of 14:32, 31 May 2018
Line 14:		Line 14:
	* <s>A user [https://gist.github.com/renatolond/5102b210849beaef0a443d5d93f4636a now reports] some IPv6 issues on the https side as well. ([https://rm.vbz.net/issues/124 Addressing this] first.)</s>		* <s>A user [https://gist.github.com/renatolond/5102b210849beaef0a443d5d93f4636a now reports] some IPv6 issues on the https side as well. ([https://rm.vbz.net/issues/124 Addressing this] first.)</s>
	===Relevant Files===		===Relevant Files===
			* <code>/etc/cron.daily/letsencrypt-renew</code> - the [[cron]] job to check/renew the cert
	* <code>/etc/letsencrypt/renewal/tootcat.conf</code> - Let's Encrypt configuration		* <code>/etc/letsencrypt/renewal/tootcat.conf</code> - Let's Encrypt configuration
	* <code>/etc/nginx/sites-available/tootcat.conf</code> - Nginx configuration		* <code>/etc/nginx/sites-available/tootcat.conf</code> - Nginx configuration
	* <code>/~~etc~~/~~cron~~.~~daily/letsencrypt-renew~~</code> - ~~the [[cron]] job to check/renew the cert~~		* <code>/var/log/nginx/access.log</code> - Nginx access log (showed 404 when accessing via IPv6)

	==Narration / Notes==		==Narration / Notes==
	===Phase 1===		===Phase 1===

Woozle: files; tidying

2018-05-31T13:47:04Z

files; tidying

← Older revision		Revision as of 13:47, 31 May 2018
Line 12:		Line 12:
	* It's not clear whether Let's Encrypt is now set up properly; I did the challenge installation by hand.		* It's not clear whether Let's Encrypt is now set up properly; I did the challenge installation by hand.
	* I turned off ''all'' http -> https redirects in Nginx in order to be absolutely certain that Nginx wasn't causing the redirects I was still seeing (because of HSTS).		* I turned off ''all'' http -> https redirects in Nginx in order to be absolutely certain that Nginx wasn't causing the redirects I was still seeing (because of HSTS).
	* A user [https://gist.github.com/renatolond/5102b210849beaef0a443d5d93f4636a now reports] some IPv6 issues on the https side as well. ([https://rm.vbz.net/issues/124 Addressing this] first.)		* <s>A user [https://gist.github.com/renatolond/5102b210849beaef0a443d5d93f4636a now reports] some IPv6 issues on the https side as well. ([https://rm.vbz.net/issues/124 Addressing this] first.)</s>
			===Relevant Files===
			* <code>/etc/letsencrypt/renewal/tootcat.conf</code> - Let's Encrypt configuration
			* <code>/etc/nginx/sites-available/tootcat.conf</code> - Nginx configuration
			* <code>/etc/cron.daily/letsencrypt-renew</code> - the [[cron]] job to check/renew the cert
	==Narration / Notes==		==Narration / Notes==
			===Phase 1===
	It looks like nginx was set to use a different set of certificate files than the ones [[Let's Encrypt]] was set to renew.		It looks like nginx was set to use a different set of certificate files than the ones [[Let's Encrypt]] was set to renew.

Line 48:		Line 53:
	</pre>		</pre>
	The key piece of information there seems to be "Missing --webroot-path for domain". What seems to be happening is that Nginx is redirecting from http to https even though the file exists.		The key piece of information there seems to be "Missing --webroot-path for domain". What seems to be happening is that Nginx is redirecting from http to https even though the file exists.
	==Phase 2==		===Phase 2===
	It turned out there was an [[HSTS]] policy that was forcing the browser to redirect even though Nginx wasn't, I think? But opening a toot.cat URL in an anonymous window fixed that. However it still wasn't finding the test file, so I changed the webroot on both Nginx and Let'sEncrypt to /var/www/challenges, and then was able to access the test file.		It turned out there was an [[HSTS]] policy that was forcing the browser to redirect even though Nginx wasn't, I think? But opening a toot.cat URL in an anonymous window fixed that. However it still wasn't finding the test file, so I changed the webroot on both Nginx and Let'sEncrypt to /var/www/challenges, and then was able to access the test file.

Line 74:		Line 79:
	root@tootcat2:/var/www/challenges#		root@tootcat2:/var/www/challenges#
	</pre>		</pre>
	==Phase 3==		===Phase 3===
	<pre>		<pre>
	root@tootcat2:/var/www/challenges/.well-known# letsencrypt renew		root@tootcat2:/var/www/challenges/.well-known# letsencrypt renew
Line 95:		Line 100:
	root@tootcat2:/var/www/challenges/.well-known#		root@tootcat2:/var/www/challenges/.well-known#
	</pre>		</pre>
	==Phase 4==		===Phase 4===
	<pre>		<pre>
	root@tootcat2:/var/www/challenges/.well-known# letsencrypt --dry-run renew		root@tootcat2:/var/www/challenges/.well-known# letsencrypt --dry-run renew
Line 150:		Line 155:
	root@tootcat2:/var/www/challenges/.well-known#		root@tootcat2:/var/www/challenges/.well-known#
	</pre>		</pre>
	==Phase 5==		===Phase 5===
	Relevant excerpts from Discord:		Relevant excerpts from Discord:
	<blockquote>		<blockquote>

Woozle at 11:36, 31 May 2018

2018-05-31T11:36:19Z

← Older revision		Revision as of 11:36, 31 May 2018
Line 12:		Line 12:
	* It's not clear whether Let's Encrypt is now set up properly; I did the challenge installation by hand.		* It's not clear whether Let's Encrypt is now set up properly; I did the challenge installation by hand.
	* I turned off ''all'' http -> https redirects in Nginx in order to be absolutely certain that Nginx wasn't causing the redirects I was still seeing (because of HSTS).		* I turned off ''all'' http -> https redirects in Nginx in order to be absolutely certain that Nginx wasn't causing the redirects I was still seeing (because of HSTS).
	* A user [https://gist.github.com/renatolond/5102b210849beaef0a443d5d93f4636a now reports] some IPv6 issues on the https side as well. (Addressing this first.)		* A user [https://gist.github.com/renatolond/5102b210849beaef0a443d5d93f4636a now reports] some IPv6 issues on the https side as well. ([https://rm.vbz.net/issues/124 Addressing this] first.)
	==Narration / Notes==		==Narration / Notes==
	It looks like nginx was set to use a different set of certificate files than the ones [[Let's Encrypt]] was set to renew.		It looks like nginx was set to use a different set of certificate files than the ones [[Let's Encrypt]] was set to renew.

Woozle: wrap-up

2018-05-31T11:18:41Z

wrap-up

← Older revision		Revision as of 11:18, 31 May 2018
Line 1:		Line 1:
			==Solved==
			There were, I think, two main problems:
			* The [[Let's Encrypt]] configuration had not been properly set up after the [[User:Woozle/toot.cat/2018/04/10\|server migration in April]]
			* [[Nginx]] was correctly configured to respond to IPv6 on port 443 (https), but ''not'' on port 80 (http).
			** This was the problem that took the most time to identify, as it caused the [[letsencrypt]] command to fail even though the URL appeared to be returning the proper file.

			Confounding factors included:
			* [[HSTS]] was turned on (this seems to be part of the SSL cert, ''not'' something in Nginx), so existing browser sessions insisted on self-redirecting from http to https even when I had turned that off in Nginx. '''Workaround''': only attempt access in an anonymous browser window. (Using [[wget]] to do the test also might have worked – and ultimately, it was use of wget from an IPv6-enabled server that provided the final clue.)
			* My home internet (and apparently everyone else's) uses only IPv4, while Let's Encrypt uses IPv6 if it is available – so Let's Encrypt was consistently reporting a problem that we weren't seeing, leading us to think (basically) that we were misunderstanding the messages.

			Cleanup to do:
			* It's not clear whether Let's Encrypt is now set up properly; I did the challenge installation by hand.
			* I turned off ''all'' http -> https redirects in Nginx in order to be absolutely certain that Nginx wasn't causing the redirects I was still seeing (because of HSTS).
			* A user [https://gist.github.com/renatolond/5102b210849beaef0a443d5d93f4636a now reports] some IPv6 issues on the https side as well. (Addressing this first.)
			==Narration / Notes==
	It looks like nginx was set to use a different set of certificate files than the ones [[Let's Encrypt]] was set to renew.		It looks like nginx was set to use a different set of certificate files than the ones [[Let's Encrypt]] was set to renew.

Woozle: Phase 5: from Discord

2018-05-31T10:36:55Z

Phase 5: from Discord

← Older revision		Revision as of 10:36, 31 May 2018
Line 80:		Line 80:
	root@tootcat2:/var/www/challenges/.well-known#		root@tootcat2:/var/www/challenges/.well-known#
	</pre>		</pre>
			==Phase 4==
			<pre>
			root@tootcat2:/var/www/challenges/.well-known# letsencrypt --dry-run renew
			Processing /etc/letsencrypt/renewal/toot.cat.conf
			2018-05-31 01:42:09,149:WARNING:letsencrypt.client:Registering without email!
			2018-05-31 01:42:09,992:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/toot.cat.conf produced an unexpected error: Missing command line flag or config entry for this setting:
			Please read the Terms of Service at https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must agree in order to register with the ACME server at https://acme-staging.api.letsencrypt.org/directory

			(You can set this with the --agree-tos flag). Skipping.
			** DRY RUN: simulating 'letsencrypt renew' close to cert expiry
			** (The test certificates below have not been saved.)

			All renewal attempts failed. The following certs could not be renewed:
			/etc/letsencrypt/live/toot.cat/fullchain.pem (failure)
			** DRY RUN: simulating 'letsencrypt renew' close to cert expiry
			** (The test certificates above have not been saved.)
			1 renew failure(s), 0 parse failure(s)
			root@tootcat2:/var/www/challenges/.well-known#
			</pre>
			<pre>
			root@tootcat2:/var/www/challenges/.well-known# letsencrypt --dry-run --agree-tos renew
			Processing /etc/letsencrypt/renewal/toot.cat.conf
			2018-05-31 01:52:59,407:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/toot.cat.conf produced an unexpected error: Failed authorization procedure. toot.c$
			t (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://toot.cat/.well-known/acme-challenge/tZFwZb9H2at2brdJYexpRqTDYOigSbT$
			J_6oL3AXwBQ: ''[404 errors]''. Skipping.
			** DRY RUN: simulating 'letsencrypt renew' close to cert expiry
			** (The test certificates below have not been saved.)

			All renewal attempts failed. The following certs could not be renewed:
			/etc/letsencrypt/live/toot.cat/fullchain.pem (failure)
			** DRY RUN: simulating 'letsencrypt renew' close to cert expiry
			** (The test certificates above have not been saved.)
			1 renew failure(s), 0 parse failure(s)
			</pre>
			[...]
			<pre>
			root@tootcat2:/var/www/challenges/.well-known# letsencrypt --dry-run --agree-tos --manual renew
			Processing /etc/letsencrypt/renewal/toot.cat.conf
			2018-05-31 01:59:54,703:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/toot.cat.conf produced an unexpected error: Missing command line flag or config entry for this setting:
			NOTE: The IP of this machine will be publicly logged as having requested this certificate. If you're running letsencrypt in manual mode on a machine that is not your server, please ensure you're okay with that.

			Are you OK with your IP being logged?


			(You can set this with the --manual-public-ip-logging-ok flag). Skipping.
			** DRY RUN: simulating 'letsencrypt renew' close to cert expiry
			** (The test certificates below have not been saved.)

			All renewal attempts failed. The following certs could not be renewed:
			/etc/letsencrypt/live/toot.cat/fullchain.pem (failure)
			** DRY RUN: simulating 'letsencrypt renew' close to cert expiry
			** (The test certificates above have not been saved.)
			1 renew failure(s), 0 parse failure(s)
			root@tootcat2:/var/www/challenges/.well-known#
			</pre>
			==Phase 5==
			Relevant excerpts from Discord:
			<blockquote>
			<poem>
			Ok, yeah, same error. It's getting a 404. But the URL it claims to be accessing is returning the challenge.
			(Just verified again, pasting URL from the logfile.)
			It'd be nice if it would include the whole URL, including the domain and protocol.
			Wait, it did. I'm just tired.
			It's as if the LE remote client is reaching a different server.(edited)
			Is there any logging of IP address...
			Can't see any.
			Okay, next idea; check Nginx server log and see what URL is being requested. If any.
			''[...]''
			I'm trying to do a close tracking of what request the server sees when I do the test run, but my ADD gets worse when I'm tired and I forget what I was doing between one screen and the next.
			Ok, I've caught the server actually returning a 404!
			Checking URL...
			URL is fine, except I'm making an assumption about the domain.
			I think that must be the problem.
			Somehow.
			There is another domain on that server, and it goes to a different webroot.
			oh, ffu.... this MIGHT be an IPv6 issue. >.<
			''[...]''
			I can test this from another DO server. They all have IPv6 and tend to default to it.
			''[...]''
			It's IPv6.
			''[...]''
			I just got a 404 from the same URL that works from here.
			Ok, this should be tractable now.
			I just have to find an example of correct ipv6 Nginx config.
			''[...]''
			I can probably just copy from the https section.
			Found.
			Now restart nginx...
			Or reload, I guess.
			200 OK
			So... do I dare try LE the easy way? No, I'd better do it manually again.
			Actually, better do a manual dry-run first.
			TEST SUCCESSFUL
			''[...]''
			Now the real thing (but manually again, because I have zero faith that the other part of the problem is solved).
			heh... URL ends in UwU...
			''[...]''
			I got a success message from LE, but still getting cert error in browser.
			owait, probly need to restart nginx.
			<hopes that's all it is>
			Yeah, the standard cron script does that too. Ok...
			OMGSUCCESS.
			</poem>
			</blockquote>

Woozle: saving work

2018-05-31T01:38:13Z

saving work

New page

It looks like nginx was set to use a different set of certificate files than the ones [[Let's Encrypt]] was set to renew.

Tentatively, LE goes through all the .conf files in /etc/letsencrypt/renewal and renews each one.

There was only one, and it pointed at files in /etc/letsencrypt/live/tootcat2.hypertwins.net/

I've changed it to point to /etc/letsencrypt/live/toot.cat/

Nginx also looks for 2 cert files in /etc/letsencrypt/live/toot.cat/, so now at least we're matched.

When I try to renew with letsencrypt renew, I get:
<pre>
root@tootcat2:/# letsencrypt renew
Processing /etc/letsencrypt/renewal/toot.cat.conf
2018-05-31 00:11:15,708:ERROR:letsencrypt.error_handler:Encountered exception during recovery
2018-05-31 00:11:15,709:ERROR:letsencrypt.error_handler:Missing --webroot-path for domain: toot.cat
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/letsencrypt/error_handler.py", line 74, in call_registered
self.funcs[-1]()
File "/usr/lib/python2.7/dist-packages/letsencrypt/auth_handler.py", line 280, in _cleanup_challenges
self.dv_auth.cleanup(dv_c)
File "/usr/lib/python2.7/dist-packages/letsencrypt/plugins/webroot.py", line 139, in cleanup
root_path = self._get_root_path(achall)
File "/usr/lib/python2.7/dist-packages/letsencrypt/plugins/webroot.py", line 108, in _get_root_path
.format(achall.domain))
PluginError: Missing --webroot-path for domain: toot.cat
2018-05-31 00:11:15,711:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/toot.cat.conf produced an unexpected error: Missing --webroot-path for domain: toot.cat. Skipping.

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/toot.cat/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
root@tootcat2:/#
</pre>
The key piece of information there seems to be "Missing --webroot-path for domain". What seems to be happening is that Nginx is redirecting from http to https even though the file exists.
==Phase 2==
It turned out there was an [[HSTS]] policy that was forcing the browser to redirect even though Nginx wasn't, I think? But opening a toot.cat URL in an anonymous window fixed that. However it still wasn't finding the test file, so I changed the webroot on both Nginx and Let'sEncrypt to /var/www/challenges, and then was able to access the test file.

...but Let's Encrypt still returns this:
<pre>
root@tootcat2:/var/www/challenges# letsencrypt renew
Processing /etc/letsencrypt/renewal/toot.cat.conf
2018-05-31 00:49:39,460:ERROR:letsencrypt.error_handler:Encountered exception during recovery
2018-05-31 00:49:39,460:ERROR:letsencrypt.error_handler:Missing --webroot-path for domain: toot.cat
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/letsencrypt/error_handler.py", line 74, in call_registered
self.funcs[-1]()
File "/usr/lib/python2.7/dist-packages/letsencrypt/auth_handler.py", line 280, in _cleanup_challenges
self.dv_auth.cleanup(dv_c)
File "/usr/lib/python2.7/dist-packages/letsencrypt/plugins/webroot.py", line 139, in cleanup
root_path = self._get_root_path(achall)
File "/usr/lib/python2.7/dist-packages/letsencrypt/plugins/webroot.py", line 108, in _get_root_path
.format(achall.domain))
PluginError: Missing --webroot-path for domain: toot.cat
2018-05-31 00:49:39,462:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/toot.cat.conf produced an unexpected error: Missing --webroot-path for domain: toot.cat. Skipping.

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/toot.cat/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
root@tootcat2:/var/www/challenges#
</pre>
==Phase 3==
<pre>
root@tootcat2:/var/www/challenges/.well-known# letsencrypt renew
Processing /etc/letsencrypt/renewal/toot.cat.conf
2018-05-31 01:30:48,656:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/toot.cat.conf produced an unexpected error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new authz :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/. Skipping.

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/toot.cat/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
root@tootcat2:/var/www/challenges/.well-known#
</pre>
<pre>
root@tootcat2:/var/www/challenges/.well-known# letsencrypt --staging renew
Processing /etc/letsencrypt/renewal/toot.cat.conf
2018-05-31 01:36:59,134:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/toot.cat.conf produced an unexpected error: You should register before running non-interactively, or provide --agree-tos and --email <email_address> flags. Skipping.

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/toot.cat/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
root@tootcat2:/var/www/challenges/.well-known#
</pre>