IPv6 test errors (untestable) with slow sites/redirect loops since 1.10.0

[rudiedirkx]

Since 1.10.6, some domains are failing:

• brandweer-oostgelre.nl
• ogtest.nl

The Batch API response is just "error" without any detail. I'd love more detail when errors occur, any detail, see #1498

The manual test fails on ipv6:

{"name":"ipv6","done":true,"success":false}

1.10.6 has only been live for one day, so I assume this will happen much more.

[baknu]

The websites on the mentioned domains are not reachable for me via a browser. After a long wait, they both give an error page from Cloudflare with the message "Connection timed out" (Error code 522).

[bwbroersma]

But error should not occur on timeout, although it might be timing related indeed.

9s delay: 100% https://internet.nl/site/ipcjcjkqfeaa.v1.b6a.nl/3468152/
10s delay: 75% https://internet.nl/site/iofsgaa.v1.b6a.nl/3468151/
20s delay: IPv6 test error (ipcjcejiryaa.v1.b6a.nl)

Related:

• [hold] Fix #714 - Switch to (mainly) sslyze for TLS testing #1218 (comment)
• Determine appropriate timeouts in DNS, HTTP, TLS #1738

[baknu]

Sure, internet.nl probably could/should handle timeouts better. Still this does not seem to be an issue that was introduced with version 1.10.6. And, of course, the reachability problem the tested websites have, is something we can not fix.

[bwbroersma]

Somehow I cannot find it explained in any issue. But this problem is due to the IPv6 tasks taking longer than the allowed time, that is:

Internet.nl/docker/defaults.env

Lines 110 to 111 in b3e813a

	# maximum test duration and time before retest can be performed
	INTERNETNL_CACHE_TTL=200

• Next to DNS requests there can be up 6 HTTP(S) requests (checking IPv4, IPv6, HTTP + redirect and HTTPS), see Reduce https IPv46 happy flow requests #1820. Note in the specially crafted b6a.nl domains the HTTP->HTTPS redirect is fast, which is also the case with Cloudflare:

echo -4 -6 | xargs -n1 curl -sS 'http{,s}://ogtest.nl' -o/dev/null -w "%{scheme}\t%{http_code}\t%{time_total}\t%{remote_ip}\n"   

HTTP	301	0.040595	188.114.97.3
HTTPS	522	39.087170	188.114.97.3
HTTP	301	0.043370	2a06:98c1:3121::
HTTPS	522	39.079139	2a06:98c1:3120::

In this case 4 HTTPS calls are made which take more then $39×4=156$ seconds alone, what leaves $200-156=44$ seconds for all other stuff, the 2 HTTP calls, all DNS requests. I just timed it, and the TLS task takes about 1m20s, so 80s (so it probably waits for 2 serial HTTPS responses taking $39×2=78$ seconds), while the IPv6 tasks times out at 2m40s, so 200s.
I think it can only happen with some response / 'slow servers', not with completely unresponsive endpoints, since otherwise 'not reachable' will finish the task within the timeout.

Maybe also related:

• Hard time limit exceeded for checks.tasks.ipv6.ns #1034

[rudiedirkx]

Still this does not seem to be an issue that was introduced with version 1.10.6.

That's the first time it happened. There are only a few of the thousands of domains/hostnames we test that really fail. They used to be very rare, but since 1.10.6 it's happened for 3 domains.

[bwbroersma]

Well there were some changes in v1.10.x, one of which also affected the IPv6 test:

• Same website on IPv6 and IPv4 not detecting difference on IPv4 HTTP reachablility if IPv4 HTTPS is reachable #1705
• Fix #1705 - Detect IPv4 HTTP unreachable when HTTPS works #1714 , I forgot I already noticed this issue, because there now is an extra HTTP+HTTPS call that has to be executed within the timeout.

I remember @mxsasha saying it are always some very long response times which results in tickets, so does it here... I think 39s for an http response is not something internet.nl should accepted, but it shouldn't 'error' on it, it should timeout and continue finishing a report.

[baknu]

Note that other instances with previous versions of the Internet.nl code indeed do not 'error':

• https://test.nic.br/site/ogtest.nl/4559/
• https://sikkerpånettet.dk/site/ogtest.nl/179373/

[rudiedirkx]

Another one: teebstad.nl.

I don't know if adding more cases help, but I'll just mention them.

[rudiedirkx]

More:

• oldambt.groningen.nl
• omgevingsvisielangedijk.nl
• westerwolde.groningen.nl

[rudiedirkx]

Another one: omgevingsvisielangedijk.nl

[apio-sys]

Those are no testing issues, but local issues with sites.

• omgevingsvisielangedijk.nl tries to forward to omgevingsvisie.ldhhw.nl and then just times out, so any test will also fail.
• westerwolde.groningen.nl tries to forward to https://www.visitgroningen.nl/nl/plekken/westerwolde?ins_rd=d2VzdGVyd29sZGUuZ3JvbmluZ2VuLm5s and then get"s in a redirection loop.
• oldambt.groningen.nl same.
• teebstad.nl crashes on a hard TLS error.

If you can't properly visit a site in your browser, well obviously most tests will fail. Unless there is a real reproducible case I don't think there is any real issue here.

[rudiedirkx]

If you can't properly visit a site in your browser, well obviously most tests will fail.

That was never true before 1.10.6. A 0% test result is fine. If a website is shit, it's shit. But the test itself should never ever ever fail. Or you all decide that from now on it does, that's fine too. But I can't imagine this was intentional.

[bwbroersma]

@rudiedirkx thanks for the more use cases and thanks @apio-sys for doing the analysis on this, it are actually quite some different cases (not just 'slow sites'). I stand by my point:

it shouldn't 'error' on it, it should timeout and continue finishing a report.

These use cases actually make it stronger case (e.g. the redirection loop), and will help to test a fixed implementation timeout.