M3AAWG June 2022 - DNSSEC & DNS Debugging

• dig is the standard tool to send DNS queries.
• It replaces the older (and obsolete nslookup)
  • stay away from nslookup, it is old, deprecated and has know bugs that will not be fixed
  • It uses it's own resolver library, which can create results differently from the operating system resolver
  • It does not show all the useful data in a DNS response (like EDNS)
  • It cannot work with DNSSEC
• For troubleshooting purposes, we recommend to use the most recent version of dig, as it includes additional troubleshooting capabilities (like EDE error codes) compared to older versions.
  • The current version of dig (June 2022) ist 9.18.3 (https://ftp.isc.org/isc/bind9/9.18.3/)
  • On the training lab virtual machines we use a version 9.11.36-RedHat that has support for extended DNS error codes and DNS cookies backported
• The output of dig resembles the structure of DNS packets on the wire

• The dig output explained

• The DNS data is separated in sections

• opcode: query, notify, update, DSO … see IANA registry for DNS OpCodes
• status/rcode:
  • NOERROR - The operation was successful
  • NXDOMAIN - The domain name requested does not exist (or is not delegated)
  • SERVFAIL - Some remote DNS server failure or DNSSEC validation failure
  • FORMERR - The query was not correct DNS
  • REFUSED - This server has an access control list that forbids the answer to this client
  • NOTIMPL - A feature used/requested that this server does not implement
  • BADCOOKIE - Bad/missing Server Cookie

  … see IANA registry for DNS RCODEs

• queryid: 16bit value to sort DNS answers to DNS queries
• flags: Information on the query and the answer (see IANA registry for DNS Header Flags)
  • AA Authoritative answer - answer is coming directly from an authoritative server
  • TC Truncated - answer does not fit into the advertised UDP packet size, please re-query over TCP
  • RD Recursion desired - this is a query from a client machine, please provide a full complete answer (no referral please)
  • RA Recursion available - this answer comes from a DNS resolver that is willing to accept queries with RD flag set
• DNSSEC flags: AD- and CD-Flag
  • AD Authentic data - the DNS resolver sending this answer has performed a successful DNSSEC validation on the data. If we trust the resolver, we can trust the data
  • CD Checking disabled - a client asking a DNSSEC validating DNS resolver to not perform DNSSEC validation but to pass all DNSSEC data unaltered (even if the data is invalid). Used for troubleshooting DNSSEC issues.
• QUERY: Number of query resource records (usually one)
• ANSWER: Count of DNS resource records in the answer. Can be more than one. Can be zero if no data is available for the query.
• AUTHORITY: Number of authority records in the answer. Can be a SOA-Record (for negative answers) or NS-Records for referrals or positive answers. Many modern DNS server only fill the authority section if required by the protocol to keep answer packets small
• ADDITIONAL: Additional resource records that not have been requested but might help with the name resolution, and the EDNS (Extended DNS) OPT-Record (see RFC 6891 Extension Mechanisms for DNS (EDNS(0))

• EDNS Version 0 is the current version
  • New Versions are being discussed in the IETF
• Additional EDNS flags. DO - DNSSEC OK, this DNS client supports DNSSEC records
• Maximum UDP answer packet size in byte as negotiated between dig and the DNS server/resolver. Current default is 4096, must be between 512 and 4096. The default changed to 1232 on DNS flag day 2020
• RFC 8914 - Extended DNS Errors defines a way to deliver additional error information in an DNS response. It is already implemented in the latest versions of dig since version BIND 9.16.4. Some DNS resolver on the Internet (like Cloudflare) do support EDE messages:

  

• RFC 7873 Domain Name System (DNS) Cookies great a lightweight session between a DNS requester and a DNS responder (DNS client and DNS resolver or DNS resolver to authoritative DNS server)
  • DNS cookies prevent certain kind of DNS spoofing and denial of service attacks
  • Support for DNS cookies can be seen in a DNS response EDNS section

$ dig @ns5.tidelock.de dnssec.works SOA +cookie +multi

; <<>> DiG 9.11.36-RedHat-9.11.36-3.el8 <<>> @ns5.tidelock.de dnssec.works SOA +cookie +multi
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18486
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: 0229126cd6b3fa8e0100000062a6f806171fd91580e27b61 (good)
;; QUESTION SECTION:
;dnssec.works.		IN SOA

;; ANSWER SECTION:
dnssec.works.		1800 IN	SOA ns3.myinfrastructure.org. hostmaster.strotmann.de. (
				3454       ; serial
				86400      ; refresh (1 day)
				7200       ; retry (2 hours)
				3542400    ; expire (5 weeks 6 days)
				3600       ; minimum (1 hour)
				)

;; AUTHORITY SECTION:
dnssec.works.		1800 IN	NS ns3.myinfrastructure.org.
dnssec.works.		1800 IN	NS ns5.myinfrastructure.org.

;; Query time: 14 msec
;; SERVER: 2001:19f0:5001:df:76d7:5703:ba0a:e220#53(2001:19f0:5001:df:76d7:5703:ba0a:e220)
;; WHEN: Mon Jun 13 08:40:38 UTC 2022
;; MSG SIZE  rcvd: 190

• The answer section contains zero (if there is no data available), one or more DNS resource records that match the query

• If present, the authority section contains the authoritative name server that hosts the content delivered in the answer. For negative (NXDOMAIN/NOERROR-NODATA) answers, the authority section contains the SOA record of the zone that is authoritative for the negative answer.

• If present, the additional section contains additional DNS records that have not been explicitly requested, but might help in the name resolution process. Because the additional section can be misused in attacks, modern DNS server software minimizes the additional section data.
• If EDNS data is available, it is also in the additional section, but dig displays this data in the OPT pseudo-section

• The footer contains the size (in bytes) of the answer packet, the DNS server that has send the answer, the time it took to receive the answer and the time and date of the DNS communication

• Usually dig sends the query to the DNS resolver configured in the operating system (file /etc/resolv.conf on Unix/Linux)
• With the @ syntax the query can be sent to other DNS servers (resolver or authoritative server)

• Command line switches can alter the query sent by dig.
  • +multi formats the output in human readable form (wrapped for 80 column terminal)
  • +norec sends a query without RD flag (non-recursive or iterative query)

• The switch +dnssec requests DNSSEC data from the upstream server
• The switch +cd sends the CD (checking disabled) flag to disable upstream DNSSEC validation (if the target of the query is a DNS resolver)

The website Zonemaster https://zonemaster.net is a collaboration between the French TLD registry Afnic and the Swedish registry IIS. The website takes a domain name and will generate a report of errors and best practice recommendations of the setup of this domain name.

DNSViz https://dnsviz.net is a tool for visualizing the status of a DNS zone. It provides a visual analysis of the DNSSEC authentication chain for a domain name and its resolution path in the DNS namespace, and it lists configuration errors detected by the tool.

The tool delv is part of the BIND 9 DNS server and implements a full DNS resolver and DNSSEC validator inside the command line tool. This tool works very similarly to dig (and shares the same command line syntax), but where dig always needs a DNS resolver to get the DNS information, delv can query the data and can validate the DNSSEC information itself.

The flag +mtrace enables the message tracing in delv, the tool will print all DNS queries and answers during the processing of the query.

$ delv example.com +mtrace

The flag +vtrace will print a debug trace of all DNSSEC validation steps the tool will do in order to validate the received DNS answers

$ delv example.com +vtrace

The tool drill is part of the LDNS tools (Debian/Ubuntu package ldnsutils) and is a dig lookalike tool with additional features. One notable feature is the function to trace the DNSSEC validation and to print the DNSSEC chain-of-trust together with the DNSSEC key information.

drill requires the current DNSSEC root trust anchor in a file to be able to work. This trust anchor can be received with drill, but also with dig or delve:

$ dig @a.root-servers.net . DNSKEY | grep 257 > root.key

This DNSSEC root trust-anchor key can be used to do a DNSSEC chain chase with drill:

$ drill -SD -k root.key example.com

A DoT service can be reached on port 853.

To send a query over TLS protocol (DNS over TLS over TCP), use the +tls command line switch

% dig @1.1.1.1 m3aawg.org A +tls

; <<>> DiG 9.18.3 <<>> @1.1.1.1 m3aawg.org A +tls
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45617
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; PAD: (409 bytes)
;; QUESTION SECTION:
;m3aawg.org.			IN	A

;; ANSWER SECTION:
m3aawg.org.		300	IN	A	34.214.179.220

;; Query time: 30 msec
;; SERVER: 1.1.1.1#853(1.1.1.1) (TLS)
;; WHEN: Mon Jun 13 08:21:37 CEST 2022
;; MSG SIZE  rcvd: 468

The use of TLS as the transport is indicated in the footer line.

DoH does work on port 443 (classic https port) and can be multiplexed with other HTTPS data (it is possible to host both a web-site and a DNS-over-HTTPS service on the same port.

The idea of DoH is to provide extra privacy by transport encryption and by hiding the DNS requests inside the HTTPS stream (web-data).

To send a DNS query over HTTPS protocol (DNS over HTTP over TLS over TCP), use the +https command line switch

% dig @1.1.1.1 m3aawg.org A +https

; <<>> DiG 9.18.3 <<>> @1.1.1.1 m3aawg.org A +https
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10673
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;m3aawg.org.			IN	A

;; ANSWER SECTION:
m3aawg.org.		300	IN	A	34.214.179.220

;; Query time: 16 msec
;; SERVER: 1.1.1.1#443(1.1.1.1) (HTTPS)
;; WHEN: Mon Jun 13 08:25:12 CEST 2022
;; MSG SIZE  rcvd: 55

The use of HTTPS as the transport is indicated in the footer line.

At https://github.com/sys4/m3aawg-dns-and-dnssec-debugging we are providing a set of simple shell scripts that will test DNS resolver or the setup of authoritative DNS servers for a zone.

These scripts are always work in progress and will be expanded in the future (check the repo). Pull requests are very welcome.

The scripts can be used as templates to build your own DNS monitoring.

The scripts are also a lightweight on-prem alternative to the web tools.

Below is an example inocation of the DNS resolver check script. The argument is the IPv6 (or IPv4) address of an DNS resolver

% bash ./dnsresolver-check.sh 194.25.0.52 
 query A-Record over IPv4/UDP:                             [  OK  ]
 query AAAA-Record over IPv4/UDP:                          [  OK  ]
 query A-Record over IPv4/TCP:                             [  OK  ]
 query AAAA-Record over IPv4/TCP:                          [  OK  ]
 query A-Record over TLS:                                  [FAILED]
 query A-Record over HTTPS:                                [FAILED]
 verify response > 512 byte (classic DNS limit for UDP):   [  OK  ]
 verify response > 1232 byte (2020 DNS Flag Day limit):    [  OK  ]
 verify response > 1500 byte (Ethernet MTU limit for UDP): [  OK  ]
 verify response > 4096 byte (EDNS max limit for UDP):     [  OK  ]
 verify DNSSEC validation of the root zone                 [  OK  ]
 verify DNSSEC validation of com TLD zone                  [  OK  ]
 verify DNSSEC validation of net TLD zone                  [  OK  ]
 verify DNSSEC validation of org TLD zone                  [  OK  ]
 verify DNSSEC validation of de ccTLD zone                 [  OK  ]
 verify DNSSEC validation of se ccTLD zone                 [  OK  ]
 verify DNSSEC validation of cz ccTLD zone                 [  OK  ]
 verify DNSSEC validation of xyz nTLD zone                 [  OK  ]
 verify DNSSEC validation of onl nTLD zone                 [  OK  ]
 verify DNSSEC validation of nrw nTLD zone                 [  OK  ]
 verify DNSSEC validation with RSA-SHA256                  [  OK  ]
 verify DNSSEC validation with RSA-SHA512                  [  OK  ]
 verify DNSSEC validation with ECDSA256                    [  OK  ]
 verify DNSSEC validation with ECDSA384                    [  OK  ]
 non validation of record with expired RRSIG               [  OK  ]
 non validation of record with RRSIG to DNSKEY mismatch    [  OK  ]
 non validation of record with DNSKEY to DS record mismatch[  OK  ]
 verify DNSSEC validation of TLSA record                   [  OK  ]
 verify DNSSEC validation of NXDOMAIN answer               [  OK  ]
 verify DNSSEC validation of NODATA answer                 [  OK  ]
 rebind protection                                         [FAILED]

The DNS zone check script will take the name of a DNS zone and will execute a number of tests against the authoritative DNS servers for this zone:

% bash ./dnsauth-check.sh dnssec.works
Testing zone dnssec.works...
 Testing UDPv4
 Test UDPv4 ns5.myinfrastructure.org. for dnssec.works:    [  OK  ]
 Test UDPv4 ns3.myinfrastructure.org. for dnssec.works:    [  OK  ]
 Testing UDPv6
 Test UDPv6 ns3.myinfrastructure.org. for dnssec.works:    [  OK  ]
 Test UDPv6 ns5.myinfrastructure.org. for dnssec.works:    [  OK  ]
 Testing TCPv4
 Test UDPv4 ns3.myinfrastructure.org. for dnssec.works:    [  OK  ]
 Test UDPv4 ns5.myinfrastructure.org. for dnssec.works:    [  OK  ]
 Testing TCPv6
 Test UDPv6 ns3.myinfrastructure.org. for dnssec.works:    [  OK  ]
 Test UDPv6 ns5.myinfrastructure.org. for dnssec.works:    [  OK  ]
 Testing EDNS Response Size (Policy: 1232 bytes)
 Test ns3.myinfrastructure.org. for dnssec.works:          [  OK  ]
 Test ns5.myinfrastructure.org. for dnssec.works:          [FAILED]
 Test Parent - Child Delegation
 Test delegation for dnssec.works                          [  OK  ]

dig's default output is divided into seven sections. Not every section is shown with each dig command.

• cmd (initial comment)
• comment
• question (the query)
• answer
• authority
• additional
• stats

• The sections are toggled to appear or be hidden with query options.
  • Query options begin with a +.
  • All query options can be disabled with +noall, then individual options can be re-enabled.

    $ dig +noall server01.dnslab.org
    

• The initial comment (cmd) section shows the dig version number and query options.

  $ dig +noall +cmd server01.dnslab.org
  ; <<>> DiG 9.10.3 <<>> server01.dnslab.org +noall +cmd
  ;; global options: +cmd
  

• stats is the last section displayed.
  • It shows the time for the query, the server queried, and the total size of the response packet.

    $ dig +noall server01.dnslab.org +stats
    ;; Query time: 7 msec
    ;; SERVER: 192.168.53.251#53(192.168.53.251)
    ;; WHEN: Wed Nov 25 12:55:36 UTC 2015
    ;; MSG SIZE  rcvd: 92
    

• The output of dig gets considerably smaller without the first and last sections (initial comment & stats).

  $ dig +nocmd +nostats ns01a.dnslab.org
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21736
  ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
  
  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 512
  ;; QUESTION SECTION:
  ;ns01a.dnslab.org.                        IN      A
  
  ;; ANSWER SECTION:
  ns01a.dnslab.org.         60      IN      A       178.128.233.110
  

• The comment section comes after the cmd (initial comment) section.
  • It shows all message fields except the four with RRs.
  • RCODE is listed as status.

    $ dig +noall  server01.dnslab.org +comment
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42739
    ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    

• The option +nocomment removes the comments section, and comments before other sections.

  $ dig +nocmd +nostats ns01a.dnslab.org | grep SECTION
  ;; OPT PSEUDOSECTION:
  ;; QUESTION SECTION:
  ;; ANSWER SECTION:
  
  $ dig +nocmd +nostats  ns01a.dnslab.org +nocomment | grep SECTION
  $
  

• The question section shows the query as it appears in the response message.

  $ dig +noall +question ns01a.dnslab.org
  ;ns01a.dnslab.org.                        IN      A
  

• The answer section shows the query response in standard Master File Format.

  $ dig +noall +answer ns01a.dnslab.org
  ns01a.dnslab.org.         36      IN      A       178.128.233.110
  

• The authority section shows the authority section in the query response.

  $ dig +noall +authority ns01a.zone100.dnslab.org @localhost
  zone100.dnslab.org.        30      IN      SOA     ns01a.dnslab.org. hostmaster.zone100.dnslab.org. 1001 3600 1800 3542400 30
  

• The additional section shows the additional section in the query response.
• The OPT pseudo-header for EDNS is sent in the additional section, but not displayed there by dig.
  • It is in the comments section.

    $ dig +noall +additional  ns01a.dnslab.org
    $
    

• By default, dig shows the response message.
  • The +qr option adds the outgoing query.
  • The query is shown after the text: "Sending:"

    $ dig +qr ns01a.dnslab.org
    ; <<>> DiG 9.11.13-RedHat-9.11.13-6.el8_2.1 <<>> +qr ns01a.dnslab.org
    ;; global options: +cmd
    ;; Sending:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57808
    ;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 4096
    ; COOKIE: 7c29201b7c89b54e
    ;; QUESTION SECTION:
    ;ns01a.dnslab.org.                        IN      A
    
    ;; QUERY SIZE: 55
    
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57808
    ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    <OUTPUT SUPPRESSED>
    

• +short disables the default output. It shows only the response's RDATA.
  • Contrast +short, to +answer, which displays the entire RRSet.

    $ dig +short ns01a.dnslab.org
    192.168.53.101
    
    $ dig +noall +answer ns01a.dnslab.org
    ns01a.dnslab.org.         30      IN      A       192.168.53.101
    

• The response displayed by dig is the last received.
  • If a message returns with tc set, the query is resent with TCP. (Therefore, tc is rarely seen in dig output.)
  • The +ignore flag tells dig to not re-query with TCP (Example below.)
  • When dig re-queries with TCP, a message is displayed that can't be suppressed.

    $ dig +noall dnslab.org DNSKEY +dnssec
    ;; Truncated, retrying in TCP mode.
    $
    

• The response is too big, but dig doesn't re-query with TCP.

  $ dig dnslab.org dnskey +dnssec +ignore
  ; <<>> DiG 9.9.4 <<>> dnslab.org dnskey +dns +ignore
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20521
  ;; flags: qr tc rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
  
  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags: do; udp: 800
  
  ;; QUESTION SECTION:
  ;dnslab.org.			IN	DNSKEY
  
  ;; Query time: 9 msec
  ;; SERVER: 192.168.53.251#53(192.168.53.251)
  ;; WHEN: Wed Nov 25 14:05:05 UTC 2015
  ;; MSG SIZE  rcvd: 39