Polishing Your Curl Expertise

Previous post covers bare minimum of curl you need to know for testing RESTful microservices. Read it first if you need basics. This writing focuses on corner cases and advanced options, making curl experience more enjoyable.

Microservice for experiments

For demonstrations, I’ve created a simple RESTful microservice in Golang. Use it if you have nothing to experiment with.

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
)

type RequestSummary struct {
	URL     string
	Method  string
	Headers http.Header
	Params  url.Values
	Auth    string
	Body    string
}

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		bytes, err := ioutil.ReadAll(r.Body)
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		auth := ""
		if user, pass, ok := r.BasicAuth(); ok {
			auth = user + ":" + pass
		}

		rs := RequestSummary{
			URL:     r.URL.RequestURI(),
			Method:  r.Method,
			Headers: r.Header,
			Params:  r.URL.Query(),
			Auth:    auth,
			Body:    string(bytes),
		}

		resp, err := json.MarshalIndent(&rs, "", "\t")
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}
		w.Write(resp)
		w.Write([]byte("\n"))
	})

	http.ListenAndServe(":8080", nil)
	fmt.Println("Exiting...")
}

Suppressing progress meter

curl displays progress meter while fetching data and hides it when data transfer is over. Seeing progress is nice, but it causes an issue when you pipe curl with less for reading a long response. When you do it, progress meter mixes with response, making output clumsy:

$ curl -X GET http://localhost:8080/something | less

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   171  100   171    0     0   4228      0 --:--:-- --:--:-- --:--:--  4275
{
        "URL": "/something",
        "Method": "GET",
        "Headers": {
                "Accept": [
                        "*/*"
                ],
                "User-Agent": [
                        "curl/7.43.0"
                ]
        },
        "Params": {},
        "Auth": null,
        "Body": ""
}

To suppress progress meter, curl can be switched to silent mode with -s (or --silent) option:

$ curl -X GET -s http://localhost:8080/something | less

{
        "URL": "/something",
        "Method": "GET",
        "Headers": {
                "Accept": [
                        "*/*"
                ],
                "User-Agent": [
                        "curl/7.43.0"
                ]
        },
        "Params": {},
        "Auth": null,
        "Body": ""
}

This solves issue with progress meter, but introduce another one - silent mode also suppresses error output:

$ curl -X GET -s http://unknownhost:8080/something | less

(There should have been an error that host cannot be resolved)

A full solution is to use -s together with -S (or --show-error), which displays errors in silent mode:

$ curl -X GET -s -S http://unknownhost:8080/something | less

curl: (6) Could not resolve host: unknownhost

Authentication

To authenticate a user with a password, use -u <user>:<password> (or --user) user:password option.

Don’t pass password in command line though, unless you really know what you do. This is insecure, because executed command line (including your password) will be stored in history and log.

The right way is to pass user only. When password is not provided, curl will ask for it and let you type it securely:

$ curl -X GET -u login http://localhost:8080/something
Enter host password for user 'login':
{
    "URL": "/something",
    "Method": "GET",
    "Headers": {
        "Accept": [
            "*/*"
        ],
        "Authorization": [
            "Basic bG9naW46cGFzc3dvcmQ="
        ],
        "User-Agent": [
            "curl/7.43.0"
        ]
    },
    "Params": {},
    "Auth": "login:password",
    "Body": ""
}

Outputting metadata

Sometimes you need to output information about response, instead of response itself. Option -w <format> (or --write-out <format>) lets you fully control the output and is often used to display response metadata, e.g. HTTP status code:

$ curl -X GET -w "HTTP code is %{http_code}\n" -o /dev/null -Ss http://localhost:8080/something
HTTP code is 200

Note variable http_code is specified as %{http_code}. See curl documentation for list of available variables.

When you control output with -w option, response is often not needed and suppressed, e.g. with -o /dev/null.

Debugging

When something goes wrong, the first step is usually to look at what is sent to endpoint and what returns back. You can do it with -v (or --verbose) option:

curl -X GET -v http://localhost:8080/something
*   Trying ::1...
* Connected to localhost (::1) port 8080 (#0)
> GET /something HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 23 Aug 2016 22:49:21 GMT
< Content-Length: 169
< Content-Type: text/plain; charset=utf-8
<
{
    "URL": "/something",
    "Method": "GET",
    "Headers": {
        "Accept": [
            "*/*"
        ],
        "User-Agent": [
            "curl/7.43.0"
        ]
    },
    "Params": {},
    "Auth": "",
    "Body": ""
}
* Connection #0 to host localhost left intact

Tracing

When debugging doesn’t help and you need to dig deeper, try --trace <file> instead:

$ curl -X GET --trace t.txt http://localhost:8080/something
{
    "URL": "/something",
    "Method": "GET",
    "Headers": {
        "Accept": [
            "*/*"
        ],
        "User-Agent": [
            "curl/7.43.0"
        ]
    },
    "Params": {},
    "Auth": "",
    "Body": ""
}

$ cat t.txt
== Info:   Trying ::1...
== Info: Connected to localhost (::1) port 8080 (#0)
=> Send header, 87 bytes (0x57)
0000: 47 45 54 20 2f 73 6f 6d 65 74 68 69 6e 67 20 48 GET /something H
0010: 54 54 50 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 6c TTP/1.1..Host: l
0020: 6f 63 61 6c 68 6f 73 74 3a 38 30 38 30 0d 0a 55 ocalhost:8080..U
0030: 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f ser-Agent: curl/
0040: 37 2e 34 33 2e 30 0d 0a 41 63 63 65 70 74 3a 20 7.43.0..Accept:
0050: 2a 2f 2a 0d 0a 0d 0a                            */*....
<= Recv header, 17 bytes (0x11)
0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a                                              .
<= Recv header, 37 bytes (0x25)
0000: 44 61 74 65 3a 20 54 75 65 2c 20 32 33 20 41 75 Date: Tue, 23 Au
0010: 67 20 32 30 31 36 20 32 32 3a 34 33 3a 30 35 20 g 2016 22:43:05
0020: 47 4d 54 0d 0a                                  GMT..
<= Recv header, 21 bytes (0x15)
0000: 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 Content-Length:
0010: 31 36 39 0d 0a                                  169..
<= Recv header, 41 bytes (0x29)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te
0010: 78 74 2f 70 6c 61 69 6e 3b 20 63 68 61 72 73 65 xt/plain; charse
0020: 74 3d 75 74 66 2d 38 0d 0a                      t=utf-8..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a                                           ..
<= Recv data, 169 bytes (0xa9)
0000: 7b 0a 09 22 55 52 4c 22 3a 20 22 2f 73 6f 6d 65 {.."URL": "/some
0010: 74 68 69 6e 67 22 2c 0a 09 22 4d 65 74 68 6f 64 thing",.."Method
0020: 22 3a 20 22 47 45 54 22 2c 0a 09 22 48 65 61 64 ": "GET",.."Head
0030: 65 72 73 22 3a 20 7b 0a 09 09 22 41 63 63 65 70 ers": {..."Accep
0040: 74 22 3a 20 5b 0a 09 09 09 22 2a 2f 2a 22 0a 09 t": [...."*/*"..
0050: 09 5d 2c 0a 09 09 22 55 73 65 72 2d 41 67 65 6e .],..."User-Agen
0060: 74 22 3a 20 5b 0a 09 09 09 22 63 75 72 6c 2f 37 t": [...."curl/7
0070: 2e 34 33 2e 30 22 0a 09 09 5d 0a 09 7d 2c 0a 09 .43.0"...]..},..
0080: 22 50 61 72 61 6d 73 22 3a 20 7b 7d 2c 0a 09 22 "Params": {},.."
0090: 41 75 74 68 22 3a 20 22 22 2c 0a 09 22 42 6f 64 Auth": "",.."Bod
00a0: 79 22 3a 20 22 22 0a 7d 0a                      y": "".}.
== Info: Connection #0 to host localhost left intact