Polishing Your Curl Expertise

Previous post covers bare minimum of curl you need to know for testing RESTful microservices. Read it first if you need basics. This writing focuses on corner cases and advanced options, making curl experience more enjoyable.

Microservice for experiments

For demonstrations, I've created a simple RESTful microservice in Golang. Use it if you have nothing to experiment with.

package main

import (  
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "net/url"
)

type RequestSummary struct {  
    URL     string
    Method  string
    Headers http.Header
    Params  url.Values
    Auth    string
    Body    string
}

func main() {  
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        bytes, err := ioutil.ReadAll(r.Body)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }

        auth := ""
        if user, pass, ok := r.BasicAuth(); ok {
            auth = user + ":" + pass
        }

        rs := RequestSummary{
            URL:     r.URL.RequestURI(),
            Method:  r.Method,
            Headers: r.Header,
            Params:  r.URL.Query(),
            Auth:    auth,
            Body:    string(bytes),
        }

        resp, err := json.MarshalIndent(&rs, "", "\t")
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        w.Write(resp)
        w.Write([]byte("\n"))
    })

    http.ListenAndServe(":8080", nil)
    fmt.Println("Exiting...")
}
Suppressing progress meter

curl displays progress meter while fetching data and hides it when data transfer is over. Seeing progress is nice, but it causes an issue when you pipe curl with less for reading a long response. When you do it, progress meter mixes with response, making output clumsy:

$ curl -X GET http://localhost:8080/something | less

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   171  100   171    0     0   4228      0 --:--:-- --:--:-- --:--:--  4275  
{
        "URL": "/something",
        "Method": "GET",
        "Headers": {
                "Accept": [
                        "*/*"
                ],
                "User-Agent": [
                        "curl/7.43.0"
                ]
        },
        "Params": {},
        "Auth": null,
        "Body": ""
}

To suppress progress meter, curl can be switched to silent mode with -s (or --silent) option:

$ curl -X GET -s http://localhost:8080/something | less

{
        "URL": "/something",
        "Method": "GET",
        "Headers": {
                "Accept": [
                        "*/*"
                ],
                "User-Agent": [
                        "curl/7.43.0"
                ]
        },
        "Params": {},
        "Auth": null,
        "Body": ""
}

This solves issue with progress meter, but introduce another one - silent mode also suppresses error output:

$ curl -X GET -s http://unknownhost:8080/something | less

(There should have been an error that host cannot be resolved)

A full solution is to use -s together with -S (or --show-error), which displays errors in silent mode:

$ curl -X GET -s -S http://unknownhost:8080/something | less

curl: (6) Could not resolve host: unknownhost  
Authentication

To authenticate a user with a password, use -u <user>:<password> (or --user) option.

Don't pass password in command line though, unless you really know what you do. This is insecure, because executed command line (including your password) will be stored in history and log.

The right way is to pass user only. When password is not provided, curl will ask for it and let you type it securely:

$ curl -X GET -u login http://localhost:8080/something
Enter host password for user 'login':  
{
    "URL": "/something",
    "Method": "GET",
    "Headers": {
        "Accept": [
            "*/*"
        ],
        "Authorization": [
            "Basic bG9naW46cGFzc3dvcmQ="
        ],
        "User-Agent": [
            "curl/7.43.0"
        ]
    },
    "Params": {},
    "Auth": "login:password",
    "Body": ""
}
Outputting metadata

Sometimes you need to output information about response, instead of response itself. Option -w <format> (or --write-out <format>) lets you fully control the output and is often used to display response metadata, e.g. HTTP status code:

$ curl -X GET -w "HTTP code is %{http_code}\n" -o /dev/null -Ss http://localhost:8080/something
HTTP code is 200  

Note variable http_code is specified as %{http_code}. See curl documentation for list of available variables.

When you control output with -w option, response is often not needed and suppressed, e.g. with -o /dev/null.

Debugging

When something goes wrong, the first step is usually to look at what is sent to endpoint and what returns back. You can do it with -v (or --verbose) option:

curl -X GET -v http://localhost:8080/something  
*   Trying ::1...
* Connected to localhost (::1) port 8080 (#0)
> GET /something HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK  
< Date: Tue, 23 Aug 2016 22:49:21 GMT  
< Content-Length: 169  
< Content-Type: text/plain; charset=utf-8  
<  
{
    "URL": "/something",
    "Method": "GET",
    "Headers": {
        "Accept": [
            "*/*"
        ],
        "User-Agent": [
            "curl/7.43.0"
        ]
    },
    "Params": {},
    "Auth": "",
    "Body": ""
}
* Connection #0 to host localhost left intact
Tracing

When debugging doesn't help and you need to dig deeper, try --trace <file> instead:

$ curl -X GET --trace t.txt http://localhost:8080/something
{
    "URL": "/something",
    "Method": "GET",
    "Headers": {
        "Accept": [
            "*/*"
        ],
        "User-Agent": [
            "curl/7.43.0"
        ]
    },
    "Params": {},
    "Auth": "",
    "Body": ""
}

$ cat t.txt
== Info:   Trying ::1...
== Info: Connected to localhost (::1) port 8080 (#0)
=> Send header, 87 bytes (0x57)
0000: 47 45 54 20 2f 73 6f 6d 65 74 68 69 6e 67 20 48 GET /something H  
0010: 54 54 50 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 6c TTP/1.1..Host: l  
0020: 6f 63 61 6c 68 6f 73 74 3a 38 30 38 30 0d 0a 55 ocalhost:8080..U  
0030: 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f ser-Agent: curl/  
0040: 37 2e 34 33 2e 30 0d 0a 41 63 63 65 70 74 3a 20 7.43.0..Accept:  
0050: 2a 2f 2a 0d 0a 0d 0a                            */*....  
<= Recv header, 17 bytes (0x11)  
0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.  
0010: 0a                                              .  
<= Recv header, 37 bytes (0x25)  
0000: 44 61 74 65 3a 20 54 75 65 2c 20 32 33 20 41 75 Date: Tue, 23 Au  
0010: 67 20 32 30 31 36 20 32 32 3a 34 33 3a 30 35 20 g 2016 22:43:05  
0020: 47 4d 54 0d 0a                                  GMT..  
<= Recv header, 21 bytes (0x15)  
0000: 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 Content-Length:  
0010: 31 36 39 0d 0a                                  169..  
<= Recv header, 41 bytes (0x29)  
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te  
0010: 78 74 2f 70 6c 61 69 6e 3b 20 63 68 61 72 73 65 xt/plain; charse  
0020: 74 3d 75 74 66 2d 38 0d 0a                      t=utf-8..  
<= Recv header, 2 bytes (0x2)  
0000: 0d 0a                                           ..  
<= Recv data, 169 bytes (0xa9)  
0000: 7b 0a 09 22 55 52 4c 22 3a 20 22 2f 73 6f 6d 65 {.."URL": "/some  
0010: 74 68 69 6e 67 22 2c 0a 09 22 4d 65 74 68 6f 64 thing",.."Method  
0020: 22 3a 20 22 47 45 54 22 2c 0a 09 22 48 65 61 64 ": "GET",.."Head  
0030: 65 72 73 22 3a 20 7b 0a 09 09 22 41 63 63 65 70 ers": {..."Accep  
0040: 74 22 3a 20 5b 0a 09 09 09 22 2a 2f 2a 22 0a 09 t": [...."*/*"..  
0050: 09 5d 2c 0a 09 09 22 55 73 65 72 2d 41 67 65 6e .],..."User-Agen  
0060: 74 22 3a 20 5b 0a 09 09 09 22 63 75 72 6c 2f 37 t": [...."curl/7  
0070: 2e 34 33 2e 30 22 0a 09 09 5d 0a 09 7d 2c 0a 09 .43.0"...]..},..  
0080: 22 50 61 72 61 6d 73 22 3a 20 7b 7d 2c 0a 09 22 "Params": {},.."  
0090: 41 75 74 68 22 3a 20 22 22 2c 0a 09 22 42 6f 64 Auth": "",.."Bod  
00a0: 79 22 3a 20 22 22 0a 7d 0a                      y": "".}.  
== Info: Connection #0 to host localhost left intact
comments powered by Disqus