Linux Blog

cURL Gotcha’s

Filed under: Shell Script Sundays — at 4:31 pm on Sunday, February 17, 2008

I’ve been using cURL for a couple of projects recently and I thought I would just post a couple of the “Gotcha’s”

Feel free to add to the list by leaving a comment.

1) User Agent. Certain websites especially Google like to block the use of curl because some people use curl for abusive reasons. This can be fixed by changing your user agent.

User Agents can be switched with curl by using the -A or –user-agent switch.

To change your user agent to Internet Explorer 7 or IE7 on Vista do the following when requesting a page:

curl -A "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" [URL]

Should you want to change your user agent to Netscape 4.8 or Opera 9.2 on Vista you can use the following agent strings:

Netscape user agent string

Mozilla/4.8 [en] (Windows NT 6.0; U)

Opera user agent string

Opera/9.20 (Windows NT 6.0; U; en)

2)  Separate post data with ampersands or put spaces in between your -d’s  This one got me once.

3) Don’t try to post and try to use -G for get requests if you want to post data.

-G makes everything that is in -d get put into a get request instead of a post. Use the following format if you want to post and use get requests.

curl -d "post=data&more_post=moredata" urlgoeshere.php?get=getdata

I’ll post more of these as I remember them, again as stated above if something has got you with curl post them here and I’ll add it to the list with a link to your site! Thats all for this week – Owen.