Linux Blog

Shell scripts is a really useful skill to have. Creating a script to do avoid repeating a task can save you time in the long run. What a lot of people don’t know is that shell scripting is not hard, especially if you have some Linux command line experience. You can pretty much do anything you want with a script, and they are great for automating tasks. To create a shell script from a one-liner all you really have to do is:

echo "[your one-liner here]" > [your-script-name-here]

That will create your file, which you can then change the permissions on and move to your ~/bin/ directory. It would be wise to add $!/bin/bash as the first line if bash is your shell of choice, if you choose to distribute it. Once you’ve translated your one-liner into a file that can be executed you can start adding functionality to the script to ease use for future use.

Here are some explanations of basic functionality you can add to your scripts:
Variables

If-then-else statements If then else’s can be used to control flow or make decisions, they are very useful indeed.

read can be used to get input from the user, when dealing with an unknown or a variable

functions are great to use to store a particular set of instructions that can be called repeatedly without having to re-write the script.

loops do exactly as it sounds, loop. That is re-iterate over a variable, or repeat an instruction. You could use a loop to call a function over and over until a clause is met.

getopt’s is an instruction that can be used to read input from when the script is called. Often this is used to change functionality or display usage information.

redirection techniques are used throughout shell scripts and is one of the fundamentals of shell scripting. Master this and you’ll be piping and redirecting output to files and other programs in no time.

error checking and handling, often overlooked but shouldn’t be underestimated. Checking for errors before they happen can save time, and undesired results.

Many of these techniques are covered throughout this blog (feel free to browse or search), and there are many great online resources and books available (both free and paid) to help you with your journey. A good place to start for most of the techniques is in my Shell Scripting 101 article. Good luck!

This may be cheating but I consider it a break from the download cleanup script.

Amazingly I got a comment out of the blue from an article I wrote in 2007 about ISBN Data Mining. The comment, stated the fact that the script didn’t work. I did a little investigating and was able to find out why. I figured it was just old and didn’t work but that was not the case. Apparently when I formatted my posts for code, a while back it appears that some of the formatting got a bit fubar.

Luckily for me and Gabe I was able to find an old copy:

Here is his script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#!/bin/bash
ISBN="$1"
 
function fetchInfo () {
  ### Using barnesandnoble.com to fetch info...
  lynx -source "http://search.barnesandnoble.com/booksearch/isbninquiry.asp?ISBN=${ISBN}" |\
   tr -d '[:cntrl:]' | sed 's/></>\n</g' | while read -a lineArray; do
 
  ### Parsing book title.
  if [ "${lineArray[0]}" == "<h1" ]; then
   echo "4|Title: ${lineArray[*]}" | sed 's/<[^>]*>//g;s/ ([^)]*)//g'
 
  ### Parsing book author.
 elif [ "$(echo ${lineArray[*]} | grep "id=\"contributor\"")" ]; then
  echo "3|Author(s): ${lineArray[*]}" | sed 's/by //;s/<[^>]*>//g'
 
  ### Parsing additional data.
  elif [ "${lineArray[0]}" == "<li" ] &&
     [ "$(echo ${lineArray[*]} | grep -ve "bullet" -ve "title")" ]; then
   echo "1|${lineArray[*]}" | sed 's/<[^>]*>//g;s/:/: /;s/  / /'
  fi
 
  done | sort -ur | awk -F\| '{print $2}' | grep ":"
 
}
 
if [ "${#ISBN}" -ge "10" ]; then
   fetchInfo
fi

The script should be saved to a file and called as ./isbn.sh . Amazingly after all of these years it still works, I guess that’s one of the beauties of shell scripting. Here is the output:

owen@linuxblog:~$ isbn.sh 1593275676
Title:  How Linux Works: What Every Superuser Should Know    by  Brian Ward

Here is ISBN Data Mining – Part 2 although, I cannot guarantee that it works after 8 years.

In Part 2, we added some read prompts to read which directory to run the script in and used some bash if/then/else statements to do some basic input validation. This week by using the creating script parameters with getopts article we’ll enhance the script a little to remove the echo from the example to allow the user to delete the files if they choose, defaulting to not remove files.
(Read on …)

I got a Mac a while back, out of the box there were a few things that bothered me. This isn’t a comprehensive list, I could go on and on, but here are a few things that are easy to fix.
(Read on …)

Last week I showed a one liner that could be used to remove duplicate files from your downloads folder. Using previous Shell Script Sunday articles, over the next few weeks we’ll add some additional functionality to make it a little more functional.

On its own the snippet is not that useful. The script will only run in the current directory. Adding a prompt to ask the user what directory to run in, or defaulting to the current directory would be a nice addition. Using the Shell Script to Get User Input article you’ll see that adding some prompts with read is pretty easy. Next we’ll use some bash if/then/else statements to read over that input to check for blank input and a check to make sure that it is a valid directory, exiting if it is not.
(Read on …)

This week I went through my downloads folder, cleaning up erroneous files. In light of that I’ll share a quick tip on how to clean up the multiple copies of files that inevitably pile up. The issue is, when you save a file from firefox or Chrome, the next time you download the file, it just makes another copy with (1) or (2). I have a number of these on multiple systems, so hit the jump for a quick snippet, and explanation.
(Read on …)

Introduction

This article was written by Adam Palmer, a Linux Consultant. It covers the first few steps in basic Linux forensic work – investigating unknown files. Linux provides a range of powerful tools to investigate files and filesystems.

Preparing an Environment

First and foremost, before performing any kind of forensics work on Linux, it’s important to set up a usable environment. Even if you intend to perform ‘read only’ actions, and not run any binaries for example, buffer overflows and other exploit vectors have existed in the tools we’ll be using and so running them on maliciously crafted files could result in a system compromise.
If you intend to analyze files, I recommend using a sandboxed virtual machine with no networking access. The virtual machine should be destroyed and recreated as required. If you plan to analyze a hard disk, I recommend using a write blocker to ensure that nothing can physically be altered on the disk under investigation.

Finding File Type & Status

Using the `file’ utility, we can match the file’s header to a known database:

root@kali:~# file rack.png
rack.png: PNG image data, 576 x 576, 8-bit/color RGBA, non-interlaced

Using the `stat’ utility, we can find out permissions, as well as disk, inode and meta information on the file:

root@kali:~# stat rack.png
  File: `/opt/metasploit/apps/pro/ui/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/contrib/rack.png'
  Size: 23805     	Blocks: 48         IO Block: 4096   regular file
Device: 801h/2049d	Inode: 6165        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-12-04 23:45:51.087400697 +0000
Modify: 2013-04-17 20:50:40.000000000 +0100
Change: 2013-04-27 18:10:38.317049114 +0100
 Birth: -

Hexdump

Let’s confirm the `file’ output. According to Wikipedia’s entry on PNGs, a PNG begins with an 8-byte signature: 89 50 4E 47 0D 0A 1A 0A. Let’s use `hexdump’ to confirm this:

root@kali:~# hexdump -C -n8 rack.png 
00000000  89 50 4e 47 0d 0a 1a 0a                           |.PNG....|

Note the arguments provided. -C will print the ASCII data alongside the hexdecimal output, whilst -n8 displays 8 bytes.

Strings

Using `strings’, we are able to locate any ASCII printable strings within a file:

root@kali:~# strings rack.png
IHDR
sRGB
bKGD
	pHYs
tIME
[...]

In this case, there’s little of interest to see, however let’s look at running `strings’ on an executable binary:

root@kali:~# strings /bin/ls
[...]
Try `%s --help' for more information.
Usage: %s [OPTION]... [FILE]...
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
Mandatory arguments to long options are mandatory for short options too.
  -a, --all                  do not ignore entries starting with .
  -A, --almost-all           do not list implied . and ..
[...]

Interesting.. strings can be used to uncover hidden printable ASCII.

Linked Libraries

Using `ldd’, we can investigate any shared libraries that have been linked in:

root@kali:~# ldd /bin/ls
	linux-gate.so.1 =>  (0xb77a7000)
	libselinux.so.1 => /lib/i386-linux-gnu/libselinux.so.1 (0xb776c000)
	librt.so.1 => /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xb7763000)
	libacl.so.1 => /lib/i386-linux-gnu/libacl.so.1 (0xb7758000)
	libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb75f5000)
	libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb75f1000)
	/lib/ld-linux.so.2 (0xb77a8000)
	libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb75d8000)
	libattr.so.1 => /lib/i386-linux-gnu/libattr.so.1 (0xb75d2000)

Tracing system calls

Using `strace’ we are able to trace system calls. If running an unknown binary, it is especially important to perform this step within a safe sandboxed environment. `strace’ sends its output to stderr, and so we’ll need to redirect that to stdout before being able to work with it using standard stream utilities such as `grep’. Let’s look at any open calls made when running `/bin/ls’:

root@kali:~# strace -f /bin/ls 2>&1|grep "open"
open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/lib/i386-linux-gnu/libselinux.so.1", O_RDONLY) = 3
open("/lib/i386-linux-gnu/i686/cmov/librt.so.1", O_RDONLY) = 3
[...]

The listed entries are required libraries being opened

Further research

Further steps in debugging binaries can be achieved through `ltrace’, `gdb’, `objdump’ and `ndisasm’. Some of these tools have a significant learning curve however enable detailed debugging of unknown binaries

In order to reduce the amount of crap stuff I have laying around, in addition to making a box of old hardware to donate, I’ve been going through old Linux CD’s. Deciding which ones to pitch was a harder task than I first thought, so I figured I’d take a methodical approach to it. The criteria was this:

Have I:

Used it recently?
Ever used it?

Does it:

Have a download available?
Have a current community?
Contain customized content?
Have a special meaning?

Will:

I ever use it?
Someone else be able to use it?

Since most of these were burned disks, the only reason to keep something would be if it is not available anymore, has a special meaning, or I’ve used it a bunch and it contains customized content. Based on this, I’ve decided to pitch (in no particular order)
(Read on …)

It happened, I finally got around to writing a post. Ever since the demise of Google Reader it has been sort of a downward spiral for this blog, my last post almost a year ago. From being number 1 in the search engines for “Linux Blog” to somewhere down on page who knows where and what desperate folks click those links?

I’ve not lost interest in Linux (I’ve used it every day since before the conception of this blog), or writing in general, but perhaps I’ve lost interest or have a lack of time for writing about Linux. There isn’t much to write about that hasn’t already been covered or can’t be for lack of a better term (and because no one uses Bing) “Googled.” I’d ask, what do people want to read about, but I could probably google that, or write about what I think people want to hear about, but that would just be regurgitating content for the sake of it.

This blogs not dead, I’m just going to shift topics a bit and write about what interests me about Linux and tech in general, projects I’m working on or problems encountered and see where that takes it. If I take it back to the beginning and make it fun again perhaps I’ll write more.

Until next time:
“Sorry no catchy closing here”

off to delete the accumulated comment spam now.

I had an issue with my dual monitor setup where my primary monitor was my second, but only in X. Rearranging the monitors in Gnome preferences did nothing to solve the problem. While not exactly a shell script, here is a one-liner to change your primary monitor with xrandr.

#!/bin/bash
xrandr --output DVI-0 --primary

The above uses xrandr to set the primary to DVI-0. I put this in my ~/bin folder, chmod’d and set it to start when Gnome starts. Problem solved!

I never jumped on the Pi bandwagon, sure I thought it was cool but when I wanted one, there were supply demands and the want wore off. I recently purchased a Model B revision Two and have to say I’m very impressed. It is an awesome piece of hardware but what really makes the Raspberry Pi great is the community that has been built around them. There are many projects and tutorials based and plenty of hackers working on tweaking and expanding them. Here are a few of my favorite projects, incase you’ve been living under a rock for the past couple of years like me:
(Read on …)