Linux Blog

CORRECTION – Using BASH to sort a book collection. ISBN Data Mining – Part 1

Filed under: Shell Script Sundays — TheLinuxBlog.com at 8:00 am on Sunday, January 25, 2015

This may be cheating but I consider it a break from the download cleanup script.

Amazingly I got a comment out of the blue from an article I wrote in 2007 about ISBN Data Mining. The comment, stated the fact that the script didn’t work. I did a little investigating and was able to find out why. I figured it was just old and didn’t work but that was not the case. Apparently when I formatted my posts for code, a while back it appears that some of the formatting got a bit fubar.

Luckily for me and Gabe I was able to find an old copy:

Here is his script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/bin/bash
ISBN="$1"
 
function fetchInfo () {
  ### Using barnesandnoble.com to fetch info...
  lynx -source "http://search.barnesandnoble.com/booksearch/isbninquiry.asp?ISBN=${ISBN}" |\
   tr -d '[:cntrl:]' | sed 's/></>\n</g' | while read -a lineArray; do
 
  ### Parsing book title.
  if [ "${lineArray[0]}" == "<h1" ]; then
   echo "4|Title: ${lineArray[*]}" | sed 's/<[^>]*>//g;s/ ([^)]*)//g'
 
  ### Parsing book author.
 elif [ "$(echo ${lineArray[*]} | grep "id=\"contributor\"")" ]; then
  echo "3|Author(s): ${lineArray[*]}" | sed 's/by //;s/<[^>]*>//g'
 
  ### Parsing additional data.
  elif [ "${lineArray[0]}" == "<li" ] &&
     [ "$(echo ${lineArray[*]} | grep -ve "bullet" -ve "title")" ]; then
   echo "1|${lineArray[*]}" | sed 's/<[^>]*>//g;s/:/: /;s/  / /'
  fi
 
  done | sort -ur | awk -F\| '{print $2}' | grep ":"
 
}
 
if [ "${#ISBN}" -ge "10" ]; then
   fetchInfo
fi

The script should be saved to a file and called as ./isbn.sh . Amazingly after all of these years it still works, I guess that’s one of the beauties of shell scripting. Here is the output:

owen@linuxblog:~$ isbn.sh 1593275676
Title:  How Linux Works: What Every Superuser Should Know    by  Brian Ward

Here is ISBN Data Mining – Part 2 although, I cannot guarantee that it works after 8 years.

Auto Clean-up Downloaded Files – Part III

Filed under: Shell Script Sundays — TheLinuxBlog.com at 8:00 am on Sunday, January 18, 2015

In Part 2, we added some read prompts to read which directory to run the script in and used some bash if/then/else statements to do some basic input validation. This week by using the creating script parameters with getopts article we’ll enhance the script a little to remove the echo from the example to allow the user to delete the files if they choose, defaulting to not remove files.
(Read on …)

Mac Annoyances and Fixes.

Filed under: General Linux — TheLinuxBlog.com at 6:00 am on Wednesday, January 14, 2015

I got a Mac a while back, out of the box there were a few things that bothered me. This isn’t a comprehensive list, I could go on and on, but here are a few things that are easy to fix.
(Read on …)

Auto Clean-up Downloaded Files – Part II

Filed under: Shell Script Sundays — TheLinuxBlog.com at 8:00 am on Sunday, January 11, 2015

Last week I showed a one liner that could be used to remove duplicate files from your downloads folder. Using previous Shell Script Sunday articles, over the next few weeks we’ll add some additional functionality to make it a little more functional.

On its own the snippet is not that useful. The script will only run in the current directory. Adding a prompt to ask the user what directory to run in, or defaulting to the current directory would be a nice addition. Using the Shell Script to Get User Input article you’ll see that adding some prompts with read is pretty easy. Next we’ll use some bash if/then/else statements to read over that input to check for blank input and a check to make sure that it is a valid directory, exiting if it is not.
(Read on …)

Auto Clean-up Downloaded Files

Filed under: Shell Script Sundays — TheLinuxBlog.com at 11:37 am on Sunday, January 4, 2015

This week I went through my downloads folder, cleaning up erroneous files. In light of that I’ll share a quick tip on how to clean up the multiple copies of files that inevitably pile up. The issue is, when you save a file from firefox or Chrome, the next time you download the file, it just makes another copy with (1) or (2). I have a number of these on multiple systems, so hit the jump for a quick snippet, and explanation.
(Read on …)

Introduction to Investigating Unknown Files on Linux

Filed under: General Linux — TheLinuxBlog.com at 12:00 pm on Saturday, December 20, 2014

Introduction

This article was written by Adam Palmer, a Linux Consultant. It covers the first few steps in basic Linux forensic work – investigating unknown files. Linux provides a range of powerful tools to investigate files and filesystems.

Preparing an Environment

First and foremost, before performing any kind of forensics work on Linux, it’s important to set up a usable environment. Even if you intend to perform ‘read only’ actions, and not run any binaries for example, buffer overflows and other exploit vectors have existed in the tools we’ll be using and so running them on maliciously crafted files could result in a system compromise.
If you intend to analyze files, I recommend using a sandboxed virtual machine with no networking access. The virtual machine should be destroyed and recreated as required. If you plan to analyze a hard disk, I recommend using a write blocker to ensure that nothing can physically be altered on the disk under investigation.

Finding File Type & Status

Using the `file’ utility, we can match the file’s header to a known database:

root@kali:~# file rack.png
rack.png: PNG image data, 576 x 576, 8-bit/color RGBA, non-interlaced

Using the `stat’ utility, we can find out permissions, as well as disk, inode and meta information on the file:

root@kali:~# stat rack.png
  File: `/opt/metasploit/apps/pro/ui/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/contrib/rack.png'
  Size: 23805     	Blocks: 48         IO Block: 4096   regular file
Device: 801h/2049d	Inode: 6165        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-12-04 23:45:51.087400697 +0000
Modify: 2013-04-17 20:50:40.000000000 +0100
Change: 2013-04-27 18:10:38.317049114 +0100
 Birth: -

Hexdump

Let’s confirm the `file’ output. According to Wikipedia’s entry on PNGs, a PNG begins with an 8-byte signature: 89 50 4E 47 0D 0A 1A 0A. Let’s use `hexdump’ to confirm this:

root@kali:~# hexdump -C -n8 rack.png 
00000000  89 50 4e 47 0d 0a 1a 0a                           |.PNG....|

Note the arguments provided. -C will print the ASCII data alongside the hexdecimal output, whilst -n8 displays 8 bytes.

Strings

Using `strings’, we are able to locate any ASCII printable strings within a file:

root@kali:~# strings rack.png
IHDR
sRGB
bKGD
	pHYs
tIME
[...]

In this case, there’s little of interest to see, however let’s look at running `strings’ on an executable binary:

root@kali:~# strings /bin/ls
[...]
Try `%s --help' for more information.
Usage: %s [OPTION]... [FILE]...
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
Mandatory arguments to long options are mandatory for short options too.
  -a, --all                  do not ignore entries starting with .
  -A, --almost-all           do not list implied . and ..
[...]

Interesting.. strings can be used to uncover hidden printable ASCII.

Linked Libraries

Using `ldd’, we can investigate any shared libraries that have been linked in:

root@kali:~# ldd /bin/ls
	linux-gate.so.1 =&gt;  (0xb77a7000)
	libselinux.so.1 =&gt; /lib/i386-linux-gnu/libselinux.so.1 (0xb776c000)
	librt.so.1 =&gt; /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xb7763000)
	libacl.so.1 =&gt; /lib/i386-linux-gnu/libacl.so.1 (0xb7758000)
	libc.so.6 =&gt; /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb75f5000)
	libdl.so.2 =&gt; /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb75f1000)
	/lib/ld-linux.so.2 (0xb77a8000)
	libpthread.so.0 =&gt; /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb75d8000)
	libattr.so.1 =&gt; /lib/i386-linux-gnu/libattr.so.1 (0xb75d2000)

Tracing system calls

Using `strace’ we are able to trace system calls. If running an unknown binary, it is especially important to perform this step within a safe sandboxed environment. `strace’ sends its output to stderr, and so we’ll need to redirect that to stdout before being able to work with it using standard stream utilities such as `grep’. Let’s look at any open calls made when running `/bin/ls':

root@kali:~# strace -f /bin/ls 2&gt;&amp;1|grep "open"
open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/lib/i386-linux-gnu/libselinux.so.1", O_RDONLY) = 3
open("/lib/i386-linux-gnu/i686/cmov/librt.so.1", O_RDONLY) = 3
[...]

The listed entries are required libraries being opened

Further research

Further steps in debugging binaries can be achieved through `ltrace’, `gdb’, `objdump’ and `ndisasm’. Some of these tools have a significant learning curve however enable detailed debugging of unknown binaries

Media Purge

Filed under: General Linux,Linux Software — TheLinuxBlog.com at 8:23 pm on Tuesday, September 16, 2014

In order to reduce the amount of crap stuff I have laying around, in addition to making a box of old hardware to donate, I’ve been going through old Linux CD’s. Deciding which ones to pitch was a harder task than I first thought, so I figured I’d take a methodical approach to it. The criteria was this:

Have I:

Used it recently?
Ever used it?

Does it:

Have a download available?
Have a current community?
Contain customized content?
Have a special meaning?

Will:

I ever use it?
Someone else be able to use it?

Since most of these were burned disks, the only reason to keep something would be if it is not available anymore, has a special meaning, or I’ve used it a bunch and it contains customized content. Based on this, I’ve decided to pitch (in no particular order)
(Read on …)

TheLinuxBlog.com Wrote a Post. You wont believe what happens next.

Filed under: The Linux Blog News — TheLinuxBlog.com at 11:08 pm on Monday, September 8, 2014

It happened, I finally got around to writing a post. Ever since the demise of Google Reader it has been sort of a downward spiral for this blog, my last post almost a year ago. From being number 1 in the search engines for “Linux Blog” to somewhere down on page who knows where and what desperate folks click those links?

I’ve not lost interest in Linux (I’ve used it every day since before the conception of this blog), or writing in general, but perhaps I’ve lost interest or have a lack of time for writing about Linux. There isn’t much to write about that hasn’t already been covered or can’t be for lack of a better term (and because no one uses Bing) “Googled.” I’d ask, what do people want to read about, but I could probably google that, or write about what I think people want to hear about, but that would just be regurgitating content for the sake of it.

This blogs not dead, I’m just going to shift topics a bit and write about what interests me about Linux and tech in general, projects I’m working on or problems encountered and see where that takes it. If I take it back to the beginning and make it fun again perhaps I’ll write more.

Until next time:
“Sorry no catchy closing here”

off to delete the accumulated comment spam now.

xrandr – Set Primary Monitor

Filed under: Linux Hardware,Linux Software,Shell Script Sundays — Owen at 11:04 pm on Sunday, October 27, 2013

I had an issue with my dual monitor setup where my primary monitor was my second, but only in X. Rearranging the monitors in Gnome preferences did nothing to solve the problem. While not exactly a shell script, here is a one-liner to change your primary monitor with xrandr.

#!/bin/bash
xrandr --output DVI-0 --primary

The above uses xrandr to set the primary to DVI-0. I put this in my ~/bin folder, chmod’d and set it to start when Gnome starts. Problem solved!

Raspberry Pi – Awesome!

Filed under: General Linux,Linux Hardware — TheLinuxBlog.com at 3:17 pm on Thursday, October 24, 2013

Raspberry Pi

I never jumped on the Pi bandwagon, sure I thought it was cool but when I wanted one, there were supply demands and the want wore off. I recently purchased a Model B revision Two and have to say I’m very impressed. It is an awesome piece of hardware but what really makes the Raspberry Pi great is the community that has been built around them. There are many projects and tutorials based and plenty of hackers working on tweaking and expanding them. Here are a few of my favorite projects, incase you’ve been living under a rock for the past couple of years like me:
(Read on …)

Login Script to Phone Home

Filed under: Quick Linux Tutorials — TheLinuxBlog.com at 11:59 pm on Monday, October 7, 2013

If you’re a little paranoid like me, you often wonder what will happen if your laptop gets stolen. I’ve seen news articles and the like where an thief happened to steal a laptop and got caught because they stole the wrong persons laptop.


Today we have a one liner that will phone home when a user logs in. While this wont work if you have a password on your laptop, which is recommended, if you keep a dummy account called “User” or “Guest” with no password and the thief happens to log in, you could be in luck.

#!/bin/bash 
ssh -N -R2222:localhost:22 <user>@<yourhost> -p<port> -i /home/<user>/.ssh/id_rsa

The simple SSH command opens up a remote port 2222 to the local port 22 which of course requires SSH to be running locally. It also uses the ssh identity file, for ano password ssh login, and the -N is for no shell. Set it up as an application that starts on login and if that account is set to auto connect to WiFi, it will connect as the user logs in. If you wanted to take it a step further you could combine it with autossh to continue trying to connect. It will also help if you have a static IP or DNS setup so that it will be able to connect if your device unfortunately goes missing.

Next Page »