Intersting shell scripts

Byobu is a GPLv3 open source text-based window manager and terminal multiplexer.

byobu

For loop to unzip a bunch of zips files in the same directory.

#!/bin/bash

#Stop shell splitting on spaces
ORIGINAL_IFS=$IFS #internal field separator
IFS=$(echo -n "\n")
for FILENAME in $(ls *.zip)
do
    echo "Unzipping $FILENAME..."
    unzip -o $FILENAME
done
IFS=$ORIGINAL_IFS

Determine size on disk of directories.

for FOLDER in $(ls); do du -sh --apparent-size $FOLDER; done 

The following will sort it by size.

for FOLDER in $(ls); do du -sh –apparent-size $FOLDER; done | sort -h

Determine size of specefic folder.

du -sh --apparent-size <folder_name>

Make list of all files in a directory, recursively.

find . -type f > ../filenames.txt

Given text document containing a listing of directories and filenames (paths). One can use the following commands to search for filename types. This will output a count of files with filenames ending with .docx.

cat ../filenames.txt | grep -i .docx$ | wc -l

Give the same document as the preceding command. One can use the following commands to search for filename types. This will output a count of files with filenames ending with .docx where the path begins with <directory_name. 

cat ../filenames.txt | grep ^./<directory_name>/ | grep -i .docx$ | wc -l

To search CSV files containing "find this,"  use the below command. The -i argument in this command makes the search case insensitive. The */*.csv argument is the path to search tell grep to open every folder, then each file within those folders that ends in .csv, and search for the "find this" keyword.

grep -i "find this" */*.csv

If you have docker and docker-compose installed, start an Ubuntu container as follows.

sudo docker run -it ubuntu:latest bash

Remove all stopped containers at once.

sudo docker container prune

Display all running Docker containers.

sudo docker ps

Kill a running docker container.

sudo docker kill <container_name>

Start an Ubuntu docker container by mount a volume for persistent data. Mount type will be volume.

sudo docker run -it --mount type=volume,src=test-data,dst=/mnt
ubuntu:latest bash

Display docker volumes.

sudo docker volume ls

Remove docker volumes. Note volumes can only be removed if containers have been completely removed.

sudo docker volume rm <volume_name>

Start an Kali docker container by mount a volume for persistent data. Mount type will be bind.

sudo docker run -it --mount type=bind,src=/host/folder,dst=/container/folder
kalilinux/kali-rolling bash

Start a docker container passing database login environment variables to be used by the container. 

sudo docker run -it -e DB_USER=root -e DB_PASSWORD=yourefired ubuntu:latest bash

Verify environmental variables in container.

echo $DB_PASSWORD

Start jupyter docker container with forwarded ports to host. On the host use localhost/127.0.01 with host port to connect to container port.

sudo docker run -p host_port:container_port jupyter/scipy-notebook:latest

Docker can create a lot of files. To free up space quickly run the following command. Since this command doesn’t delete volumes, it won’t delete any of your important data.

sudo docker system prune -a

Services defined in the same docker-compose.yaml file will be part of the same docker network.

To drop into a container's bash, run the following command.

sudo docker-compose run --rm <name_of_container> bash

Running your own local Aleph server with Docker on Linux

Reference Hacks, Leaks, and Revelation Book by Micah Lee

First obtain docker-compose.yml and aleph.env.tmpl files from GitHub. Copy files to the directory you'll use as the Aleph docker root (i.e., aleph). 

Comment out - "~:/host" in docker-compose.yml. 

Rename aleph.env.tmpl to aleph.env. If only one user will be using this server update ALEPH_SINGLE_USER in aleph.env to true. 

Using the below command, update result for ALEPH_SECRET_KEY in aleph.env.

openssl rand -hex 24

Ensure your computers vm.max_map_count is greater or equal to 262144. Check vm.map_count value:

sudo sysctl vm.max_map_count

Update vm.max_map_count value:

sudo sysctl -w vm.max_map_count=262144

Start server.

sudo docker-compose up

If first time starting aleph docker instance then, in a separate window, run following command after above command finishes. Once below command finishes, you should be all set.

sudo docker-compose run --rm shell aleph upgrade

To administer your aleph instance you must use bash terminal. The following command allows you to do that.

sudo docker-compose run --rm shell bash

Run the following to see the list of commands Aleph supports. And if you want to learn more about a command add --help on it.

aleph --help

Adding data to Aleph. First ensure your Aleph instance is running, then run the following command. This command will provide readonly (ro) access to the files you want to index. Change source and destination as necessary.

sudo docker-compose run --rm -v source:destination:ro shell bash

After above command finshes you'l be at the aleph docker terminal. At this point you can index the directory you want with this command. Where destination is the path you used in the previous command. Change the lang (-l) on the language of the files you want to index.

Note: Aleph uses Tesseract to do OCR.

aleph crawldir -l eng destination/some-directory

Depending on your system and size of data, indexing may take a long time. In the meantime you can view the status with the following commands.

View all datasets and investigations (known as collections).

aleph collections

Check status.

aleph status