Sometimes I want to run a shell script or command for a series of dates, so I have created a bash library function which takes a start date and an end date, and iterates over the dates passing each date to another command/script.

This is moving away from SQL/Database specific content. Much of what I do is around ETL integration, shell (in particular bash) scripting is a big part of ETL and process flow control in unix environments. This is also an opportunity for me to introduce my bash library on github.

I created this when working in an environment where there was a shell script for each data source to be loaded. The scripts took a date as an optional argument, so we could control which date we loaded/processed data for. This came in useful if we needed to reload/reprocess data for a specific date. However, if we needed to run for a range of days (the previous weeks’ numbers require reprocessing) the command would need to be run multiple times.

I wanted to automate commands like these:

~/data_processing_scripts/task_to_run/script_file_name.sh "2018-09-30"
~/data_processing_scripts/task_to_run/script_file_name.sh "2018-10-01"
~/data_processing_scripts/task_to_run/script_file_name.sh "2018-10-02"
~/data_processing_scripts/task_to_run/script_file_name.sh "2018-10-03"
~/data_processing_scripts/task_to_run/script_file_name.sh "2018-10-04"
~/data_processing_scripts/task_to_run/script_file_name.sh "2018-10-05"

Getting the Library

This script (and others to come) is hosted using github.. If you’d like to pull a copy of the bash library from git (feel free). Install git, and then check out the whole library:

# I prefer to have this in the base of my home
cd ~/

# Clone the repository (pulling everything down)
git clone https://github.com/d-roman-halliday/bash_library.git

# Getting updates (when I add new stuff to it)
cd ~/bash_library
git pull

Or, if you just want the one script, copy it from github. Or see below:

# Pull raw text using wget
wget https://raw.githubusercontent.com/d-roman-halliday/bash_library/master/date_loop.bash

View the script on github: date_loop.bash

Including the Library

To load a library into a session (so its functions can be called like applications) precede the library script reference with a dot:

. ~/bash_library/date_loop.bash

Without doing this, the below examples won’t work. Note that a library can be included in another bash script, or for all sessions by adding it to the bashrc file.

Note: I’ve prepended the functions in my library:

• bf – Bash Function: A user facing bash function, prepended for clarity when using tab completion.
• hbf – Helper Bash Function: Bash functions called by the user facing functions, with a different start to limit options for tab completion.

Using the date loop function

The function has these required parameters:

• s – The start date (in format YYYY-MM-DD)
• e – The end date (in format YYYY-MM-DD)
• c – The command that you wish to execute.
• f – The format of the date string to hand to the command (given above), this can be any format string accepted by the date command. See the date man page for options.

There is also a handy help function (using the argument h or H).

david@roman-halliday:~$ bf_date_loop -h

This is the help output:

Take a command and execute it for each date in a range (using date as argument)
    -s  : Start Date YYYY-MM-DD
    -e  : End Date   YYYY-MM-DD
    -c  : Command (string)
    -f  : Format (see date command man page for formats)
 -h -H  : Help (this text)

Examples:
bf_date_loop -s "2018-04-28" -e "2018-05-03" -c "echo date: " -f "%Y%m%d"
bf_date_loop -s "2018-04-28" -e "2018-05-03" -c "date --date " -f "%F"

Example Execution

So, for my opening example the command would be:

bf_date_loop -s "2018-09-30" -e "2018-10-05" -c "data_processing_scripts/task_to_run/script_file_name.sh" -f "%F"

Which has the output (I’ve highlighted the output from my demo script to make clear which output belongs to which application):

Loop From: 2018-09-30
       To: 2018-10-05
==================================
Processing command  : data_processing_scripts/task_to_run/script_file_name.sh
Processing for date : 2018-09-30
Demo script: data_processing_scripts/task_to_run/script_file_name.sh
Processing date: Sun Sep 30 00:00:00 UTC 2018
==================================
Processing command  : data_processing_scripts/task_to_run/script_file_name.sh
Processing for date : 2018-10-01
Demo script: data_processing_scripts/task_to_run/script_file_name.sh
Processing date: Mon Oct  1 00:00:00 UTC 2018
==================================
Processing command  : data_processing_scripts/task_to_run/script_file_name.sh
Processing for date : 2018-10-02
Demo script: data_processing_scripts/task_to_run/script_file_name.sh
Processing date: Tue Oct  2 00:00:00 UTC 2018
==================================
Processing command  : data_processing_scripts/task_to_run/script_file_name.sh
Processing for date : 2018-10-03
Demo script: data_processing_scripts/task_to_run/script_file_name.sh
Processing date: Wed Oct  3 00:00:00 UTC 2018
==================================
Processing command  : data_processing_scripts/task_to_run/script_file_name.sh
Processing for date : 2018-10-04
Demo script: data_processing_scripts/task_to_run/script_file_name.sh
Processing date: Thu Oct  4 00:00:00 UTC 2018
==================================
Processing command  : data_processing_scripts/task_to_run/script_file_name.sh
Processing for date : 2018-10-05
Demo script: data_processing_scripts/task_to_run/script_file_name.sh
Processing date: Fri Oct  5 00:00:00 UTC 2018
==================================