{"id":223,"date":"2018-10-13T18:00:28","date_gmt":"2018-10-13T18:00:28","guid":{"rendered":"https:\/\/datablog.roman-halliday.com\/?p=223"},"modified":"2018-09-01T17:28:09","modified_gmt":"2018-09-01T17:28:09","slug":"iterating-dates-on-the-command-line","status":"publish","type":"post","link":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/","title":{"rendered":"Iterating Dates On The bash Command Line"},"content":{"rendered":"<p>Sometimes I want to run a shell script or command for a series of dates, so I have created a bash library function which takes a start date and an end date, and iterates over the dates passing each date to another command\/script.<\/p>\n<p>This is moving away from SQL\/Database specific content. Much of what I do is around ETL integration, shell (in particular bash) scripting is a big part of ETL and process flow control in unix environments. This is also an opportunity for me to introduce my <a href=\"https:\/\/github.com\/d-roman-halliday\/bash_library\">bash library on github<\/a>.<\/p>\n<p>I created this when working in an environment where there was a shell script for each data source to be loaded. The scripts took a date as an optional argument, so we could control which date we loaded\/processed data for. This came in useful if we needed to reload\/reprocess data for a specific date. However, if we needed to run for a range of days (the previous weeks&#8217; numbers require reprocessing) the command would need to be run multiple times.<\/p>\n<p>I wanted to automate commands like these:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\">~\/data_processing_scripts\/task_to_run\/script_file_name.sh \"2018-09-30\"\r\n~\/data_processing_scripts\/task_to_run\/script_file_name.sh\u00a0\"2018-10-01\"\r\n~\/data_processing_scripts\/task_to_run\/script_file_name.sh \"2018-10-02\"\r\n~\/data_processing_scripts\/task_to_run\/script_file_name.sh \"2018-10-03\"\r\n~\/data_processing_scripts\/task_to_run\/script_file_name.sh \"2018-10-04\"\r\n~\/data_processing_scripts\/task_to_run\/script_file_name.sh \"2018-10-05\"<\/pre>\n<h1>Getting the Library<\/h1>\n<p>This script (and others to come) is hosted using github.. If you&#8217;d like to pull a copy of the bash library from git (feel free). Install git, and then check out the whole library:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># I prefer to have this in the base of my home\r\ncd ~\/\r\n\r\n# Clone the repository (pulling everything down)\r\ngit clone https:\/\/github.com\/d-roman-halliday\/bash_library.git\r\n\r\n# Getting updates (when I add new stuff to it)\r\ncd ~\/bash_library\r\ngit pull<\/pre>\n<p>Or, if you just want the one script, copy it from github. Or see below:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># Pull raw text using wget\r\nwget https:\/\/raw.githubusercontent.com\/d-roman-halliday\/bash_library\/master\/date_loop.bash<\/pre>\n<p>View the script on github:\u00a0<a href=\"https:\/\/github.com\/d-roman-halliday\/bash_library\/blob\/master\/date_loop.bash\" target=\"_blank\" rel=\"noopener\">date_loop.bash<\/a><\/p>\n<h1>Including the Library<\/h1>\n<p>To load a library into a session (so its functions can be called like applications) precede the library script reference with a dot:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\">. ~\/bash_library\/date_loop.bash<\/pre>\n<p>Without doing this, the below examples won&#8217;t work. Note that a library can be included in another bash script, or for all sessions by adding it to the <code>bashrc<\/code> file.<\/p>\n<p>Note: I&#8217;ve prepended the functions in my library:<\/p>\n<ul>\n<li><code>bf<\/code> &#8211; Bash Function: A user facing bash function, prepended for clarity when using tab completion.<\/li>\n<li><code>hbf<\/code> &#8211; Helper Bash Function: Bash functions called by the user facing functions, with a different start to limit options for tab completion.<\/li>\n<\/ul>\n<h1>Using the date loop function<\/h1>\n<p>The function has these required parameters:<\/p>\n<ul>\n<li>s &#8211; The start date (in format <code>YYYY-MM-DD<\/code>)<\/li>\n<li>e &#8211; The end\u00a0date (in format <code>YYYY-MM-DD<\/code>)<\/li>\n<li>c &#8211; The command that you wish to execute.<\/li>\n<li>f &#8211; The format of the date string to hand to the command (given above), this can be any format string accepted by the date command. See the <a href=\"http:\/\/man7.org\/linux\/man-pages\/man1\/date.1.html\">date man page<\/a> for options.<\/li>\n<\/ul>\n<p>There is also a handy help function (using the argument <code>h<\/code> or <code>H<\/code>).<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\">david@roman-halliday:~$ bf_date_loop -h<\/pre>\n<p>This is the help output:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">Take a command and execute it for each date in a range (using date as argument)\r\n    -s  : Start Date YYYY-MM-DD\r\n    -e  : End Date   YYYY-MM-DD\r\n    -c  : Command (string)\r\n    -f  : Format (see date command man page for formats)\r\n -h -H  : Help (this text)\r\n\r\nExamples:\r\nbf_date_loop -s \"2018-04-28\" -e \"2018-05-03\" -c \"echo date: \" -f \"%Y%m%d\"\r\nbf_date_loop -s \"2018-04-28\" -e \"2018-05-03\" -c \"date --date \" -f \"%F\"\r\n<\/pre>\n<h2>Example Execution<\/h2>\n<p>So, for my opening example the command would be:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\">bf_date_loop -s \"2018-09-30\" -e \"2018-10-05\" -c \"data_processing_scripts\/task_to_run\/script_file_name.sh\" -f \"%F\"<\/pre>\n<p>Which has the output (I&#8217;ve highlighted the output from my demo script to make clear which output belongs to which application):<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-highlight=\"6,7,11,12,16,17,21,22,26,27,31,32\">Loop From: 2018-09-30\r\n       To: 2018-10-05\r\n==================================\r\nProcessing command  : data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing for date : 2018-09-30\r\nDemo script: data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing date: Sun Sep 30 00:00:00 UTC 2018\r\n==================================\r\nProcessing command  : data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing for date : 2018-10-01\r\nDemo script: data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing date: Mon Oct  1 00:00:00 UTC 2018\r\n==================================\r\nProcessing command  : data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing for date : 2018-10-02\r\nDemo script: data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing date: Tue Oct  2 00:00:00 UTC 2018\r\n==================================\r\nProcessing command  : data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing for date : 2018-10-03\r\nDemo script: data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing date: Wed Oct  3 00:00:00 UTC 2018\r\n==================================\r\nProcessing command  : data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing for date : 2018-10-04\r\nDemo script: data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing date: Thu Oct  4 00:00:00 UTC 2018\r\n==================================\r\nProcessing command  : data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing for date : 2018-10-05\r\nDemo script: data_processing_scripts\/task_to_run\/script_file_name.sh\r\nProcessing date: Fri Oct  5 00:00:00 UTC 2018\r\n==================================\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sometimes I want to run a shell script or command for a series of dates, so I have created a bash library function which takes a start date and an end date, and iterates over the dates passing each date to another command\/script. This is moving away from SQL\/Database specific content. Much of what I&hellip;<\/p>\n<p class=\"read-more\"><a class=\"readmore-btn\" href=\"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/\">Read More<span class=\"screen-reader-text\">  Read More<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,21],"tags":[23],"class_list":["post-223","post","type-post","status-publish","format-standard","hentry","category-bash","category-linux","tag-bash-library"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Iterating Dates On The bash Command Line - Rows Across The Lake<\/title>\n<meta name=\"description\" content=\"I have created a bash library function which takes a start date and an end date, and iterates over the dates passing each date to another command\/script.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Iterating Dates On The bash Command Line - Rows Across The Lake\" \/>\n<meta property=\"og:description\" content=\"I have created a bash library function which takes a start date and an end date, and iterates over the dates passing each date to another command\/script.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/\" \/>\n<meta property=\"og:site_name\" content=\"Rows Across The Lake\" \/>\n<meta property=\"article:published_time\" content=\"2018-10-13T18:00:28+00:00\" \/>\n<meta name=\"author\" content=\"david\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@d_roman_h\" \/>\n<meta name=\"twitter:site\" content=\"@d_roman_h\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"david\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/index.php\\\/2018\\\/10\\\/13\\\/iterating-dates-on-the-command-line\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/index.php\\\/2018\\\/10\\\/13\\\/iterating-dates-on-the-command-line\\\/\"},\"author\":{\"name\":\"david\",\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/#\\\/schema\\\/person\\\/575f96d2590c3085923ff9e1b565748b\"},\"headline\":\"Iterating Dates On The bash Command Line\",\"datePublished\":\"2018-10-13T18:00:28+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/index.php\\\/2018\\\/10\\\/13\\\/iterating-dates-on-the-command-line\\\/\"},\"wordCount\":471,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/#\\\/schema\\\/person\\\/575f96d2590c3085923ff9e1b565748b\"},\"keywords\":[\"bash library\"],\"articleSection\":[\"bash\",\"Linux\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/datablog.roman-halliday.com\\\/index.php\\\/2018\\\/10\\\/13\\\/iterating-dates-on-the-command-line\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/index.php\\\/2018\\\/10\\\/13\\\/iterating-dates-on-the-command-line\\\/\",\"url\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/index.php\\\/2018\\\/10\\\/13\\\/iterating-dates-on-the-command-line\\\/\",\"name\":\"Iterating Dates On The bash Command Line - Rows Across The Lake\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/#website\"},\"datePublished\":\"2018-10-13T18:00:28+00:00\",\"description\":\"I have created a bash library function which takes a start date and an end date, and iterates over the dates passing each date to another command\\\/script.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/index.php\\\/2018\\\/10\\\/13\\\/iterating-dates-on-the-command-line\\\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/datablog.roman-halliday.com\\\/index.php\\\/2018\\\/10\\\/13\\\/iterating-dates-on-the-command-line\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/index.php\\\/2018\\\/10\\\/13\\\/iterating-dates-on-the-command-line\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Iterating Dates On The bash Command Line\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/#website\",\"url\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/\",\"name\":\"Rows Across The Lake\",\"description\":\"Data &amp; Databases\",\"publisher\":{\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/#\\\/schema\\\/person\\\/575f96d2590c3085923ff9e1b565748b\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/datablog.roman-halliday.com\\\/#\\\/schema\\\/person\\\/575f96d2590c3085923ff9e1b565748b\",\"name\":\"david\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/acddbc676a1d5c73795edcf0627ee39e5aa947da9033b58373e03d93122cb3b7?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/acddbc676a1d5c73795edcf0627ee39e5aa947da9033b58373e03d93122cb3b7?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/acddbc676a1d5c73795edcf0627ee39e5aa947da9033b58373e03d93122cb3b7?s=96&d=mm&r=g\",\"caption\":\"david\"},\"logo\":{\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/acddbc676a1d5c73795edcf0627ee39e5aa947da9033b58373e03d93122cb3b7?s=96&d=mm&r=g\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Iterating Dates On The bash Command Line - Rows Across The Lake","description":"I have created a bash library function which takes a start date and an end date, and iterates over the dates passing each date to another command\/script.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/","og_locale":"en_GB","og_type":"article","og_title":"Iterating Dates On The bash Command Line - Rows Across The Lake","og_description":"I have created a bash library function which takes a start date and an end date, and iterates over the dates passing each date to another command\/script.","og_url":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/","og_site_name":"Rows Across The Lake","article_published_time":"2018-10-13T18:00:28+00:00","author":"david","twitter_card":"summary_large_image","twitter_creator":"@d_roman_h","twitter_site":"@d_roman_h","twitter_misc":{"Written by":"david","Estimated reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/#article","isPartOf":{"@id":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/"},"author":{"name":"david","@id":"https:\/\/datablog.roman-halliday.com\/#\/schema\/person\/575f96d2590c3085923ff9e1b565748b"},"headline":"Iterating Dates On The bash Command Line","datePublished":"2018-10-13T18:00:28+00:00","mainEntityOfPage":{"@id":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/"},"wordCount":471,"commentCount":0,"publisher":{"@id":"https:\/\/datablog.roman-halliday.com\/#\/schema\/person\/575f96d2590c3085923ff9e1b565748b"},"keywords":["bash library"],"articleSection":["bash","Linux"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/","url":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/","name":"Iterating Dates On The bash Command Line - Rows Across The Lake","isPartOf":{"@id":"https:\/\/datablog.roman-halliday.com\/#website"},"datePublished":"2018-10-13T18:00:28+00:00","description":"I have created a bash library function which takes a start date and an end date, and iterates over the dates passing each date to another command\/script.","breadcrumb":{"@id":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/datablog.roman-halliday.com\/index.php\/2018\/10\/13\/iterating-dates-on-the-command-line\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/datablog.roman-halliday.com\/"},{"@type":"ListItem","position":2,"name":"Iterating Dates On The bash Command Line"}]},{"@type":"WebSite","@id":"https:\/\/datablog.roman-halliday.com\/#website","url":"https:\/\/datablog.roman-halliday.com\/","name":"Rows Across The Lake","description":"Data &amp; Databases","publisher":{"@id":"https:\/\/datablog.roman-halliday.com\/#\/schema\/person\/575f96d2590c3085923ff9e1b565748b"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datablog.roman-halliday.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":["Person","Organization"],"@id":"https:\/\/datablog.roman-halliday.com\/#\/schema\/person\/575f96d2590c3085923ff9e1b565748b","name":"david","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/secure.gravatar.com\/avatar\/acddbc676a1d5c73795edcf0627ee39e5aa947da9033b58373e03d93122cb3b7?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/acddbc676a1d5c73795edcf0627ee39e5aa947da9033b58373e03d93122cb3b7?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/acddbc676a1d5c73795edcf0627ee39e5aa947da9033b58373e03d93122cb3b7?s=96&d=mm&r=g","caption":"david"},"logo":{"@id":"https:\/\/secure.gravatar.com\/avatar\/acddbc676a1d5c73795edcf0627ee39e5aa947da9033b58373e03d93122cb3b7?s=96&d=mm&r=g"}}]}},"_links":{"self":[{"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/posts\/223","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/comments?post=223"}],"version-history":[{"count":7,"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/posts\/223\/revisions"}],"predecessor-version":[{"id":232,"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/posts\/223\/revisions\/232"}],"wp:attachment":[{"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/media?parent=223"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/categories?post=223"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datablog.roman-halliday.com\/index.php\/wp-json\/wp\/v2\/tags?post=223"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}