ansible include with array slices

Question

In Ansible 2.2, I want to loop over a large list of files read from S3.

Here is my role/tasks/main.yml

  - name: Simulate variable read from S3
    set_fact:
      list_of_files_to_import: [
        "a.tar.gz",
        "b.tar.gz",
        "c.tar.gz",
        "d.tar.gz",
        "e.tar.gz",
        "f.tar.gz",
        "g.tar.gz",
        "h.tar.gz",
        ...
        "zz.tar.gz"
      ]

  - name: Process each file from S3
    include: submodule.yml
    with_items: list_of_files_to_import

Here is role/tasks/submodule.yml

---
  - name: Restore TABLE {{ item }}
    debug: var={{ item }}

This crashes because there are too many files.

I have found that I can slice the array and send sections at a time:

  - name: Process each file from S3
    include: submodule.yml
    with_items: "{{ list_of_files_to_import[0:5] }}"

  - name: Process each file from S3
    include: submodule.yml
    with_items: "{{ list_of_files_to_import[5:10] }}"

  - name: Process each file from S3
    include: submodule.yml
    with_items: "{{ list_of_files_to_import[10:15] }}"

  - name: Process each file from S3
    include: submodule.yml
    with_items: "{{ list_of_files_to_import[15:20] }}"

Instead of hardcoding all these little blocks, I wanted to try something like

  - name: Process each file from S3
    include: submodule.yml
    with_items: "{{ list_of_files_to_import[{{start}}:{{end}}] }}"

But we cannot get variable-defined variable names

How can I process a large list of items in Ansible 2.2?

How large is the list? And what is the reason of crash? Too many includes? — Konstantin Suvorov, Apr 07 '17 at 15:08
It is on the order of 300 files. The only reason I see is "out of memory." But yes, it seems like it cannot handle so many includes. — Thunder Rabbit, Apr 07 '17 at 15:59
I'm not sure on your details, but it kinda looks like issue [#16391](https://github.com/ansible/ansible/issues/16391). If that's the case the issue should be fixed on the next Ansible release. — nelsonda, Apr 11 '17 at 14:37
@Doomy you may be right. I ended up solving it with Bash. I'll post my solution. — Thunder Rabbit, Apr 11 '17 at 22:05

score 0 · Answer 1 · answered Apr 11 '17 at 22:15

I ended up solving it with a shell script, repeatedly calling the playbook with some --extra-vars to specify which files to process.

This only works because the list of files in S3 have similar filenames. Basically it loops through the filenames and processes each one at a time.

#!/usr/bin/env bash

# return 0(true) if the year/month combination is valid
valid() {
   yr=$1
   mn=$2
   # do not run on months after this month
   if [ "$yr" -eq "2017" -a "$mn" -gt "$(date +%m)" ]
   then
     return 1
   fi

   return 0
}

# For every year from 2002 to this year
for year in `seq 2002 $(date +%Y)`
do
   # for every zero-padded month, 01-12
   for month in `seq -f "%02g" 01 12`
   do
      # for each type of item in inventory
      for object in widgets doodads
      do
          if valid $year $month;
          then
             ansible-playbook playbook_name.yml --extra-vars "object=$object year=$year month=$month"
          else
             echo skipping invalid combo $object $year $month
          fi
      done
   done
done

ansible include with array slices

1 Answers1