How to specify 'os' for free gitlab runners ('runs-on' equivalent)

You can specify that you want gitlab to execute your CI job on a windows machine using tags.

Gitlab made Windows Shared Runners available as Beta in Jan 2020. Their announcement shows how to use the tags key in your .gitlab-ci file to specify Windows machines. You can also run in a docker container in windows:

https://docs.gitlab.com/runner/executors/docker.html#using-windows-containers

For example, I was able to execute a simple "hello world" powershell script with the following job in my .gitlab-ci file:

windows:
  script: "build/windows/buildExe.ps1"
  tags:
    - shared-windows
    - windows
    - windows-1809

And here's the buildExe.ps1 powershell script reference above:

Write-Host "listing contents of C drive root"
Get-ChildItem -Path C:\ -Force

Write-Host "listing contents of cwd"
Get-ChildItem -Force

The above config produced the following output in my CI job:

[0KRunning with gitlab-runner 12.9.0 (4c96e5ad)
[0;m[0K  on windows-shared-runners-manager 6QgxEPvR
[0;msection_start:1590842482:prepare_executor
[0K[0K[36;1mPreparing the "custom" executor[0;m
[0;m[0KUsing Custom executor with driver autoscaler dev (64a348d)...
[0;mCreating virtual machine for the job...
Virtual machine created!
section_end:1590842624:prepare_executor
[0Ksection_start:1590842624:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on PACKER-5E557E8E via
runner-6qgxepvr-wsrm-87cf700a3349de62a18e...
section_end:1590842640:prepare_script
[0Ksection_start:1590842640:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Initialized empty Git repository in C:/GitLab-Runner/builds/maltfield/cross-platform-python-gui/.git/
[32;1mCreated fresh repository.[0;m
[32;1mChecking out 91f0d9e3 as master...[0;m
From https://gitlab.com/maltfield/cross-platform-python-gui
 * [new ref]         refs/pipelines/151176370 -> refs/pipelines/151176370
 * [new branch]      master                   -> origin/master
git-lfs/2.8.0 (GitHub; windows amd64; go 1.12.2; git 30af66bb)

[32;1mUpdating/initializing submodules...[0;m
git-lfs/2.8.0 (GitHub; windows amd64; go 1.12.2; git 30af66bb)
section_end:1590842661:get_sources
[0Ksection_start:1590842661:restore_cache
[0K[0K[36;1mRestoring cache[0;m
[0;msection_end:1590842670:restore_cache
[0Ksection_start:1590842670:download_artifacts
[0K[0K[36;1mDownloading artifacts[0;m
[0;msection_end:1590842677:download_artifacts
[0Ksection_start:1590842677:build_script
[0K[0K[36;1mRunning before_script and script[0;m
[0;m[32;1m$ build/windows/buildExe.ps1[0;m
listing contents of C drive root

    Directory: C:\

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d--hs-       12/11/2019   2:47 AM                $RECYCLE.BIN
d--hs-        2/25/2020   8:52 PM                Boot
d--hsl       12/11/2019  10:29 AM                Documents and Settings
d-----        2/25/2020   8:16 PM                Git
d-----        5/30/2020  12:44 PM                GitLab-Runner
d-----        2/25/2020   8:16 PM                GitLFS
d-----        2/25/2020   8:34 PM                Go
d-----        9/15/2018   7:12 AM                PerfLogs
d-r---        2/25/2020   8:35 PM                Program Files
d-----        2/25/2020   8:27 PM                Program Files (x86)
d--h--        2/25/2020   8:51 PM                ProgramData
d--hs-        2/25/2020   8:10 PM                Recovery
d--hs-       12/11/2019  10:28 AM                System Volume Information
d-----        2/25/2020   8:32 PM                tools
d-r---        5/30/2020  12:43 PM                Users
d-----        2/25/2020   8:38 PM                vcpkg
d-----        2/25/2020   8:51 PM                Windows
-arhs-        2/25/2020   8:45 PM         408834 bootmgr
-a-hs-        9/15/2018   7:09 AM              1 BOOTNXT
-a-hs-        5/30/2020  12:41 PM     1073741824 pagefile.sys
listing contents of cwd

    Directory: C:\GitLab-Runner\builds\maltfield\cross-platform-python-gui

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d--h--        5/30/2020  12:44 PM                .git
d-----        5/30/2020  12:44 PM                build
d-----        5/30/2020  12:44 PM                src
-a----        5/30/2020  12:44 PM            534 .gitlab-ci.yml
-a----        5/30/2020  12:44 PM          35787 LICENSE
-a----        5/30/2020  12:44 PM           1171 README.md
-a----        5/30/2020  12:44 PM              6 requirements.txt

section_end:1590842686:build_script
[0Ksection_start:1590842686:archive_cache
[0K[0K[36;1mSaving cache[0;m
[0;msection_end:1590842694:archive_cache
[0Ksection_start:1590842694:upload_artifacts_on_success
[0K[0K[36;1mUploading artifacts for successful job[0;m
[0;msection_end:1590842701:upload_artifacts_on_success
[0K[32;1mJob succeeded
[0;m

As for MacOS, gitlab currently doesn't support it natively. If this changes, updated answers are welcome.

Other answers showing how to run MacOS inside a VM (eg using vagrant and libvirt) on a gitlab shared runner are very welcome :)


Advantages of Linux Containers over Docker

Docker is a specific implementation of Linux containers, or if you want to be more precise Docker is a distribution of tools that includes runc which is an implementation of Linux containers. Other implementations include rkt, LXC, LXD, and (I think) Snappy from Ubuntu.


How to make custom SLES (SuSe Linux Enterprise) Dockerfiles?

As @Tensibai already indicated in one of the comments, some packages are missing:

Use 'zypper addrepo' or 'zypper modifyrepo' commands to add or enable repositories.

Either use zypper addrepo or zypper modifyrepo in your dockerfile to solve the issue.


How can I get a list of encrypted disks for instances that are running on the AWS CLI?

Here is what I ended up using. Enjoy.

for instance in $instances;
do
  count=`aws ec2 describe-volumes --filters Name=attachment.status,Values=attached Name=attachment.instance-id,Values=$instance --query "Volumes[]"  | jq  -r '. | length';`
  name=`aws ec2 describe-tags --filters Name=resource-id,Values=$instance Name=key,Values=Name --query Tags[].Value | jq -r '.[0]'`
  if [ $count -gt 0 ]; then
    START=0
    END=$count
    for ((i=START; i<END; i++))
    do
       #echo "i: $i"
       encrypted=`aws ec2 describe-volumes --filters Name=attachment.status,Values=attached Name=attachment.instance-id,Values=$instance --query "Volumes[]"  | jq  -r ".[$i].Encrypted";`
       volumeid=`aws ec2 describe-volumes --filters Name=attachment.status,Values=attached Name=attachment.instance-id,Values=$instance --query "Volumes[]"  | jq  -r ".[$i].VolumeId";`
       echo "$instance $name Volumes: $count VolumeId: $volumeid Encrypted: $encrypted "
    done
  fi
done


Limit throughput between Jenkins and Jenkins Agent

hmmm, tuff one.  dont think it's possible do do this directly
counter question, why do you wish to limit traffic ?  If you could do this, say through a network config of some kind, it owuld negatively impact build time.
From experience things that have an impact on traffic between agent and sever :

Artifacts: if you are storing artifacts within your builds they will be stored on the server. So at these points in your build process you will upload them from the agent to the server and depending on artifact size it will generate traffic.
Sync: Depending on what source control you use (if any) and how you use it it's possible that the sync to occur on the master and then transfered over to the agent.  You can probably change this so the sync only occur on the agent which will pretty much eliminate this part of the traffic between them.
Logs: all logs generated on the your jobs will be transfered back to the server.  Though to max out a connection here you'dhave to generate a whole lot of it !

It's posible some plugins can generate their own traffic.  Without knowing more on your tasks it's difficult to go further.
As to the actual question my first reflex here would be to look for ways to achieve this through a firewall or some other network system to throttle traffic.  Some will allow to target a specific application in addition to source/target IPs in their rules.


Q: "Check if all items are own by root."
A: Put the list of the tools into the variable audit_tools. Compare the length of the lists. For example
- hosts: localhost
  vars:
    audit_tools:
      - auditctl
      - aureport
      - ausearch
      - autrace
      - auditd
      - audispd
      - augenrules
  tasks:
    - block:
        - stat:
            path: "/sbin/{{ item }}"
          loop: "{{ audit_tools }}"
          register: result
        - assert:
            that: no_audit_tools == no_owner_root
            fail_msg: "One or more tools are not own by root."
          vars:
            no_audit_tools: "{{ audit_tools|length }}"
            no_owner_root: "{{ result.results|
                               json_query('[?stat.pw_name==`root`]')|
                               length }}"
      rescue:
        - debug:
            msg: "Rescue: audit tools not owned by root."

If not all items are owned by root assert will fail and the block will proceed to the rescue section
TASK [assert] ****
fatal: [localhost]: FAILED! => changed=false
  assertion: no_audit_tools == no_owner_root
  evaluated_to: false
  msg: One or more tools are not own by root.

TASK [debug] ****
ok: [localhost] =>
  msg: 'Rescue: audit tools not owned by root.'

Q: "This solution requires JMESPath to be installed. Is there an alternative solution?"
A: Yes. It is. Use Jinja filter selectattr
            no_owner_root: "{{ result.results|
                               selectattr('stat.pw_name', 'eq', 'root')|
                               list|length }}"


monitor for docker stats similar to htop?

bcicen on githup with tool ctop is working for me
ref. https://github.com/bcicen/ctop


How to move EC2 instance to another ECS cluster

You need to follow these steps:

Set ECS_CLUSTER=devcluster in /etc/ecs/ecs.config
Stop all tasks/containers
Remove checkpoint file - /var/lib/ecs/data/ecs_agent_data.json
Start ECS agent again as explained here - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html

To make sure it uses correct cluster, check logs - docker logs ecs-agent. It should have a message like:

[INFO] Registration completed successfully. I am running as 'arn:aws:ecs:eu-west-1:ACCOUNT_ID:container-instance/CLUSTER_ID' in cluster 'devcluster'


Azure deploy pipeline with/without approval on separated enviroments

Environment approvals work per-stage. Split your jobs into separate stages:
stages:
  - stage: development
    condition: contains(variables['Build.SourceBranch'], 'refs/heads/develop')
    jobs:
    - deployment: Deploy_Dev
      displayName: 'Deploy Develop to Dev Machine'
      pool:
        name: Development
      environment: 'development'
      strategy:
        runOnce:
          deploy:
            steps:
              - checkout: none
              - task: Bash@3
                inputs:
                  targetType: 'inline'
                  script: |
                    echo $(hostname)
                  workingDirectory: '/var/www'
                  noProfile: false
                  noRc: false

  - stage: production
    condition: contains(variables['Build.SourceBranch'], 'refs/heads/master')
    jobs:
    - deployment: Deploy_Master_to_Hotfix
      displayName: 'Deploy Master to Hotfix Machine'
      pool:
        name: Hotfix
      environment: 'production'
      strategy:
        runOnce:
          deploy:
            steps:
              - checkout: none
              - task: Bash@3
                inputs:
                  targetType: 'inline'
                  script: |
                      echo $(hostname)
                  workingDirectory: '/var/www'
                  noProfile: false
                  noRc: false


How to specify a filter for Kubectl -o flag=custom-column

Since the result is JSON, pipe the result to jq and go from there.
kubectl get | jq '.items[0].metadata.name'

yields
➜ cat abc.json | jq '.items[0].metadata.name'
"kubernetes-bootcamp-5c69669756-cxhwf"


What are the signs of an understaffed DevOps team?

There are four main reasons why you can feel your team is understaffed:

Poor organization and planning of work
Doing work someone else should be doing
Doing work that should not be done at all
Being actually understaffed

Start with a review of the first three points. Read the Phoenix Project on ideas how to do the first. Ask yourself for every task you help anyone with if it should be done at all and if it is you that should be doing the task or if you should simply enable whoever needs it done to do it themselves. This will give you some documentation on why all the work you do is necessary.

Next review the four types of work mentioned in the Phoenix project:

Business Projects - what you do for other teams in the organization
Internal Projects - what you do to make your work easier in future
Scheduled Maintenance - what you do to keep the lights on
Unplanned Interrupts - what you do because something went wrong

If the work of your team is sustainable, you will spend roughly the same amount of time on each of the four. If the unplanned work starts to creep up close to 50% of your time, it is a sign you are definitely understaffed.

You should be able to hire to stay about one person ahead of the unplanned work reaching 25% of your time, otherwise, one person leaving will send your entire team into a tailspin you might never recover from. Overprovisioning of people and technology have same reasons and benefits.


Running two containers in a kubernetes job each with their own initContainers

What you are describing is best achieved by either using two jobs or let task 1 also be an init Container as they run in sequence.
Alternatively you could use Argo Workflows.
https://github.com/argoproj/argo


Hosted CI/CD for iOS that's not in the US?

One could run Jenkins on MacOS.

https://medium.com/@dixitakansha15/continuous-integration-in-ios-using-jenkins-8e081753cc72

https://savvyapps.com/blog/continuous-integration-ios-jenkins


Create backend Address pool for basic load balancer azure arm template

I had the same issue, I downloaded the Template from Azure which effectively had my Load Balancer's Backend Address Pool as a "dependsOn" and as a "property".
This is under the load balancer's "dependsOn" list :
    "[resourceId('Microsoft.Network/loadBalancers/backendAddressPools', 'LB-db-phi-dbphipri', 'LoadBalancerBEAddressPool')]"

And the property under the Load Balancer's property list:
    "backendAddressPools": [
      {
        "name": "LoadBalancerBEAddressPool",
        "properties": {}
      }
    ]

Lastly, I had a child definition of the backendAddressPool as well
{
  "type": "Microsoft.Network/loadBalancers/backendAddressPools",
  "apiVersion": "2020-05-01",
  "name": "LB-db-phi-dbphipri/LoadBalancerBEAddressPool",
  "properties": {}
}

You don't need both. In my case I removed the "child" and the "dependsOn" and just have the property defined under the load balancer.


How to promote a build when certain value in output is above threshold?

What you are looking for is at the bottom of the GitHub README describing Job DSL for the Promoted Builds Plugin. Link to GitHub is on every plugin's page. There are also links to plugins it depends on and you should read the documentation for those like the Job DSL plugin wiki. You can also ask questions in the Job DSL forum or simply read the documentation. Here is a plugin that extends the promotion syntax of the DSL.


How to log native commands run by Ansible Playbook

Set the log file to use inline or in Ansible config, then run ansible-playbook with verbose option:
ANSIBLE_LOG_PATH="ansible-$(date +%Y%m%d%H%M%S).log" \
ansible-playbook -vvvv \
    --diff \
    -i inventory/myinventory \
    myplaybook.yml

ANSIBLE_LOG_PATH - is an environment variable that maps to the log_path configuration item. Sets the file to log to. When using an environment variable you can use shell substitution to add the date and time to the file name so the next run doesn't overwrite the log file.
-vvvv - be very very very verbose. Will show SSH commands executed.
--diff - bonus, when making a change show what has been done.
the rest are standard options.

The log will be in a file in the current directory named something like ansible-20200818155322.log


Are there dialects of YAML or just differences in parsing it?

There is only one YAML spec. That would lead me to say "There are no YAML dialects". There are certainly differences in implementation, perhaps due to what the article claims as the spec's ambiguity:

  YAML specification is so ambiguous,

It's a matter of opinion, I believe, but there is no doubting that various language-specific parsers of the same data result in different output. So, to answer the question(s):

no there are no YAML dialects (since there is one spec)
yes the spec provides much leeway for implementation and thus there are differences in how it is parsed.


How can I access port 80 of another container, when port 80 is already exposed?

You can use a proxy to forward traffic between the two services, I typically do something like this which includes automated SSL with certbot. In the example below I have two services listening on port 80/443 (Sonarqube and Jenkins). You can also see I can reference container hosts by name (https://jenkins:8080)
nginx:
container_name: nginx-certbot
restart: unless-stopped
image: staticfloat/nginx-certbot
networks:
  - local
ports:
  - "0.0.0.0:80:80"
  - "0.0.0.0:443:443"
environment:
  CERTBOT_EMAIL: <insert_email>
volumes:
  - ./conf.d:/etc/nginx/user.conf.d:ro
  - letsencrypt:/etc/letsencrypt

And this is what my nginx config looks like
server {
listen              443 ssl;
server_name jenkins.hauntedmansion.io;
ssl_certificate     /etc/letsencrypt/live/jenkins.hauntedmansion.io/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/jenkins.hauntedmansion.io/privkey.pem;

client_max_body_size 20M;

  location / {
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_pass         http://jenkins:8080;
    proxy_read_timeout 90;
    proxy_redirect     http://127.0.0.1:8080 https://jenkins.hauntedmansion.io;
  }
}

If you want to see the full docker compose example check it out here:
https://gist.github.com/cvega/66e0e5a2815e9b923da2c616f74dea7e


Are they in the same VPC? If yes it should just work as the VPC CIDR route will take care of all the subnets intercommunication.


This is something I have implemented recently.
NOTE: I'm assuming you are aware ho to add users. or users in group in this case.
here is what I have applied:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: readonly_role
rules:
- apiGroups:
  - ""
  resourceNames:
  - ""
  resources:
  - "pods"
  - "pods/logs"
  - "deployments"
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - authorization.k8s.io
  resources:
  - selfsubjectaccessreviews
  - selfsubjectrulesreviews
  verbs:
  - watch
  - get
  - list

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: bind_readonly_dev_role
subjects:
- kind: Group
  name: myorg:readonly
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: readonly_role
  apiGroup: rbac.authorization.k8s.io

You can use either ClusterRole & ClusterRoleBinding or else just Role & Rolebinding, depending on your requirement. In case you are not looking for cluster scoped access, instead just a namespace, I would strongly suggest going with the Role & Rolebinding kind of object.
What I'm doing/creating in the above?
I'm creating one RBAC Role (readonly_role) scoped to Cluster.
and in that I'm specifying I want to let access pods, logs (i.e pods/logs), and deployments. additionally you. can also add things like services or else if you just want to give access to all specify "" instead.
in the 2nd part, I'm creating a ClusterRoleBinding Object named bind_readonly_dev_role  which ensures only users in group myorg:readonly are having this role or privileges in the role we just created (readonly_role ).
NOTE: Incase you want to allow proxy access to certain applications, like kubectl proxy.
you might have to specify them separately in the ClusterRole rules.
I do like this for one of my svc:
- apiGroups:
  - ""
  resourceNames:
  - dashboard:5000
  resources:
  - services/proxy
  verbs:
  - '*'

at last but not least, you have to test if the user has those privileges correct,
you can test like this:
kubectl auth  can-i get pods --as myorg:readonly

in the above replace the pods with the object you want the users of the group to have access to.
Hope this helps!


Importing non-default workspaces in Terraform

As the docs state, workspaces are for distinct resources:

Named workspaces allow conveniently switching between multiple instances of a single configuration within its single backend. They are convenient in a number of situations, but cannot solve all problems.

A common use for multiple workspaces is to create a parallel, distinct copy of a set of infrastructure in order to test a set of changes before modifying the main production infrastructure. For example, a developer working on a complex set of infrastructure changes might create a new temporary workspace in order to freely experiment with changes without affecting the default workspace.

If you would like all developers to see the same state, they have to use the same
workspace. If you want this to be the case for arbitrary workspaces, then it seems that this is an abuse of the concept of the workspace. It is possible, but I would strongly advise against it, to import resources from non-default workspaces into the default one, in some kind of automated way. You could also use the default workspace state as the non-default's remote state, but again, I would strongly advise against this, arguing that it introduces unnecessary complexity and fragility to your process.
If you want all developers to see the same resources in the state, then they should simply all use the same workspace. If they are not seeing the correct state, it is because the workspace or the remote state location that they are using is different.


I'd say it is an anti-pattern as you lose the portability of Docker.  Now you will be relying on the build server's configuration instead of portable base image.

If reducing complexity of the Dockerfile is the goal why not do something like this:

Build the .net core project in Docker (eliminates reliance on the build server).
Run the tests (inside or outside of Docker).
Now push the image and run it.

Basically break your Dockerfile into three discrete pipeline steps.  This allows the build to fail on a discrete step.  It also reduces the complexity of the Dockerfile (at the cost of increased complexity in your pipeline).


Why is developing Ansible Playbooks using Roles a best practice?

There are many reasons that using roles is better than using long playbooks, and very few reasons that using a single long playbook is better than using a role. These are almost always clear only at a certain scale. Team size, workload number and complexity and frequency of execution all come into play here and the larger any of them get, the more obvious it becomes that organising things hierarchically and modularly into roles is the right thing to do.

From the Ansible docs, there is an emphasis on re-usability:

  Roles are ways of automatically loading certain vars_files, tasks, and handlers based on a known file structure. Grouping content by roles also allows easy sharing of roles with other users.

This is also why Ansible Galaxy is potentially useful. One can summarise the effect of a long list of tasks, into a single concept - a role. Although the role may be considered as simply a long list of tasks -- and so not that different from a playbook -- the main benefit comes from the way it scopes variables. Out of the several analogies which may be applied, one may use the Object Oriented analogy. A role is like a class. The usage of that role, applied in a playbook, is like the instantiation of a class into an object. If you consider object-oriented programming useful, you will consider object-oriented infrastructure provisioning useful.

For me, however it comes down to testing. A playbook is tactical and is useful in specific situations. A good role is strategic and is written to solve general problems. Since good roles are modular, they should contain unit tests for various scenarios to prove to the downstream user that they do indeed solve the more general problem, as well as making it clear what is actually covered in the tests.

Typically roles should be maintained, with proper documentation:

what variables do they use ?
what scenarios are they written for?

and wide coverage:

several base operating systems
several deployment scenarios (docker, AMIs, bare-metal, etc)

and organise tasks cleanly:

use tags
dynamically import tasks based on execution conditons and user-provided variables

etc.

In short, I wouldn't consider it a fair comparison to compare a bunch of playbooks with the usage of different roles. When it comes time to refactor your playbooks into something maintainable, you will need roles.


how is software defined infrastructure related to devops?

if I setup automate deployment of code, is this considered devops?

Sure, automating everything is a big part of DevOps.

  I tried ansible but I prefer writing my own code for learning

Now, this is laudable, but using Ansible (or Puppet or Chef or ...) would be "learning" as well. The benefit of using established tools over DIY is that if/when you meet other developers, you have a common toolset to work with. Doing it yourself is nice sometimes as well, but basing a whole software development process on 100% self-written tools (from scratch) is not a good idea, unless you have deeply evaluated the existing tools and consider them all unfit for the purpose.

When your team grows and you encounter a new developer - is it more likely that they already know about Ansible, or that they know about the code you wrote?

Also, the fact that the tools mentioned above do not do their stuff in the "code" directly, but that they have text files which configure them, is important as well. It abstracts the configuration stuff away from actual code, which will be hard to maintain a few years down the road, no matter how good you are.


How to prune all but two most recent docker images?

The problem is that the date format isn't ideal.

Supported formats for date formatted time stamps include RFC3339Nano, RFC3339, 2006-01-02T15:04:05, 2006-01-02T15:04:05.999999999, 2006-01-02Z07:00, and 2006-01-02

Reference: https://docs.docker.com/engine/reference/commandline/system_prune/#filtering
So this is really just a half answer, since it will use the date and the local timestamp.

The local timezone on the daemon will be used if you do not provide either a Z or a +-00:00 timezone offset at the end of the timestamp.

Thankfully, this comes sorted.
~: docker images --format '{{.CreatedAt}}'
2020-06-09 12:44:57 -0700 PDT
2020-05-29 14:19:46 -0700 PDT
2020-05-05 13:40:47 -0700 PDT
2020-04-28 11:55:11 -0700 PDT
2020-03-09 10:35:34 -0700 PDT
2020-03-09 10:14:43 -0700 PDT
2020-03-09 10:14:43 -0700 PDT
2020-03-09 10:14:43 -0700 PDT
2020-02-26 07:10:58 -0800 PST

I don't know how to transform timestamps in bash off the top of my head, so let's just do this the quick and messy way... the whole date!
 ~: docker images --format '{{.CreatedAt}}' | sed -n '2p' | awk '{print $1;}'
2020-05-29

Now we can do command substitution.
~: docker image prune --force --filter "until=`docker images --format '{{.CreatedAt}}' | sed -n '2p' | awk '{print $1;}'`"
Total reclaimed space: 1.344GB

I didn't put a heck of a lot of thought in to this answer, so there may be some edges cases I didn't consider. Also, if someone with better bash-fu can transform that timestamp, that would be awesome.


Advice needed - automate everything or only new/current projects

You can spend a lot of time, to automate something, which is done monthly and do routine staff manually on a daily basis - not the best way.

If you are time/resource limited, then find the bottleneck in the software delivery process and improve it, after that bottleneck is not a bottleneck anymore, find a new one and so on. So if you are spending 80% of your development time/resources on the new project, automate it first, and leave the old ones for the next step or vice a versa, depending where the main bottleneck is.


Grafana Status Panel Plugin - genereate status box for each entry from a Prometheus querry

So far I have upgraded from 6.6 to 7 and switched to the new Stat panel that managed to visualize properly. The limitation I was trying to avoid is that I can't  control fonts.


Replace/Consume Istio's Prometheus/Grafana by existing Prometheus-operator

Followed this guide https://medium.com/@emirmujic/collect-cluster-and-istio-metrics-with-prometheus-operator-42d4499d621a and it worked out for me on Istio 1.2.2:
https://medium.com/@emirmujic/collect-cluster-and-istio-metrics-with-prometheus-operator-42d4499d621a


How to cache third party resources in a build pipeline?

I'm not overly familiar with Docker particularly but it sounds a lot like issues one would get when building a normal .NET application or java app with the need to store their external dependencies (DLL or jar).

To solve this I'm quite fond of artifactories. Two of the most popular are sonatype's Nexus and JFrog's Artifactory.  Both offer a free Open Source version and both will have some way to store customised package or use it as a raw depot.  There are other options out there, I just listed these two because I've worked with them already and know they will do what you need.  As a bonus you might also be able to use them to manage the docker image directly as well.

There are others too worth mentioning, you can find a comparison table for them, you'll get a better view of their capabilities. (thanks to Karl Harnagy for the link)

Note that some of these options might require one or another of the paid versions so YMMV.

Some continuous integration systems will also support the concept of artifacts (from experience Jenkins and team-city do this, others might as well though) which should allow you do achieve similar results directly.

If none of these do it for you you can look into other technologies or roll your own.  I personally like to leverage whatever I already have this reduces the maintenance required in long term and makes passing the buck easier when comes time to look for new challenges elsewhere.

Hope this helps


How can I list all environment variables configured in Terraform?

On your command line, run printenv
Reference: https://man7.org/linux/man-pages/man1/printenv.1.html


Problem Restore EF Core 2.1.1 in VSTS, Hosted VS2017

I also could not restore packages after upgrading to 2.1.1. I got multiple errors similar to this:

Detected package version outside of dependency constraint: Microsoft.AspNetCore.App 2.1.0 requires Microsoft.EntityFrameworkCore (= 2.1.0) but version Microsoft.EntityFrameworkCore 2.1.1 was resolved.

I solved my problem by explicitly specifying version 2.1.301 in the build definition.


How do you handle the huge number of possible permutations of Dockerfiles / Teraform files?

So let's imagine you have 20 microservices, each in a separate repository. Each microservice needs to be deployed into 5+ environments: dev local, dev cloud, test, staging, prod. Understandably all configuration will be different in each environment. However, you don't need to put it into the Dockerfile - in fact you don't put any configuration into the Dockerfile. Your app gets all required parameters from environment variables, or may be from a config file, if it gets too complex.

So you would have one Dockerfile per project, and if your developers need to spin it up locally, they will just use docker-compose, which has all environment variables set up, and maybe a config file mounted as volume. In stage envs the configuration will be totally different, but the Docker image will be the same. Your app will need to be able to read this configuration at start up time, and bail out early if something doesn't work.

Read https://12factor.net/config for some inspiration on how to configure your app.

Now, provisioning the infrastructure with Terraform would follow the same approach. You would have a generic module which provisions a set of some resources, and it would accept a bunch of parameters which would depend on environment. You can use workspaces to manage this quite easily.

As for the last question, large companies solve this with a CI/CD system. This can be Jenkins, or Spinnaker, or any of the other tools, whichever you prefer.


Gitlab CI\CD Error: Failed to create container volume for /builds/root Unable to load image

Do you have a volume mounted with gitlab-runner-prebuilt.tar.xz? Your error log makes it seem like it can't find the file when it executes.


How to create AWS application load balancer using saltstack

Please follow official doc to create LB

Please create a IAM role or mentioned secret and access key as below

elb.keyid: HHTYHLSHDJFJFJFJFJFJFFSVJS
elb.key: basjbsdfbdsfdsjfbkgbsfbdsjfbfdgdfgdfs

Then Create config for ELB

Ensure myelb ELB exists:
    boto_elb.present:
        - name: myelb
        - region: us-east-1
        - availability_zones:
            - us-east-1a
            - us-east-1c
            - us-east-1d
        - keyid: GKTADJGHEIQSXMKKRBJ08H
        - key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
        - listeners:
            - elb_port: 443
              instance_port: 80
              elb_protocol: HTTPS
              instance_protocol: HTTP
              certificate: 'arn:aws:iam::1111111:server-certificate/mycert'
              policies:
                  - my-ssl-policy
                  - cookie-policy
            - elb_port: 8210
              instance_port: 8210
              elb_protocol: TCP
        - backends:
            - instance_port: 80
              policies:
                  - enable-proxy-protocol
        - health_check:
            target: 'HTTP:80/'
        - attributes:
            cross_zone_load_balancing:
              enabled: true
            access_log:
              enabled: true
              s3_bucket_name: 'mybucket'
              s3_bucket_prefix: 'my-logs'
              emit_interval: 5
            connecting_settings:
              idle_timeout: 60
        - cnames:
            - name: mycname.example.com.
              zone: example.com.
              ttl: 60
            - name: myothercname.example.com.
              zone: example.com.
        - security_groups:
            - my-security-group
        - policies:
            - policy_name: my-ssl-policy
              policy_type: SSLNegotiationPolicyType
              policy:
                Protocol-TLSv1.2: true
                Protocol-SSLv3: false
                Server-Defined-Cipher-Order: true
                ECDHE-ECDSA-AES128-GCM-SHA256: true
            - policy_name: cookie-policy
              policy_type: LBCookieStickinessPolicyType
              policy: {}  # no policy means this is a session cookie
            - policy_name: enable-proxy-protocol
              policy_type: ProxyProtocolPolicyType
              policy:
                ProxyProtocol: true

# Using a profile from pillars
Ensure myelb ELB exists:
    boto_elb.present:
        - name: myelb
        - region: us-east-1
        - profile: myelbprofile

# Passing in a profile
Ensure myelb ELB exists:
    boto_elb.present:
        - name: myelb
        - region: us-east-1
        - profile:
            keyid: GKTADJGHEIQSXMKKRBJ08H
            key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs

Register instance with ELB.
salt.states.boto_elb.register_instances(name, instances, region=None, key=None, keyid=None, profile=None)

add-instances:
  boto_elb.register_instances:
    - name: myloadbalancer
    - instances:
      - instance-id1
      - instance-id2

You may find this blog post useful also.


AWS lambda: works locally but fails in deplyment

I've resolved the issue now.

First of all, console.log(e) should be used to log the result. Then the error will appear in CloudWatch.

The problem was that the role that was used by the lambda did not permit invoking other lambdas. This can be fixed by granting the missing InvokeFunction permission for the role.


How can I develop and support mainframe software without going broke?

One possibility would be to find a 3rd party business owning such infra and selling on-demand usage/access time/connectivity to it as a service for more convenient expenses. For example Mathru Technologies (found through plain google search).

Another possible approach is to obtain such services from strategic partners or maybe even your very customers. Maybe even for free, depending on the quality of your relationship.

You could also ask the mainframe community for advice and/or leads in this direction.


When setting up Foreman should this be setup on the same server as my puppet ro ansible master?

The Foreman comes with a Puppet Master whether you like it or not since The Foreman uses puppet as part of it's installer. This can't be disabled and The Forman will leave it behind and running - so you might as well use what is deployed anyway and make the Puppet Master and your Foreman server one in the same. You can keep your Ansible installation elsewhere, however The Forman integrates with Ansible and has Ansible callbacks
 so it might make sense to keep your configuration management stack all in one place.

On the other hand, The Forman can be a big install with lots of moving parts that seem to get gummed up from time to time,
 so you might be better off putting together an Ansible playbook to redeploy your Foreman stack since Ansible can also manage your Forman resources - so this would be a good argument to keep the Ansible bits seperated.


How to programmatically enable and disable SQS triggering AWS Lambda?

Call Lambda's UpdateEventSourceMapping API with the {"Enabled":true/false} request body.
If, for some reason, you want your own API for this, you can create one in API Gateway as described below.
Before you begin, you need the UUID of the event source mapping you wish to enable/disable. Use the AWS CLI to get this (it isn’t shown on the UI):
aws lambda list-event-source-mappings --function-name my-func

{
    "EventSourceMappings": [
        {
            "UUID": "b962dcee-173c-47r1-9600-5c6d81catf85",
            "BatchSize": 1,
            "EventSourceArn": "arn:aws:sqs:us-east-1:123456789012:my-queue",
            "FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:my-func",
            "LastModified": 1595058588.141,
            "State": "Enabled",
            "StateTransitionReason": "USER_INITIATED"
        }
    ]
}

Next, either create a new API in API Gateway or add a resource & a GET method to your existing API. Configure it as shown below. The UUID you got above is highlighted below:

The execution role shown above must allow API Gateway to invoke Lambda APIs.
Scroll further down & provide this mapping template:
Content-Type: application/json

{
    "Enabled" : $input.params("Enabled")
}

Save & deploy to a stage. Now you can hit this API with the query param Enabled=true / Enabled=false to enable/disable your Lambda's SQS trigger. Here's a test done from the API Gateway console:

For a detailed screenshot-guided walkthrough of this entire process, see my blog post.


Untracable glassfish docker container image or clean installation?

Look into Payara.  It's a Glassfish fork/drop-in replacement that's better in most ways, including support.
There is a Payara 5 docker image available right on docker hub:
https://hub.docker.com/r/payara/server-full


TL;DR: the "Canonical" version of Kubernetes has been customized by Canonical, the developers of Ubuntu, based upon their experiences of running Kubernetes.  "Kubernetes Core" on the other hand is the standard, official, distribution you would find on Kubernetes.io.  If you are looking to understand Kubernetes on Ubuntu then I would recommend the Canonical version, if you are wanting a standardised version then go for Kubernetes Core.

A Spell is a function of Conjure-Up which is a product created by Canonical to be able to provide "Turn Key" installations of popular Open Source technical stacks such as Kubernetes and Open Stack.  These stacks are typically non-trivial to install, so Conjure Up combines the installation of the software and configuration based upon guided best practice through a GUI and CLI interface.

  conjure-up lets you summon up a big-software stack as a “spell” — a model of the stack, combined with extra know-how to get you from an installed stack to a fully usable one. Start using your big software instead of learning how to deploy it.

Conjure Up is capable of deploying to Ubuntu, MacOSX and popular public clouds.

At the time of writing the following spells are available:

kubernetes
openstack
bigdata incorporating: Hadoop, Spark and Kafka


How should we automatically rebuild immutable infrastructure when new packages are available?

Part of adopting the Immutable Infrastructure Pattern is decomposing your system into small manageable pieces that can move through CI/CD Pipeline very quickly, this means that OS patches can be done quickly and in a controlled manner.  I often see clients ending up with a halfway house where infrastructure is mostly immutable.

However, there are a few approaches to this which I have used in large-scale deployments of cloud architecture, typically I implement more than one as part of a Defense in Depth strategy:

Security Information and Event Management (SIEM): These products, for example, LogRhythm Security Intelligence Platform and more generic products such as ElasticStack take feeds from the operating system, which includes the output from a frequent check for updates.  The trick here is to get the information about what to patch quickly and automatically to inform decisions about when to roll those through your pipeline.
Vulnerability Management systems are more tailored than SIEM in that they are only focusing on vulnerabilities across the system so could catch issues with libraries installed as part of the software deployed to the system but not managed by the Operating System.  This might highlight vulnerabilities for which there is no patch (yet... hopefully).
Dependency Check tools form part of your pipeline and can be configured to fail the build if check-ins add vulnerabilities, this also works if a new vulnerability is added to the tool since the last check-in.


What are ChatOps approach use cases of production infrastructure management?

ChatOps is an absorbing topic; practically it means something very different to each team using it.  So much so that Atlassian have put together what could be considered to be a Chat maturity model.

Sputnik: Trying chat out to see how it works.
Mercury: Starting to move to chat.
Gemini: Chat as the primary source of communication, experimenting with bots.
Apollo: Chat as the nexus of communication and work, including bots.
Elon Musk: Chat is the primary centre of mission critical work.

Important: Chat and ChatOps is very specific to an organisation, think of Chat as the Fabric for your team's bots to existing within it's the bots that provide the functionality that enables business and technical processes.

With the brief lesson in ChatOps Theory over I can talk about the experiences that I have had with ChatOps:

DevOps Support

Slack and PagerDuty has fantastic integration allowing any newly raised incidents in PagerDuty then posted in one or more Slack Channels with information about the incident and buttons for acting upon the knowledge:

Beyond connecting PagerDuty and Slack we also connected all of our monitoring and alerting tools into Slack so that it was the central point for all real and potential problems, including the ability to query New Relic directly from Chat.

Chat as the Modern Command Line

With another client we built custom bots that executed scripts on remote machines, we did this using a combination of calling the management APIs in Azure and running Cluster Fork commands:

  @checky scale serviceA 30: Scale ServiceA to 30 machines.

  @checky kill serviceA#4 : Kill instance number for on ServiceA.

  @checky exec serviceA#1,5 ps -awx : run the ps command on serviceA instances 1 and 5 and post the output to chat

StackOverflow Close Vote Reviewers

We have some chatbots running in the SOCVR channels that help us to:

Find questions that need close votes.
Identify comments that contain "Heat" - i.e. might be offensive or lead to escalations.
Identify questions that are of poor quality that might need review.

Further Reading

So, What is ChatOps? And How do I Get Started?
VictorOps / O'Reilly ChatOps Book
ChatOps Reddit
The Cog Book


Redirect to a custom page on 503 error in Ingress

There is no built-in configuration/annotation to handle this.
Although, you can use the server-snippet annotation to create a custom configuration that intercepts the error 503, proxying the request to a service that is serving your custom error page. Example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_intercept_errors on;
      error_page 503 = @errorpages;

      location @errorpages {
        proxy_set_header X-Code $status;
        proxy_pass http://errors-svc.default.svc.cluster.local;
      }
spec:
  rules:
  - host: ameba.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80


How to patch AssemblyInfo in TeamCity for .net core 2.1 projects?

You can use File Content Replacer build feature with predefined templates for .NET Core
https://www.jetbrains.com/help/teamcity/file-content-replacer.html#.NET+Core+csproj+templates


JFrog Artifactory or JFrog Bintray , when to use one over the other?

JFrog Artifactory and JFrog Bintray both manage binaries (and any other file type you can think of). I'd like to see them as two different parts of your CI/CD pipeline.

Artifactory is mostly meant to be used inside the organization. For managing all binaries coming in as dependencies (like maven central jars) and being produced by your build process (like your pack.rpm or org/pack-docker docker image). These can later be used by others in the org or released to Bintray.

Bintray is meant to provide a universal distribution platform for your end deliverables. For example, the pack.rpm you want to be available for your customers to download. And also that org/pack-docker docker image.

Look into the features lists of each and see how they differ in order to best match their intended role.

Artifactory features
Bintray features

I hope this helps explaining the differences.

Disclosure: I work at JFrog


Should we name names in a blameless post-mortem/retrospective?

Usually for a blameless postmortem, the best idea is to go further than the human error (which for a proper 5 whys should not arise, that's rule 11, but we're just humans :)) and complete the 5 Whys with the 5 Hows.

To follow on this particular case for a blameless postmortem I'd go few step further whith those iterations (For the sake of the example) of How possible follow up interpretation (possible answers):

How this fix has been left unmerged after creation ? (it wasn't reviewed and didn't follow the usual path)
How did it miss review ? (That was urgent and straightforward, review wasn't necessary and decision was taken to deploy)
How this process has been defined ? (It wasn't)
How was the bug report closed without merge (No validation/no defined process for this case)
How to prevent the problem arise again ? (Define the process and set validation/review even for quick patches)

Now you don't have someone who have forget to merge something, you have an analysis of how it was possible and more important, you have something actionnable to improve which is the main, if not the sole, reason to do a postmortem.

A more detailled criticism of the 5 whys (which as you show end often with a Who instead of a Why) can be found on O'reilly's blog post The infinite hows

To address the specific point of naming in the analysis, I'd argue neither the name nor the fix itself are of value, wording the step in the analysis as "An engineer wrote a fix and didn't merge it" without any link would be enough for the report.


Jenkins Cancel (jobs) from Queue by Label

Found it by myself, if anyone will be interested check:

getAssignedLabel()

in.
https://javadoc.jenkins-ci.org/hudson/model/Queue.Item.html


Q: " Prevent misspelled variables passed to Ansible roles ... remove the defaults/main.yml, ... But then I can't take advantage of the benefits of having defaults. Are there other solutions?

A: Yes. Tell include_role to read other file defaults_from. Fit the content of this file to your needs. For example

- include_role:
    name: foo
    defaults_from: main_special.yml
  vars:
    fooo_version: "2.0"


You do not need to use rewrite for your case, since you want to append _plugin/kibana to any routes.
Assuming the nginx config is for http://origin.com/:
server {
    listen 80;
    server_name origin.com;
    location / {
        proxy_set_header X-Forwared-Proto $scheme;

        # http://origin.com/req/1/2 will go to http://target.com/_plugin/kibana/req/1/2
        proxy_pass http://target.com/_plugin/kibana;

    }
}


Fluentd unable to read logs from all namespaces

Million dollar question - are you running your fluentD as a DaemonSet?  If not, there's a chance that fluentd simply isn't running on the node where some containers live.
Otherwise, my best guess is, it's missing permissions to read from all namespaces, especially since you mentioned you haven't attached a ServiceAccount to it.  If you used a public FluentD Helm chart such as this one, try making sure you're deploying it with rbac enabled, i.e.:
rbac:
  create: true

serviceAccount:
  create: true

It'll also help if you post some logs from your fluentd container.


Deployment custom App in Microsoft Teams

One could use NuGet. This tool is embedded in various tools, including Chocolatey.


First you need to install sshpass.

Ubuntu/Debian: apt-get install sshpass
Fedora/CentOS: yum install sshpass

Example:

sshpass -p "YOUR_PASSWORD" ssh -o StrictHostKeyChecking=no YOUR_USERNAME@SOME_SITE.COM

Custom port example:

sshpass -p "YOUR_PASSWORD" ssh -o StrictHostKeyChecking=no YOUR_USERNAME@SOME_SITE.COM:2400

Notes:

sshpass can also read a password from a file when the -f flag is passed.

Using -f prevents the password from being visible if the ps command is executed.
The file that the password is stored in should have secure permissions.


Jenkins Not pulling the latest commit into master or slave

You can use a plugin called "Workspace Cleanup Plugin" which wipes out the workspace post build. This can be a part of the pipeline and would not require manual cleanup every time when a build should be triggered.

For the second issue, I would recommend you to check if you're able to checkin to the same repository with git credentials configured in jenkins settings through the command line in the slave.


Globbing is a term used for matching file name patterns with wild cards.  You can see a more detailed definition here https://searchsecurity.techtarget.com/definition/globbing

As it relates to Salt Stack, it's finding the minion id using wild cards.  Additional details here: https://docs.saltstack.com/en/latest/topics/targeting/globbing.html


Your assumption is correct: you will not need to perform patching on an RDS instance, AWS handles that for you. There is a caveat, though: the majority of updates are not forced by AWS, but require an automated maintenance window to execute.

Periodically, Amazon RDS performs maintenance on Amazon RDS resources. Maintenance most often involves updates to the DB instance's underlying hardware, underlying operating system (OS), or database engine version. Updates to the operating system most often occur for security issues and should be done as soon as possible.

Some maintenance items require that Amazon RDS take your DB instance offline for a short time. Maintenance items that require a resource to be offline include required operating system or database patching. Required patching is automatically scheduled only for patches that are related to security and instance reliability. Such patching occurs infrequently (typically once every few months) and seldom requires more than a fraction of your maintenance window.

Source: Maintaining a DB Instance
That document has a lot of useful information about the types of maintenance, the maintenance windows, and the manual actions you can take.
Database engines need patching, too, and AWS refers to this as a minor engine upgrade. When necessary, these upgrades happen during the aforementioned maintenance windows, but you need to ensure your RDS instance has Auto Minor Version Upgrade enabled to take advantage of this feature.
Source: Automatically Upgrading a Minor Engine Version


Grant access permission to file when mounting from host

I don't think volumes can be specified with relative paths.
Put an absolute path instead of ./elastic-stack-ca.p12 and you should be fine.


docker-compose run container before other container stop

If you mean make sure that container A does not execute and then exit then no.  Docker containers will stop if the container crashes or if the main process exits.

However, if you are attempting to run container B and it is dependent on container A, I think the depends_on flag may be what you are looking for.  This will allow you to ensure that service B will run after service A is started.  However, you may need to add something to service A that ensures that it is ready before attempting to connect to it (e.g. a database).


Multi jobs over local (IntelliJ) and Jenkins

You can check out this MultiJob plugin for Jenkins. Or if its the same job you want to run in different configuration you can make use of Multi-configuration job type in Jenkins.

I have tried searching for JUnit5 supporting multi jobs scenario as specified didn't find any decent information on the same.


convert declarative pipeline to scripted pipeline

The scripted equivalent to the declarative
steps {
  // pipeline steps here
post {
  failure {
    // failure handler here
  }
  success {
    // success handler here
  }
  always {
    // always handler here
  }
}

is the native Groovy try { } catch { }:
try {
  // pipeline steps here

  // success handler here
} catch(Exception e) {
  // failure handler here
} finally {
  // always handler here
}

I'm not sure off the top of my head how to add an abort handler, but I think you might be able to do it by inspecting the currentBuild.result variable in the failure handler block.


Changed the challenge from tls-sni to:

  --preferred-challenges http

and the certificates are renewed.


Load balancing Nexus artifact repository

Sonatype's Nexus 3 Pro supports High Availability through a couple of mechanisms that are collectively known as Component Fabric:

Peer-to-peer Repository Managers means there is no one master, also known as a single point of failure.  Packages are replicated between the nodes to ensure they are eventually consistent.
Storage Backends mean you can use high durability storage such as S3.
Dynamic Nodes enable auto-scaling support to increase capacity when demand is high and decrease it when demand is low to reduce costs.

It's not really in Sonatype's interests to support HA for the community project as it would cannibalize some of the enterprise customers from their paid product.


If you're previsioning your raspberry pi SD cards then you would probably be best using something like packer to build the actual image (you can use ansible as the provisioner for packer if you are really keen) then once you have your pi image, burn it onto the SD card.

If you need to make subsequent configuration changes them that would be where ansible fits most neatly in managing the ongoing configuration of the pi's


post jenkins build console output to slack

I figured it out by myself, there is no plugin needed.

build.sh &> /dev/stdout | tee jenkins.txt
and then curl -F file=@jenkins.txt -F channels=yourchannel -H "Authorization: Bearer your token" https://slack.com/api/files.upload

There is no '#' needed for slack channel and channel should not be private.

And if you use buils.sh | tee jenkins.log then you can't get the log from stderr.


Visiting https://youjenkinsdomain/safeRestart will set it in the mode when it waits for jobs to stop and then restarts.


What are formal criteria for establishing declarative pipelines, or pipelines as a code?

Pipeline as Code (PaC) means that the CI/CD configuration will be stored in the VCS instead of in the CI/CD tool itself, e.g. configuring it manually by using a UI.

  Any experiences what can be the "critical mass" where this investment
  is a clear pay-off and not an overkill?

Imagine that more than 100 jobs have been configured manually the last couple of years and one would like to transform all these jobs to PaC then it would mean that less resources will be available to create new features and therefor not advisable, but if one would create tasks to address this and transform a manual job to a PaC one every sprint then it could be beneficial in the long run.

Advantages of PaC

No endless searching for configuration anymore

One advantage is that one does not have to look in various places, e.g. CI/CD tool, documentation how a CI/CD job has been configured as the relevant stuff resides in the same place as the code, i.e. VCS.

No endless and unnecessary debugging anymore

Another advantage is that someone cannot break the CI/CD job easily like it is possible when it is configured using a UI. If for example someone has to change the CI/CD could then one has to change the code in the VCS. As every code change could be tracked, it will be immediately clear who changed the pipeline and what was changed when a pipeline is broken.


How to mount a single 100TB SSD on AWS and access the device over nfs from different machines within the same VPC?

create storage in AWS Console
mount created storage into VM
configure nfs deamon to present the storage volume in the VPC
configure Clients to use/mount NFS volume coming from server

sometimes the easiest is the best, this even works on other systems, not just AWS.


Understanding how Docker volumes work with compose

db_data:/var/lib/mysql simply means that the db_data volume that was previously created will be made available in the container at /var/lib/mysql.

In you example it's created at the very end of the compose file.
From the Docker documentation: "The docker volume db_data persists any updates made by Wordpress to the database."

Compose volumes works the same way as the Docker engine.

In you case db_data is just a named volume instead of a path like your second example.


Your idea will work. The organization I work for maintains some open source examples how to do deploy to AWS using a pipeline. In comparison we use GitLab for the purpose. Instead of Travis we use GitLab Runner. You configure it using the file .gitlab-ci.yml which runs the tests and deploy the AWS Lambda to dev, test and production environment every time a commit is done.  You store your secret account variables in the CI/CD settings of the repository.

There are the usual drawbacks of AWS Lambda such as difficult or impossible to test and run locally and waiting for deployments takes time, but your deployment process will be good practice.


How do you simulate "sudo su " in Ansible?

I met the same situation. I have login1 for my machine, passwordless sudo and login2, under which I should perform some actions. I did not solve it by ansible means. But I made such a workaround:

- name: "Install nvm"
  shell: sudo -u buildkite-agent bash -c "<my commands to be performed on behalf of buildkite-agent>"
  become: true

Such a case:

- name: "Install nvm"
  shell: sudo -u buildkite-agent <my commands to be performed on behalf of buildkite-agent>
  become: true

do not change home to buildkite-agent's home, i.e. uses /root as home.


How to run Ansible command to update imunify360 settings

It turns out single quote works where double quote and backslash didn't.

- name: Enable rapid scan
  command: imunify360-agent config update '{"MALWARE_SCANNING"':' {"rapid_scan"':' true}}'

I learned that folding block scalar (>) is an easy alternative and a good way to avoid escaping.


Make a job easily searchable in Azure Runbook

AFAIK currently there is no way to label job so that it is easily identifiable from the Overview screen. If interested, you may raise feature request in this UserVoice / feedback forum. I see this related feature request already raised so you may even upvote it. In general, Azure feature team would check feasibility of a feature request, triage it, prioritize against existing feature backlog, add in roadmap as appropriate and would announce and/or update the related Azure document once a feature request is addressed.


Can GitHub's Pull Request model be used for implementing post approvals?

Yes, I do believe so.  To explain that I need to lay some groundwork on how I have implemented something similar, I have simplified the model in an attempt to make it as clear as possible.

Assumptions

I am making the assumption here that Jenkins, TeamCity or similar is being used as the CI/CD tool of choice. Additionally GitHub is being used and that there is a well-defined and appropriately controlled branching structure:

Configuration

In this example, GitHub is configured as follows:

The Black 'Master' branch can only be merged into using Pull Requests, direct commits are disallowed.
The Blue 'Development' branch can accept direct commits or merges; in practice, there may be additional restrictions on this branch.
The Red 'Hotfix' branch can accept direct commits and merges.
Required Status Checks are enabled in strict mode to prevent pull requests merging when the branch is failing the build.
If the Hotfix branch is ahead of Master then pull requests into Master will be blocked, either with the Status API or the Pre-Receive Hooks on GitHub Enterprise.

The CI/CD tools are configured as follows:

Builds from the Development branch cannot be deployed to production.
Builds from Master can be deployed to all environments.
Builds from Hotfix can be deployed to all environments.
Deployments from Hotfix will notify some non-development function, for example, the Change/Release team and ask them to perform the post-approval.

Notes

Master is protected as it represents the current state of production, to do this practically you may have another "Release" branch that deployments are made from and only when successful merge into the Master branch.

Key Points

The Blue Development branch is basically a free-for-all.  Hotfix is kind of a free-for-all but any deployments trigger a type of Break Glass by notifying a non-development function who will perform the post-approval and in the process will merge the change into Master.

It's essential merges into Master stop while Hotfix is ahead of Master to:

Prevent a deployment from Master overwriting the Hotfix, which could result in an regression.
Prevent a Hotfix sitting in the Hotfix branch languishing without being merged in.

In some organisations it may help to prevent all pushes to the central GitHub repository while a Hotfix is pending post-approval.


Have you considered setting up a build server (i.e. Jenkins, TeamCity) with disposable build agents?

If you do it in the cloud (i.e. via ec2 integration), you can literally set up an AMI with generic dependencies, then configure it to spin up a new VM for each build agent, which can be configured whichever way, and then delete the VM once you've done publishing your artifact (i.e. RPM).


Can we reuse Variables across variable groups in Azure DevOps?

My understanding of the variable groups is that you would create this to leverage the same variables across multiple pipelines.  If you needed to update the value, you'd only have to update it on the library instead of updating it on dozens of pipelines.  If you are only going to use the variable group on a single pipeline definition, variables groups appear to be overkill.

If you have concerns about duplicating code and maintaining the exact same deployment steps across your stages, I would recommend you group them into a task group.  That way you can manage versioned updates and will also have it reflected across all pipelines and stages where it is referenced.

I think you would have some name resolution issues if you tried to use the different variable groups that have the same name on a single pipeline.  Instead, you should probably leverage the variable scope on the pipeline.  It will grab the appropriate value staged to the scope you are executing.


Ansible: Other option available for telnet check of open ports?

You can use the Ansible wait_for module which checks a specific TCP port is open.
Since in this case, all ports should be open already, we can use a minimal no. of retries, just enough to cover network issues:
- name: Check all port numbers are accessible from the current host
  wait_for:
    host: mywebserver.com
    port: "{{ item }}"
    state: started         # Port should be open
    delay: 0               # No wait before first check (sec)
    timeout: 3             # Stop checking after timeout (sec)
  ignore_errors: yes
  with_items:
    - 443
    - 80
    - 80443

By default, Ansible will check once every second (configurable in Ansible 2.3 using the sleep attribute), so this will check 3 times per port.
Run this in a playbook against your inventory of 400+ hosts - Ansible will check in parallel that all hosts can reach mywebserver.com on those ports.

the parallelism is subject to the forks setting in your ansible.cfg.

We use ignore_errors: yes here so that any errors are marked in red but do not stop execution.
Open ports are reported as ok items in output and closed ports are reported as failed (you must use -vv flag on ansible-playbook to see this output).
Fine-tuning output
If you want more specific output for the success and failure cases, the code must be more complex, adding a second task:

wait_for task must register a variable
the second task produces output using debug based on success/failure condition (e.g. using Jinja2 conditional expression)
then you need to put both these tasks in an include file (without any with_items loop), and write a main playbook task that uses an include ... with_items to call the include file once per port.


Docker convenience script stuck after "Scanning linux images"

Just found this question from 5 days ago because I ran into the same thing on a RPi 4, also using the convenience script.

Just waiting a bit did not seem to help. So I killed the process after a while. And Docker is running just fine, now.


Helm Repo: can it host dockerfiles and other image-building artifacts

Well that's a legit question IMHO, but there are multiple way of doing things (and no one is worth than other).
1- Everything from dev to prod at the same place
If you want to grab everything from dev to production in a single project, it's possible, but you cannot build the docker image from hell command.
Helm is only here to organize, use template in yaml and sync releases following kubernetes file structure.
You can follows helm repository structure and add other scripts, but you have to execute steps in the right order: build, push, deploy
2- The "regular" way to do
I more often saw a separation of production configuration files and application files.
What I do is:

One repository which carry my application source code + everything for the CI (test, build, docker file)
One repository which carry my helm.charts for all my infrastructure.

Choose what suits the best for you, but keep in mind that helm is a templating system for kubernetes yaml files, so it will not build the docker image.


What means “system engineer practicing DevOps”?

What the comment means is that DevOps is a way of working, a culture if you will, not a job title. There are plenty of blog posts about it.

While they have a point, one could also argue that Cubism is a way of painting, but "cubist painter" is still perfectly valid.

This is a potentially long and opinionated discussion, which is out of scope of this forum.

The point is, the definition of DevOps is too lacking on specifics about what exactly the person practicing it does and what kind of tools they use to do it to be of any actual use as a job title. Which is why it is often misunderstood and misused. And there is (almost?) always a better title to describe the job.

For example, if your job consists of solely setting up pipelines in Jenkins, then you might get slapped with the title "DevOps Engineer". However:

A job that specialized would be contrary to the idea of DevOps in the first place.
"Jenkins specialist" is a more apt title. Just an example, there are better names for it.

The title is avoided by most companies these days, often replaced by "Automation Engineer". Which, while still vague, it is less so then DevOps Engineer.

Going back to the specifics of the comment, many times the people that installed Jenkins (and the like) in a company were the Systems Engineers. And they probably also explained how it works to other people in the company, maybe even used the term DevOps to explain the philosophy behind it. Which is why the two are often confused. "The guy that takes care of Jenkins? Oh, you mean the DevOps guy? Yeah, DevOps Engineer". No, he's a Systems Engineer practicing DevOps. If nothing else, he should know a lot more than just Jenkins.


StatefulSet or Deployment for Jira on Kubernetes

Deployments are primarily based on stateless services, where data is externalized to be stored on some other storage block or database. Using deployments for stateful services is not recommended as you will encounter scaling problems, since multiple replicas will read from the same persistent volume.

Instead, StatefulSets are the best approach to stateful services. Each replica of the StatefulSet has its own state mounted on a dedicated persistent volume, thus scaling a StatefulSet will create a new persistent volume and attach the new replica to it.


stop pulling a docker image from the public docker-hub registry

You cannot change the default registry (see this). You have to specify which registry to use like this:

docker pull myprivatereg:port/library/image

Or, in a Dockerfile:

FROM myprivatereg:port/library/image

(This question was also asked in StackOverlow).


docker-compose.yml - make network name configurable via environment variable

If you have envsubst installed (part or gnu gettext) you can use something like that:

export STAGE_NAME="xyz"
export STAGE_NETWORK_PREFIX="1.2.3.4"
docker-compose -f <(envsubst docker-compose.yml) ...

If your shell does not support process substitution, something like that would do the trick:

envsubst docker-compose.yml | docker-compose -f - ...


Linking nodes in a cluster to run a single instance of a docker container

This is not really possible with container orchestration. Containers are bound by host and cannot be larger than host machines.

At the same time, depending on your specific tasks there may exist specialized distributed software that may be suitable for your needs - i.e. Apache Spark covers a lot of computational use cases.


git - develop base and extended feature - commits in base should reflect in extended in pull

Because you have a large project with multiple teams using multiple development branches will give you merge headaches, especially if the core and extended features have a significant overlap in the codebase.

The extended feature team(s) will spend a lot of time periodically merging the core feature changes and dealing with the fallout from each such merge, which can not be properly/realistically estimated ahead of time. If there are multiple teams working on the extended feature they might decide to go on separate branches to further isolate and protect themselves against these periodic merge instabilities. It's just an illusion, that'll make things even worse at the overall project level - it can lead to Branch Merge Hell.

You may want to take a look at Trunk-Based Development (TBD), with feature-flags and/or branch by abstraction. Most effective when using a gated-commit CI system.


how to run multiple environments in single job in Jenkins

A pretty standard way of doing this would be to chain your jobs or your pipelines.  This lets you keep separate jobs (which can be useful), but lets you trigger one job from the other and visualize the end-to-end flow.

So, you still just tell one job to run, but it runs both properly.  Their config and logs are still well separated and you get all the other isolation related benefits.

Example: https://www.jrebel.com/blog/using-jenkins-build-pipelines

You can also do this in normal Jenkins jobs not using a pipeline; there is an option to trigger other jobs near the bottom usually.


EC2 EBS Free Tier - Is Storage Reusable?

While I can't find it explicitly written anywhere, I do know that textbooks frequently reference that you can use the free tier to avoid costs, and that they frequently have you create and destroy instances between chapters.  So, I find it extremely hard to believe that you would get billed for deleting and creating new EBS volumes.

So, I believe that if you create a 30GB one, destroy it, and create another, you should be fine.  You should definitely make sure to remove the volumes and any related snapshots though just to be safe.

They definitely don't seem to explicitly confirm that though... e.g. in text like this:

  Notable offers for the first 12 months following your AWS account sign-up include some level of free usage for Amazon Elastic Compute Cloud (EC2), Amazon Elastic Block Store (EBS), Amazon S3, Amazon Relational Database Services (RDS), and many other useful services.

They use the free tier to get customers hooked.  They want you to like it and not be frustrated so you're more likely to use the real services and so that you may advocate them in your businesses.  So, by that logic, I similarly find it hard to believe they would "trick you" by billing you in this situation without making it more obvious.  That would be bad for sales.


Task to delete users without mount points in /home (using register and when)

Several problems in your playbook.

loop is basically expecting a list, not a string containing a variable name. You need to expand your variable with a jinja2 expression => loop: "{{ some_list_variable }}"
It will be hard to give you the correct syntax without seeing the exact data structure of your result var but... Typically, the register from a loop contains a results list of hashes, each containing info about the ran command for each loop iteration. This is your base loop for the second task => loop: "{{ del_users_reg.results }}"

If you still have trouble fixing your playbook, update your question with a debug of the registered var and I will update my answer accordingly.


How many builds does jenkins need to run before it has estimated times?

I checked out the Jenkins source code, and I believe Jenkins uses the last 3 successful or unstable builds for its estimation. If it out of the last 6 builds it can't find 3 successful or unstable builds, it uses 1 or more of the last completed (not aborted) builds. It then takes the total duration time of the "candidates" and simply divides it by the number of candidates (i.e. mean of the durations). Here is the code I based this off of that can be found in the Job.java file in the Jenkins repo:

    /**
     * Returns candidate build for calculating the estimated duration of the current run.
     *
     * Returns the 3 last successful (stable or unstable) builds, if there are any.
     * Failing to find 3 of those, it will return up to 3 last unsuccessful builds.
     *
     * In any case it will not go more than 6 builds into the past to avoid costly build loading.
     */
    @SuppressWarnings("unchecked")
    protected List<RunT> getEstimatedDurationCandidates() {
        List<RunT> candidates = new ArrayList<RunT>(3);
        RunT lastSuccessful = getLastSuccessfulBuild();
        int lastSuccessfulNumber = -1;
        if (lastSuccessful != null) {
            candidates.add(lastSuccessful);
            lastSuccessfulNumber = lastSuccessful.getNumber();
        }

        int i = 0;
        RunT r = getLastBuild();
        List<RunT> fallbackCandidates = new ArrayList<RunT>(3);
        while (r != null && candidates.size() < 3 && i < 6) {
            if (!r.isBuilding() && r.getResult() != null && r.getNumber() != lastSuccessfulNumber) {
                Result result = r.getResult();
                if (result.isBetterOrEqualTo(Result.UNSTABLE)) {
                    candidates.add(r);
                } else if (result.isCompleteBuild()) {
                    fallbackCandidates.add(r);
                }
            }
            i++;
            r = r.getPreviousBuild();
        }

        while (candidates.size() < 3) {
            if (fallbackCandidates.isEmpty())
                break;
            RunT run = fallbackCandidates.remove(0);
            candidates.add(run);
        }

        return candidates;
    }

    public long getEstimatedDuration() {
        List<RunT> builds = getEstimatedDurationCandidates();

        if(builds.isEmpty())     return -1;

        long totalDuration = 0;
        for (RunT b : builds) {
            totalDuration += b.getDuration();
        }
        if(totalDuration==0) return -1;

        return Math.round((double)totalDuration / builds.size());
    }


just create an ingress resource along side each application, that way you can modify individual ingress resources without impacting other applications. Load balancers do not allow you to do anything meaningful with your traffic.

With ingress you can use cert-manager, external-dns, ssl offloading, etc.


How to add rules to Kubernetes NGINX Ingress controller from different yml definitions?

I was able to combine your two example files by first converting them to JSON.

foo.json

{
  "apiVersion": "extensions/v1beta1",
  "kind": "Ingress",
  "metadata": {
    "name": "nginx-ingress"
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "foo.bar.com"
        ],
        "secretName": "tls-secret"
      }
    ],
    "rules": [
      {
        "host": "foo.bar.com",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "foo-web-svc",
                "servicePort": 80
              }
            },
            {
              "path": "/api",
              "backend": {
                "serviceName": "foo-rest-svc",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

baz.json

{
  "apiVersion": "extensions/v1beta1",
  "kind": "Ingress",
  "metadata": {
    "name": "nginx-ingress"
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "baz.bar.com"
        ],
        "secretName": "tls-secret"
      }
    ],
    "rules": [
      {
        "host": "baz.bar.com",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "baz-web-svc",
                "servicePort": 80
              }
            },
            {
              "path": "/api",
              "backend": {
                "serviceName": "baz-rest-svc",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

You can then combine the rules for each by using jq.

levi@La-Tower:~$ jq 'reduce inputs as $i (.; .spec.rules += $i.spec.rules)' foo.json baz.json
{
  "apiVersion": "extensions/v1beta1",
  "kind": "Ingress",
  "metadata": {
    "name": "nginx-ingress"
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "foo.bar.com"
        ],
        "secretName": "tls-secret"
      }
    ],
    "rules": [
      {
        "host": "foo.bar.com",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "foo-web-svc",
                "servicePort": 80
              }
            },
            {
              "path": "/api",
              "backend": {
                "serviceName": "foo-rest-svc",
                "servicePort": 80
              }
            }
          ]
        }
      },
      {
        "host": "baz.bar.com",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "baz-web-svc",
                "servicePort": 80
              }
            },
            {
              "path": "/api",
              "backend": {
                "serviceName": "baz-rest-svc",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

While yaml is more common JSON is perfectly valid for the Kubernetes API. All credit for the jq logic goes to @Jeff Mercado and his answer.


kubernetes cluster node with 4cpu can't hold two pods with 2cpu

It seems like you found your answer but let me add a little color since we went through the same thing.

Your yaml for the pods can have both a request and limit for both CPU and memory.  Since you are discussing CPU here I'll stick with that.  What you can do in order to utilize your resources efficiently is set your resource request to the minimum requirement to get scheduled. Say 1.8.  But also set the limit to a higher limit, say 2 or more.  What this will do is make your pod be schedulable on the machine and if CPU is not completely consumed, your pod will utilize the available CPU up to 2 or the limit you set.  This is the way kubernetes can make good use of the resources by "sharing" CPU to the application that needs it.

See here for requests and limits:
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/


Terraform | AWS | vpc_id causing apply to fail

I'll go out on a limb here and assume that the issue is actually in your "aws_instance" block and is likely the result of how you are assigning the security groups in the "aws_instance".

Copied from the first linked article, in the resolution:

change

security_groups = ["cloud9Test"]

to

vpc_security_group_ids = ["${aws_security_group.cloud9Test.id}"]

Two related articles:

https://github.com/KainosSoftwareLtd/aws-api-gateway-demo/issues/3

https://stackoverflow.com/questions/31569910/terraform-throws-groupname-cannot-be-used-with-the-parameter-subnet-or-vpc-se/34586893#34586893

If this is not on track, please post the "aws_instance" declaration.

For your own sanity I'd also recommend giving the resources names that are more specific to their role, and not identical, for example "cloud9Test_vpc".


Deploying a statefulSet with static volume provisioning but controller insists to use dynamic

Did you try by specifying a storageClassName since Kubernetes will be using default storage class when it's not specified?
See Dynamic Provisioning of the blog post and Dynamic from Kubernetes documentation?
Or try with by removing the default storageclass in the cluster to explicitly use given PV and PVC configuration.


The Phoenix Project and Military service

My take on the military references in the book, as a veteran, has to do with Man, Machine, Method, Measure; and, with drilling.

The military is entirely other compared to civilian operational models. Setting aside the chain of command and the Military Justice systems; consider that the way any service member mops, waxes and buffs a floor is detailed in a procedure outlining what (exact) equipment is to be used, how it is to be calibrated, how the person is to operate said equipment and measures by which to determine compliance with the procedure. The same goes for brewing coffee, making Kool-Aid, or launching aircraft from a carrier deck.

Secondly the military runs "drills" constantly so emergencies aren't chaotic and reactive; they are methodical and responsive. Hence these would be recommended practices in a DevOps context. Practice well-documented and consistently executed procedures often to increase reliability and responsiveness in the face of the unknown.

In Phoenix project they apply both these practices to tame the chaos that Bill and the team have been subject to throughout the book.


Given a blank slate, tool recommendations?

This question runs two risks:

of being marked as "primarily opinion-based", and
its answers might age poorly.

However, I think it's useful to express an opinion, if only for posterity; if and when things change, there may be better answers -- but there will always be blank slates.

I would split the problem up into four sections:

Infrastructure
Application Building, testing and integration
Deployment
Monitoring, logging and analytics

Furthermore, let's make this cloud-independent (so, this toolkit should be valid for any or all of AWS, GCE, Azure etc, as well as on-prem OpenStack, bare metal cloud (e.g. Packet, etc).

Infrastructure

Infrastructure components consist of

Your environments (pre, prod, integration, testing, etc - however you define them). These should be built with Terraform modules.

Each module in its own repo
Consider carefully separation of state, so that you can easily create and destroy components and keep costs down

"DevOps" infrastructure. This includes:

code repos
CI/CD instances (controller, agent pools, etc) - e.g. Jenkins master with cloud agent pools.
Deployment runners (see later).
Monitoring, logging and analytics.
Again, build these with terraform modules. As a example, have a Terraform module for each type of application or component.

Team, messaging and support infrastructure. This is often overlooked, but choosing the right tool for the job on Day 2 is hard. It's better to communicate to the stakeholders how issues and change requests should be made. I would venture a combination of GitHub issues and Slack at the very least.
secrets management. You will need to pass around a lot of secrets, if you want to foster collaboration. In order to do this securely, you will need a place to manage them and delegate access to and usage of them.

Building, testing, integration.

These happen in Jenkins. Each application should contain a subdirectory .pipeline with the parts necessary to build it. The part which concerns us here is everything before prod.

I would suggest a pipeline as code approach with a Jenkinsfile in the repo declaring the steps for

building the application:
Testing the application

Unit tests
Functional tests
Integration tests
(Regression tests)

Building artifacts

building and signing artifacts
Shipping them to the artifactory, with new version info and other metadata

Building instance images (if your application is "installed" in an environment, and not serverless).
Enforcing changes to the Infrastructure (terraform plan/apply)
Enforcing changes to the deployment configuration (capacity, etc)

At the end of this stage, you'll have a properly configured environment (say, pre or prod), an artifact, and an image which you can deploy. If need be, you can run performance tests (with Taurus) or acceptance tests (with cucumber) on them with a job in Jenkins.

Deployment

I would vote for a hard separation between the previous phase and this one. Deployment should be done with a different tool and the right tool for the job is in my opinion Spinnaker. The configuration for the deployment should be kept in the same repo as the application (so that developers can serve themselves). Deployments should be done on every change to the codebase, declaratively -- Spinnaker has a JSON syntax for this, making it easy to check consistency and correctness of the deployment scenario.

Monitoring, logging and analytics

As mentioned before, Day 1 is a blank slate, but it is quickly dirtied. The DevOps doesn't stop on Day 2 (ie, when apps are deployed), and it would be wise to plan ahead.

The SRE book gives good advice for instrumenting and monitoring the applications you will be deploying, but this space is very crowded with services and tools claiming solve your monitoring and alerting needs. Again going with the principle that we should be able to deploy anywhere, I would go with a Prometheus for indicator and metric extraction and Grafana for displaying setup. However, you should definitely consider this part wisely.

So - the toolkit looks like:

Vault for secrets
GitHub for code, projects, tasks and issues
Slack for communication
Terraform for infrastructure
Ansible for configuration management
Packer for image baking
Quay for (docker) image registry
Inspec for compliance and correctness of state
Cucumber for Acceptance Testing
Jenkins for pipeline execution
Spinnaker for Deployment
Prometheus for metrics Grafana for visualisation and dashboards
Taurus (Blazetool) for performance testing

Is this a lot of tools? Yes.

Will you be able to deploy anywhere? Also yes.

Will you need all of them? probably not. Also, I've chosen tools which are mostly free to use and open source.

Also note: in this scenario, the entire state of all applications can and should be represented as code, and can thus have semantic versioning (or other kind of versioning which makes sense to you). Any changes to that state can be represented by changes to the code, and can thus have good source code management principles applied to it, such as code review, automated testing, etc.


How to route to specific, of multiple datacenters, with a single domain

Proxying is gonna cause you some issues when someone takes a trip to China and tries to access data in the USA, depending on what your app does they could cause you some $$$ issues when your Chinese POP is sending/receiving all of their traffic each way.

Instead I suggest:

If your API is authenticated and you're not going to replicate any of the data and you're using AWS then I'd use lambda@edge to do what you need.

You can use GEO-DNS to route to a lambda function which handles authentication. This lambda function can also hold a record of where users need to go and issue a redirect to the correct endpoint.

This does mean that you'll still need domains like api-region3.company.com but it will allow api.company.com to perform the task of being universal. It's down to the client application then to hold on to that URL? At least using redirects instead of proxies should help with network traffic.


what do you think of making fast decisions?

start using it if you fail, you can select another technology

This approach has two major drawbacks:

Technical debt

While it may be tempting to apply "fail fast" mentality to architecture and design, skipping or abbreviating the design phase would be a mistake. fast-fail should be used in conjunction with good design practices, not in lieu of them. Making hasty decisions can often result in poorly scaled solutions. While using MySQL seems like a good idea when you are working with a 100mb database to generate reports, it scales poorly into the terabyt range for doing your archival reports.
Using DNSMasq to run an overlapping DNS Zone might seem like a good political decision at the time, migrating off of that and resolving the disparity to between your DNS Zone and the overlapping upstream zone a few years later turns into a nightmare.

Expense

Technology can be expensive. Buying a solution only to discard it later cares with it hugely wastful expenses. But even more expensive is the time expended by personnel. Salaries are every companies' largest expense. Wasting time developing a solution only to discard it and develop a replacement is wildly inefficient.

This however should not be construed to mean that decision paralysis or analysis paralysis isn't a problem. It is! Fail-fast just isn't the best way to combat it. There are other, much more tried and true ways to combat this without comprimising good design principles.


Cross region replication of Redis Docker container data

You don't mention orchestration, are you planning to implement all by your own configuration on top of Ubuntu?

I recommend you that take a look at Kubernetes and StatefulSet workloads

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

You can use Ubuntu as your host to accomplish your needs, but in my experience, you can only need to define correctly your workload and open the right ports to communicate and with the kubernetes integration on different providers you have the persistent volume easy to manage.

Also, I recommend you to review Helm and you have a redis-ha chart ready

https://github.com/helm/charts/tree/master/stable/redis-ha


Control GKE CICD from a Jenkins in a lab with private network?

Taking the url from cluster-info and creating a config file with jenkins and using at as secret did the trick to pass the connection.
I am still curious how to authenticate with a bearer token though. I have tried to use existing managed Jenkins secret, but it failed with it.
Update: The authentication happens, but after creating pods the job never runs.
Update: It looks like using jenkinsci docker image as a hardcoded podtemplate works, I try to work with Groovy provisioned yaml template that calls GCR and there I got the firstly mentioned error.


The "saltey" way of doing this is typically in a map.jinja file. This one for example...

In that, you would:

{%- load_yaml as serverlist %}
#Note that a default list could be provided here
{%- endload %}

{%- set serverlist2 = salt['grains.filter_by'](serverlist, merge=salt['pillar.get']('pillarname:server_list_de')) %}
{%- set servers = salt['grains.filter_by'](serverlist2, merge=salt['pillar.get']('pillarname:server_list_uk')) %}

Then in your sls file you would then put at the top:

{%- from "formulaname/map.jinja" import servers with context %}

Servers should then be available to you. I've never tried merging twice though, so give it a try. At a minimum this may give you some direction as the syntax for doing a pillar merge like this may get you what you are looking for.


Is there a way to run with_items loops in parallel in Ansible?

As @webKnja mentioned this is possible with async mode. I have recently discovered it myself and learned that you can use it in 3 different ways depending on your needs.

Execute and poll the results, notice the poll:5, This will poll the results every 5 seconds. You may save some time with this method.

- name: My long runing task
  some_module_name:
    ip: "{{item.fabric}}"
    username: "{{user}}"
    password: "{{password}}"
    secret: "{{secret}}"
  loop: "{{zoning_list}}"
  register: _alias_vc_0
  async: 60
  poll: 5

Fire and forget poll: 0, This is very quick option since Ansible it's just shooting out those tasks. The down side is that we don't know what was the outcome of the task i.e. changed: True/False. Of course it's a downside if you care about the feedback ;).

name: My long runing task
some_module_name:
  ip: "{{item.fabric}}"
  username: "{{user}}"
  password: "{{password}}"
  secret: "{{secret}}"
loop: "{{zoning_list}}"
register: _alias_vc_0
async: 60
poll: 0

Fire and forget with async_status, the syntax for the task is the same as example 2 whowever it will require additional task async_status. This is my favorite since it's relatively fast (faster then normal looping or the execute and poll) and allows you to capture the feedback although will need to deal with new register for your async_task.

retries: 20 -- how many attempts before failing.

delay: 2 -- how many second to wait between polls.

- name: My long runing task
  some_module_name:
    ip: "{{item.fabric}}"
    username: "{{user}}"
    password: "{{password}}"
    secret: "{{secret}}"
  loop: "{{zoning_list}}"
  register: _alias_vc_0
  async: 60
  poll: 0

- name: Wait for My long running task to finish
  async_status:
    id: "{{ item.ansible_job_id }}"
  register: _jobs_alias_vc_0
  retries: 20
  delay: 2
  until: _jobs_alias_vc_0.finished
  loop: "{{_alias_vc_0.results}}"

A word of caution, depending on the task yo may not be able to use the async option. I had examples where I was interacting with system which was not able to handle multiple requests for the same resource. I found async option best working if I have to perform the same task across multiple hosts. That's where I was able to "save" the most time.

Since you posted the link to Ansible documentation in the question I'm not going to do that.


Looking for Ansible equivalent method to yml Anchors, Aliases, when using include file

There is module_defaults playbook-level option:

- hosts: localhost
  module_defaults:
    file:
      owner: root
      group: root
      mode: 0755
  tasks:
    - file:
        state: touch
        path: /tmp/file1
    - file:
        state: touch
        path: /tmp/file2


Salt Stack environments ignoring saltenv

State and pillar environments are set independently.

Specifing

environment: prod

in the minion configuration will force the minion to use the prod state, but it will still use the default pillar data. To select a pillar environment you will also need:

pillarenv: prod

This also does not work in old versions of salt and is only supported with pillar.get as of salt version 2016.3, so you may be in need of an update.


How can I get the Docker target platform inside the Build Environment / Dockerfile

This is typically done with a script, not the Dockerfile. The script (perhaps in bash or python) can run at build time to determine what environment it is in, and act accordingly. This can be included in the RUN command, but it risks becoming quite complex and difficult to read in the Dockerfile. Furthermore, each additional RUN or CMD entry in the Dockerfile adds a layer to the image, which is considered bad practice.
This script can instead take the arguments BUILDPLATFORM and TARGETPLATFORM as suggested in the buildx docs, which would make more declarative. Instead of discovering the target, you would declare it (and write your script accordingly), so that the user understands exactly what happens without having to look inside the build.